Methods for linking streaming subtitles and emoticons
By generating temporary facial expression data during streaming voice interaction and matching it after receiving complete subtitles, the semantic mismatch problem caused by out-of-order subtitle segments is solved, ensuring that the facial expressions on the device are consistent with the semantics, and reducing transmission and decoding overhead.
Patent Information
- Application Number
- CN202611244557.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-17
- Publication Date
- 2026-11-10
AI Technical Summary
In streaming voice interaction scenarios, network jitter can cause subtitle segments to arrive out of order, be repeated, or be missing, leading to semantic mismatch between subtitles and emoticons.
Before a single subtitle segment is fully received, temporary, correctable emoji data is generated and then spliced together into a complete subtitle for emoji matching after the subtitle is received, ensuring that the non-verbal interactive emojis output by the device are consistent with the complete semantics.
Even if some subtitle segments are semantically incomplete, they can be corrected after the complete subtitles are determined, avoiding semantic mismatch and reducing transmission and decoding overhead.
Smart Images

Figure CN123075147A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method for linking streaming subtitles and emoticons. Background Technology
[0002] During intelligent agent broadcasting or text generation, the server splits the continuously generated subtitles into multiple segments and sends these segments to the terminal or intermediate service via long connections. As an important verifiable semantic carrier in streaming voice interaction, subtitles need to maintain semantic consistency with the terminal's multimodal output in order to convey the user's interaction intent.
[0003] In traditional technologies, subtitles are typically transmitted as a whole message, or processed piece by piece at the receiving end according to their arrival order to generate emoticons. However, in streaming voice interaction scenarios, subtitle segments are semantic fragments generated consecutively during the streaming generation and speech playback of the same agent's response in the large model, and each segment together constitutes a complete utterance. Due to network jitter, segments may arrive out of order, be repeated, or be missing. If emoticons are generated directly based on the arriving segments, it is easy for the emoticons to mismatch with the complete semantics. Summary of the Invention
[0004] Therefore, it is necessary to provide a method for linking streaming subtitles and emoticons that can ensure the consistency between non-verbal interactive emoticons output by the device and the complete semantics, in order to address the above-mentioned technical problems.
[0005] Firstly, this application provides a method for linking streaming subtitles and emoticons, including: Receive subtitle segments from streaming output; Before the subtitle segments corresponding to a single subtitle have been fully received, correctable temporary emoticon data is generated based on the received subtitle segments. Send the temporary emoticon data to the device to instruct the device to call local animation resources and output non-verbal interactive emoticons corresponding to the temporary emoticon data; When the subtitle segment corresponding to the single subtitle is received, the subtitle segments corresponding to the single subtitle are spliced together into a complete subtitle; Perform facial expression matching on the complete subtitle to obtain full-sentence facial expression data; The full-sentence emoticon data is sent to the device to instruct the device to call local animation resources to update or maintain the non-verbal interaction emoticons.
[0006] Secondly, this application also provides a device for linking streaming subtitles and emoticons, including: The segment receiving module is used to receive subtitle segments from streaming output; The subtitle splicing module is used to splice the subtitle segments corresponding to a single subtitle into a complete subtitle when the subtitle segments corresponding to a single subtitle are received. The temporary emoji generation module is used to generate correctable temporary emoji data based on the received subtitle segments when the subtitle segments corresponding to a single subtitle have not been fully received. The full-sentence emoji generation module is used to match emojis in complete subtitles and obtain full-sentence emoji data; The emoticon data delivery module is used to send temporary emoticon data to the device to instruct the device to call local animation resources and output non-verbal interactive emoticons corresponding to the temporary emoticon data; and to send full-sentence emoticon data to the device to instruct the device to call local animation resources and update or maintain non-verbal interactive emoticons.
[0007] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the streaming subtitle and emoticon linkage method in various embodiments.
[0008] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the streaming subtitle and emoticon linkage method in the various embodiments.
[0009] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the streaming subtitle and emoticon linkage method in various embodiments.
[0010] The aforementioned method for linking streaming subtitles and emoticons allows for the generation and transmission of correctable temporary emoticon data based on the received subtitle segments before the subtitle segments corresponding to a single subtitle have been fully received. This enables the device to output non-verbal interactive emoticons first. After the subtitle segments are received, the segments are concatenated into a complete subtitle, and emoticon matching is performed on the complete subtitle to obtain full-sentence emoticon data, which is used to update or maintain the non-verbal interactive emoticons already displayed on the device. Thus, even if temporary misjudgments occur during the streaming phase due to incomplete semantics in some subtitle segments, they can be corrected after the complete subtitle is determined, ensuring that the non-verbal interactive emoticons output by the device are consistent with the complete semantics. This avoids semantic mismatches caused by judging emoticons based solely on partial subtitle segments. Simultaneously, the device can use local animation resources to output non-verbal interactive emoticons, eliminating the need to download emoticon images or videos for each subtitle, thus reducing transmission and decoding overhead. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a diagram illustrating the application environment of a method for linking streaming subtitles and emoticons in one embodiment. Figure 2 This is a flowchart illustrating a method for linking streaming subtitles and emoticons in one embodiment; Figure 3 This is a structural block diagram of a device for linking streaming subtitles and emoticons in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0014] The method for linking streaming subtitles and emoticons provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, device 102 communicates with server 104 via a network. During intelligent agent broadcasting or text generation, server 104 splits continuously generated subtitles into multiple subtitle segments and sends these segments to device 102 or an intermediate service via a long connection; it also generates temporary emoticon data and full-sentence emoticon data and sends the emoticon data to device 102 via the network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104, or it can be located on the cloud or other network servers. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0015] Device 102 is a terminal device with facial expression output capabilities, such as a smart companion terminal, including but not limited to children's companion robots, desktop companion devices, and home smart agents with screen facial expressions, lighting effects, or motion output capabilities. It can also be various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Due to the limited display area, network bandwidth, and computing power of device 102, and the fact that users' main attention when communicating with such terminals is usually focused on the device itself and voice, it is necessary to convert the subtitle understanding results into a small number of deterministic facial expression states so that device 102 can synchronously express the interaction intention with non-verbal signals when broadcasting content such as comfort, encouragement, reminders, refusal, and apologies.
[0016] In some embodiments, such as Figure 2 As shown, a method for linking streaming subtitles and emoticons is provided, which can be applied to... Figure 1 Taking device 102 as an example, the explanation includes the following steps 202 to 212.
[0017] Step 202: Receive the streamed subtitle segments.
[0018] In this context, subtitle fragmentation refers to the process by which the server breaks down continuously generated subtitles during agent-based speech or text generation. Subtitles serve as crucial verifiable semantic carriers in streaming voice interaction, linked to the speech playback progress and the terminal's facial expression state. Subtitle fragments are semantic segments continuously generated during the same agent's response within the large model's streaming generation and speech playback process. These fragments collectively constitute a complete discourse, rather than being independent messages sent by the user that can be displayed side-by-side. To obtain complete semantics, subtitle fragments must be restored to a unique order based on device / session, message identifier, and fragment sequence number.
[0019] Specifically, the receiving end continuously receives subtitle segments from the server via a persistent connection or streaming channel, and parses each subtitle segment to extract information such as message identifier, segment sequence number, total number of segments, and text content. Upon receiving a subtitle segment, the receiving end registers it in the corresponding reception status according to the message identifier and triggers a check to determine whether the reception of a single subtitle segment is complete. The receiving end can process subtitle segments in the order they arrive, or it can buffer them first and then process them in order, as long as it can recover the complete order of a single subtitle segment based on the message identifier and segment sequence number.
[0020] Step 204: Before the subtitle segments corresponding to a single subtitle have been fully received, generate correctable temporary emoticon data based on the received subtitle segments.
[0021] In this context, a single subtitle refers to a complete subtitle text with continuous semantics, identified by the same message identifier, belonging to the same agent's response. Temporary emoticon data refers to emoticon data generated based on the received partial subtitle segments before all subtitle segments corresponding to a single subtitle have been received. Correctable temporary emoticon data refers to emoticon data that can be corrected or revoked by subsequent emoticon data generated based on more complete text. "Not yet fully received" means that not all valid subtitle segments of the single subtitle have been received.
[0022] Specifically, each time the receiving end receives a subtitle segment, it checks whether the current single subtitle has been completely received. If a single subtitle has not been completely received, the receiving end uses the received subtitle segments as input for expression matching. Following a process of expression keyword matching, contextual condition judgment, semantic parsing, negative language structure correction, and semantic confidence scoring, it generates correctable temporary expression data. The temporary expression data can carry expression status identifiers and corresponding control parameters, such as expression type, confidence level, priority, retention period, and version number. The input for generating temporary expression data can be all received subtitle segments or a portion of the received subtitle segments that meets preset conditions.
[0023] Step 206: Send temporary emoticon data to the device to instruct the device to call local animation resources and output non-verbal interactive emoticons corresponding to the temporary emoticon data.
[0024] In this context, "device-side" refers to terminal devices capable of outputting facial expressions, such as smart companion terminals, including but not limited to children's companion robots, desktop companion devices, and smart home devices with screen-based facial expressions, lighting effects, or motion output capabilities. "Local animation resources" refers to animation resources pre-installed on the device, such as on-screen facial expression animations, lighting effect animations, or motion animations. "Non-verbal interaction expressions" refers to non-verbal expressions output by the device through on-screen facial expressions, lighting effects, or animations.
[0025] Specifically, after generating temporary emoticon data, the receiving end sends the temporary emoticon data to the device via a persistent connection. The emoticon status identifier carried in the temporary emoticon data indicates which type of emoticon the device should output, while control parameters instruct the device on how to output that emoticon. Upon receiving the temporary emoticon data, the device searches its local animation resources for the animation corresponding to the emoticon status identifier and plays the animation according to the control parameters, thus outputting a real-time non-verbal interactive emoticon during the streaming phase. The device outputs the emoticon by calling local animation resources, eliminating the need to download emoticon images or video media data for each subtitle.
[0026] Step 208: When the subtitle fragments corresponding to a single subtitle are received, the subtitle fragments corresponding to the single subtitle are spliced together into a complete subtitle.
[0027] In this context, "complete subtitle" refers to the complete text obtained by concatenating all subtitle segments of a single subtitle in ascending order of segment number. "Received successfully" means that all valid subtitle segments of the single subtitle have been received.
[0028] Specifically, the receiving end continuously maintains the segment reception status of each subtitle. After confirming that all valid subtitle segments of a single subtitle have been received, it concatenates the text content of each subtitle segment sequentially according to their order within the single subtitle, resulting in a complete subtitle corresponding to the single subtitle. The concatenated complete subtitle is used for subsequent emoji matching.
[0029] Step 210: Perform facial expression matching on the complete subtitles to obtain the full sentence's facial expression data.
[0030] Full-sentence facial expression data refers to facial expression data obtained by matching facial expressions based on complete subtitles, used to ultimately determine the facial expression state on the device. Unlike temporary facial expression data, which is generated based on partial subtitle segments, full-sentence facial expression data is generated based on semantically complete subtitles, reflecting the full semantic meaning. Candidate facial expressions are determined by the semantics of the subtitle text, rather than solely based on the acoustic emotion of the speech.
[0031] Specifically, the receiving end uses the complete subtitle as input for expression matching, performs semantic parsing on the complete subtitle, identifies the matched expression keywords in the complete subtitle, and combines contextual conditions, negative language structure correction, and semantic confidence score to determine the full-sentence expression data. This full-sentence expression data is used to review or correct the non-verbal interaction expressions currently output by the device after the complete semantics are determined.
[0032] Step 212: Send full-sentence emoji data to the device to instruct the device to call local animation resources to update or maintain non-verbal interaction emojis.
[0033] Here, "update" refers to the device switching the currently output non-verbal interaction emoji to the non-verbal interaction emoji corresponding to the full-sentence emoji data. "Keep" refers to the device continuing to output the current non-verbal interaction emoji.
[0034] Specifically, when the full-sentence emoji data matches the emoji corresponding to the temporary emoji data, the receiving end instructs the device to maintain the currently output non-verbal interaction emoji. When the full-sentence emoji data does not match the emoji corresponding to the temporary emoji data, the receiving end sends the full-sentence emoji data to the device, and the device calls local animation resources to update the currently output non-verbal interaction emoji to the non-verbal interaction emoji corresponding to the full-sentence emoji data.
[0035] In this embodiment, before the subtitle segments corresponding to a single subtitle are fully received, correctable temporary emoticon data is generated and sent based on the received subtitle segments, enabling the device to output non-verbal interactive emoticons first. After the subtitle segments are received, the subtitle segments are concatenated into a complete subtitle, and emoticon matching is performed on the complete subtitle to obtain full-sentence emoticon data, which is used to update or maintain the non-verbal interactive emoticons already displayed on the device. Thus, even if temporary misjudgments occur during the streaming stage due to incomplete semantics of some subtitle segments, they can be corrected after the complete subtitle is determined, ensuring that the non-verbal interactive emoticons output by the device are consistent with the complete semantics, avoiding semantic mismatch caused by judging emoticons based only on some subtitle segments. At the same time, the device calls local animation resources to output non-verbal interactive emoticons, eliminating the need to download emoticon images or videos for each subtitle, thus reducing transmission and decoding overhead.
[0036] In some embodiments, generating correctable temporary facial expression data based on received subtitle segments includes the following steps: When an emoji keyword is matched based on the subtitle segment, the subtitle segment is semantically parsed to obtain the first candidate emoji set; Identify negative language structures in subtitle segments, and based on these negative language structures, perform negative pattern correction on the scoring parameters of at least some candidate expressions in the first candidate expression set; Based on the scoring parameters after negative pattern correction, semantic confidence scores are calculated for each candidate expression to obtain the target score for each candidate expression. Based on the target score, determine the temporary facial expression data that can be corrected.
[0037] Among them, emoji keywords refer to phrases with clear interactive intent and low ambiguity when appearing alone, such as reassuring phrases like "Don't worry, I'm here with you, it's okay," encouraging phrases like "You can do it, you did well, keep going," reminder phrases like "Please pay attention, be careful, don't touch," and apology phrases like "Sorry, I apologize." The keyword table can be configured with parameters such as keyword text (pattern), emoji type (emotion), base score (baseScore), priority (priority), and required context (requiredContext). For example, the keyword "Don't worry" corresponds to the reassuring (CALM) emoji, with a base score of 0.88 and a priority of 60. Meeting preset context conditions means that the text context of the emoji keyword meets preset context requirements, such as the keyword being located in a continuous and uninterrupted text segment starting from the initial caption segment, or the text content size within the preset window before and after the keyword meeting the required context requirements. Semantic parsing refers to performing lexical, syntactic, or semantic analysis on the caption segments to identify the emoji keywords matched within the caption segments. The first candidate emoji set refers to the set containing one or more candidate emojis obtained after semantic parsing of the caption segments.
[0038] Negative language structures refer to language structures that alter the meaning of keywords in an emoji, including at least one of negation, double negation, and contrast structures. Scoring parameters are quantitative parameters used to evaluate candidate emojis, including keyword base score, completeness coefficient, context coefficient, and source coefficient. Negative pattern correction refers to adjusting the scoring parameters of candidate emojis based on the identified negative language structures. The target score is the semantic confidence score of the candidate emoji. Correctable temporary emoji data refers to emoji data generated based on partial subtitle segments that can be corrected or revoked by subsequent emoji data.
[0039] Specifically, the receiving end compares the received subtitle segments with a preset emoji keyword list to determine whether an emoji keyword is matched in the subtitle segment. When an emoji keyword is matched in a subtitle segment and the context of the matched position meets preset conditions, the receiving end performs semantic parsing on the subtitle segment to determine the candidate emojis corresponding to the matched emoji keyword, forming a first candidate emoji set. The receiving end matches negative language structures within a preset window before and after the keyword, and adjusts the scoring parameters of at least some candidate emojis in the first candidate emoji set according to the type of negative language structure. For each candidate emoji in the first candidate emoji set, the receiving end calculates the semantic confidence score of each candidate emoji by combining its keyword base score, completeness coefficient, context coefficient, and source coefficient, and obtains the target score for each candidate emoji. The receiving end compares the target scores of each candidate emoji with a pre-judgment threshold. When the target score of a candidate emoji reaches the pre-judgment threshold, temporary emoji data corresponding to that candidate emoji is determined. When the text to be processed matches a phrase, the context meets the requirements, and the target score reaches the pre-judgment threshold (e.g., 0.85), temporary emoji data carrying the provisional=true flag is output. Single words, ordinary interjections, or discontinuous segments do not trigger prediction. When the target score of a candidate emoji does not reach the prediction threshold, neutral or default temporary emoji data can be determined, or temporary emoji data can be not generated to avoid frequent prediction. For example, if the received subtitle segment first shows "unhappy", the sad (SAD) temporary emoji is not immediately output.
[0040] For example, the text to be processed from the received subtitle segments is "Don't worry, I'll be with you." The text to be processed contains the high-confidence reassurance emoji keyword "Don't worry." Semantic parsing is performed on the subtitle segments to obtain reassurance (CALM) candidate emojis. After semantic confidence scoring of the reassurance candidate emojis, if the target score of the reassurance candidate emoji reaches the pre-judgment threshold, the temporary emoji data corresponding to the reassurance emoji is determined, and the device outputs a reassurance non-verbal interaction emoji accordingly.
[0041] In this embodiment, by matching emoji keywords based on subtitle segments, semantic parsing is performed on the subtitle segments to obtain a first set of candidate emojis when the context corresponding to the keywords meets preset conditions. Negative language structures in the subtitle segments are identified, and negative pattern correction is performed on the scoring parameters of at least some candidate emojis in the first set of candidate emojis. Based on the scoring parameters after negative pattern correction, semantic confidence scores are performed on each candidate emoji to obtain a target score. Based on the target score, correctable temporary emoji data is determined. This method can generate correctable temporary emoji data based on the received subtitle segments during the streaming stage, and considers the influence of context conditions and negative language structures during the generation process, reducing the possibility of misjudgment of temporary emoji data.
[0042] In some embodiments, matching emoji keywords based on subtitle segments includes the following steps: Extract the continuous, uninterrupted portion of the subtitle segment starting from the first subtitle segment, and use it as the text to be processed; Emoji keywords were matched based on the text to be processed.
[0043] Here, "starting subtitle segment" refers to the subtitle segment with segment number 1 within a single subtitle. "Continuous and without gaps" means that starting from the starting subtitle segment, the segment numbers exist consecutively without any missing segment numbers. "Text to be processed" refers to the text obtained by splicing together continuous and gap-free subtitle segments. "Emoji keyword matching" refers to comparing the text to be processed with an emoji keyword table to determine the emoji keywords that match in the text to be processed.
[0044] Specifically, for each subtitle segment received, the receiving end updates the currently received consecutive prefixes based on the segment sequence number. It only analyzes continuous, uninterrupted text starting from the initial subtitle segment, avoiding treating out-of-order isolated end segments as complete semantics. The text content of the subtitle segments corresponding to the consecutive prefixes is then concatenated to obtain the text to be processed. Subsequent semantic parsing and temporary emoji data generation are only performed if an emoji keyword is matched in the text to be processed. For out-of-order isolated end segments, since a continuous, uninterrupted section starting from the initial subtitle segment has not yet been formed, emoji keyword matching is not performed.
[0045] For example, a single subtitle is split into three subtitle segments, with the content being "I'll be with you," "Don't worry," and "We'll solve it together." If the subtitle segment with segment number 2 arrives first, since the subtitle segment with segment number 1 is missing, a continuous and uninterrupted section from the starting subtitle segment has not yet been formed, and emoji keyword matching is not performed. When the subtitle segment with segment number 1 arrives, the text to be processed, "Don't worry, I'll be with you," is obtained. Then, emoji keywords are matched based on this text, which can match reassuring emoji keywords such as "Don't worry."
[0046] In this embodiment, by obtaining the continuous and uninterrupted portion of the subtitle segment starting from the starting subtitle segment as the text to be processed, and matching emoji keywords based on the text to be processed, it is possible to avoid using out-of-order or incomplete subtitle segments as input for emoji keyword matching, thereby reducing the possibility of misjudgment of temporary emoji data.
[0047] In some embodiments, subtitle segments carry a message identifier, segment number, and total number of segments. Concatenating the subtitle segments corresponding to a single subtitle into a complete subtitle includes the following steps: Establish a corresponding segment buffer based on the message identifier, and store the arriving subtitle segments into the segment buffer according to the segment sequence number; When the number of valid segment numbers received in the segment buffer equals the total number of segments, and all valid segment numbers are within the range of the total number of segments, the subtitle segments are spliced into a complete subtitle in ascending order of segment number.
[0048] Specifically, when a subtitle segment with the same message identifier and segment number as a stored subtitle segment is received, the subtitle segment that arrives later is ignored; when the segment number of a subtitle segment is less than 1 or greater than the total number of segments, the subtitle segment is discarded.
[0049] The message identifier (messageId) is used to distinguish different subtitle messages. The part index (partIndex) indicates the current part's position within a single subtitle. The total number of parts (totalParts) indicates the total number of parts after a single subtitle is split. The part buffer is a data buffer area established for the same message identifier, used to temporarily store subtitle parts belonging to the same single subtitle. Subtitle parts can also carry fields such as text content (text), timestamp (timestamp), device identifier (deviceId), and session identifier (sessionId). The timestamp is used for sorting or timeout determination. The device identifier (deviceId) and session identifier (sessionId) are used to isolate different devices or different sessions. Subtitle parts can also carry fields such as message type (msgType), emotion hint (emotionHint), end identifier (isFinal), and trace identifier (traceId). The message type (msgType) is used to distinguish between ordinary subtitles, control messages, or error messages. The emotion hint (emotionHint) can be used by the server to indicate the emotion type in advance. The `isFinal` flag indicates whether this is the last segment. The `traceId` flag is used for log analysis.
[0050] A valid segment number refers to a segment number within the range of 1 to the total number of segments. A complete subtitle refers to the complete text obtained by concatenating segments in ascending order of their segment numbers. A stored subtitle segment refers to a subtitle segment that has been stored in the segment buffer. A duplicate subtitle segment refers to a subtitle segment that has the same message identifier and the same segment number as a stored subtitle segment. An invalid segment refers to a subtitle segment with a segment number less than 1 or greater than the total number of segments.
[0051] Specifically, after receiving a subtitle fragment, the device parses the message identifier in the fragment, searches for the fragment buffer corresponding to that message identifier, and if no corresponding fragment buffer exists, a new fragment buffer is created. The subtitle fragment is then stored in the corresponding position within the fragment buffer according to its fragment sequence number. Each time a subtitle fragment is stored, the device counts the number of valid fragment sequence numbers received in the fragment buffer and compares this number with the total number of fragments. When the number of valid fragment sequence numbers received equals the total number of fragments, and all valid fragment sequence numbers are within the range of 1 to the total number of fragments, the device determines that the reception of a single subtitle is complete and splices the subtitle fragments into a complete subtitle in ascending order of fragment sequence number. Before storing a subtitle fragment in the fragment buffer, the device checks whether a subtitle fragment with the same message identifier and fragment sequence number already exists in the fragment buffer. If it already exists, the device ignores the later-arriving subtitle fragments, using the content of the first received subtitle fragment as the standard, and records this information in the log. The device also checks whether the segment number of the subtitle segment is within the range of 1 to the total number of segments. If the segment number is less than 1 or greater than the total number of segments, the device discards the subtitle segment and does not store it in the segment buffer. If the segment buffer times out or the connection is broken, the corresponding incomplete text will no longer generate full-sentence emoticon data.
[0052] For example, the server generates a subtitle, "Don't worry, I'll help you finish your homework," message identifier M100, which is split into 3 subtitle fragments. Fragment number 1 contains "Don't worry," fragment number 2 contains "I'll help you," and fragment number 3 contains "Let's finish our homework together." Due to network issues, the receiving order is fragment number 2, fragment number 1, and fragment number 3. When the receiving end receives fragment number 2, it establishes a fragment buffer for M100, but the individual subtitle has not yet been fully received, so the complete subtitle is not concatenated. Similarly, after receiving fragment number 1, the individual subtitle is still not fully received. After receiving the subtitle fragment with fragment number 3, the number of valid fragment numbers received in the fragment buffer is equal to the total number of fragments, 3. And each valid fragment number, 1, 2, and 3, is within the range of 1 to 3. Therefore, the subtitle fragments are spliced together into the complete subtitle "Don't worry, I will help you complete your homework" in ascending order of fragment number.
[0053] In this embodiment, by establishing a corresponding segment buffer based on the message identifier, the arriving subtitle segments are stored in the segment buffer according to the segment sequence number. When the number of valid segment sequence numbers received in the segment buffer is equal to the total number of segments and all valid segment sequence numbers are within the range of the total number of segments, the subtitle segments are spliced into a complete subtitle in ascending order of segment sequence number. This can reliably reconstruct a complete subtitle even when subtitle segments arrive out of order, are repeated, or are missing, avoiding duplicate data from polluting the splicing result.
[0054] In some embodiments, facial expression matching is performed on the complete subtitle to obtain full-sentence facial expression data, including the following steps: Semantic analysis of the complete subtitles yields a second set of candidate emoticons; Identify negative language structures in complete subtitles, and based on these negative language structures, perform negative pattern correction on the scoring parameters of at least some candidate expressions in the second candidate expression set; Based on the scoring parameters after negative pattern correction, semantic confidence scores are calculated for each candidate expression to obtain the target score for each candidate expression. Based on the target score, determine the full sentence's facial expression data.
[0055] The second candidate emoji set refers to a set containing one or more candidate emojis obtained after semantic parsing of the complete subtitle. Unlike the first candidate emoji set, which is generated based on partial text during the streaming stage, the second candidate emoji set is generated based on semantically complete subtitles.
[0056] Negative language structures refer to language structures that alter the meaning of emoji keywords, including at least one of negation, double negation, and contrast structures. Scoring parameters are quantitative parameters used to evaluate candidate emojis, including keyword base score, completeness coefficient, context coefficient, and source coefficient. Negative pattern correction refers to adjusting the scoring parameters of candidate emojis based on the identified negative language structures. The target score is the semantic confidence score of the candidate emoji. Full-sentence emoji data refers to the emoji data determined based on the emoji matching results of complete subtitles, used to ultimately determine the emoji state on the device.
[0057] Specifically, after all segments are received or an end marker (isFinal=true) is received, the receiving end re-executes rules such as keyword matching, negative pattern correction, semantic confidence scoring, and candidate comparison on the reassembled complete subtitles to generate full-sentence emoji data carrying the provisional=false marker. The receiving end takes the complete subtitles as input and performs lexical, syntactic, or semantic analysis to identify the emoji keywords within them and determine the corresponding candidate emojis, forming a second candidate emoji set. The receiving end matches negative language structures within a preset window before and after the keywords in the complete subtitles and adjusts the scoring parameters of at least some candidate emojis in the second candidate emoji set according to the type of negative language structure. The receiving end combines the keyword base score, completeness coefficient, context coefficient, and source coefficient of each candidate emoji to calculate the semantic confidence score of each candidate emoji, obtaining the target score for each candidate emoji. The receiving end compares the target scores of each candidate emoji with a preset score threshold; if the target score of a candidate emoji reaches the preset score threshold, the full-sentence emoji data is determined. Full-sentence emoji data is used to review or correct temporary emoji data after the complete semantics have been determined. When the emoji corresponding to the full-sentence emoji data matches the emoji corresponding to the temporary emoji data, the device retains the currently output non-verbal interaction emoji, only extending the retention time. When the emoji corresponding to the full-sentence emoji data does not match the emoji corresponding to the temporary emoji data, the full-sentence emoji data takes precedence, and the currently output non-verbal interaction emoji is corrected. If there are no valid candidate emojis in the complete subtitles, the temporary state corresponding to the temporary emoji data is canceled, and the device smoothly returns to a neutral emoji after the retention period ends.
[0058] Optionally, when determining the full sentence expression data based on the target score, the most recent hit position of each candidate expression can be compared, and the most recent hit position can be used as a reference for determining the full sentence expression data.
[0059] For example, the full caption is "I'm not unhappy, I'm just a little tired." Semantic parsing of the full caption yields sad (SAD) candidate expressions. Negation structures are detected in the full caption; after negative pattern correction of the scoring parameters for sad candidate expressions, these expressions are suppressed. The final determined neutral (NEUTRAL) or tired (TIRED) expression is used as the full sentence expression data, rather than a sad expression. As another example, the full caption is "You don't need to say 'Don't worry,' I need quiet now." Quotation and negation structures are identified in the full caption, lowering the score of reassurance candidate expressions, ultimately correcting the full sentence expression data to a neutral (NEUTRAL) or quiet (QUIET) expression.
[0060] In this embodiment, a second candidate expression set is obtained by semantically parsing the complete subtitles. Negative language structures in the complete subtitles are identified, and negative pattern correction is performed on the scoring parameters of at least some candidate expressions in the second candidate expression set. Based on the scoring parameters after negative pattern correction, semantic confidence scores are performed on each candidate expression to obtain a target score. The full sentence expression data is determined based on the target score. This method can determine the full sentence expression data based on the complete semantics, ensuring that the final expression is consistent with the complete semantics and avoiding misjudgment caused by incomplete semantics in some subtitle segments.
[0061] In some embodiments, the negative language structure includes at least one of a negation structure, a double negation structure, and a contrast structure. Based on the negative language structure, negative pattern correction is performed on the rating parameters of at least a portion of the candidate expressions in the second candidate expression set, including the following steps: When a negation structure is detected, the keyword base score of the candidate emoji that matches the negation structure is set to zero; When a double negation structure is detected, restore the basic score of the keyword before the candidate emoji matched by the double negation structure is set to zero; When a transition structure is detected, the context coefficient of the candidate emoji matched with the first half of the text of the transition structure is reduced, and the context coefficient of the candidate emoji matched with the second half of the text of the transition structure is increased.
[0062] Among them, negation structures refer to linguistic structures that contain negative words and cause the meaning of keywords to be negated, such as structures formed by negative words like "no," "not," "no," and "don't," combined with emoji keywords. Double negation structures refer to linguistic structures that contain two negative words and restore the meaning to a positive state, such as "not that...". Adversative structures refer to linguistic structures that cause a semantic shift, such as "although...but...". Negative linguistic structures can also include rhetorical questions and quotation structures that change the meaning of keywords, such as "could it be...?" and the quotation in "He said 'I'm very happy'".
[0063] The keyword base score refers to the basic target score of the candidate emoji corresponding to the emoji keyword. The context coefficient refers to the influence coefficient of the degree of matching of the candidate emoji with the context on the target score.
[0064] Specifically, the receiving end matches negative language structures within a preset window before and after the keywords. When a negation structure is detected, the receiving end identifies the candidate emoji matched by the negation structure and sets the keyword base score of that candidate emoji to zero, preventing it from participating in the determination of emoji data. When a double negation structure is detected, the receiving end restores the keyword base score of the candidate emoji matched by the double negation structure to its value before being set to zero, allowing that candidate emoji to regain its ability to participate in the determination of emoji data. When a transition structure is detected, the receiving end reduces the context coefficient of the candidate emoji matched by the first half of the text of the transition structure and increases the context coefficient of the candidate emoji matched by the second half of the text of the transition structure, giving the semantics after the transition structure a higher weight in the determination of emoji data.
[0065] For example, if the full text is "I'm not unhappy, just a little tired," the detection of the negative structure "not" sets the keyword base score of the sad (SAD) candidate emoji matching "unhappy" to zero, suppressing the sad candidate emoji and ultimately allowing the output of either a neutral (NEUTRAL) emoji or a tired (TIRED) emoji. Similarly, if the text is "Great, but watch out for the steps," the detection of the contrastive structure "but" lowers the context coefficient of the happy candidate emoji matching the first part "Great," and increases the context coefficient of the warning candidate emoji matching the second part "Watch out for the steps," ultimately allowing the output of a warning emoji.
[0066] In this embodiment, by setting the keyword base score of the candidate emoji matched by the negative structure to zero when a negative structure is detected, restoring the keyword base score of the candidate emoji matched by the double negative structure before it was set to zero when a double negative structure is detected, and reducing the context coefficient of the candidate emoji matched by the first half of the text of the transition structure and increasing the context coefficient of the candidate emoji matched by the second half of the text of the transition structure when a transition structure is detected, the interference of negative language structures such as negation and transition on the meaning of emoji keywords can be eliminated, and the mismatch between emoji and complete semantics can be avoided.
[0067] In some embodiments, semantic confidence scores are performed on each candidate expression based on the scoring parameters after negative pattern correction to obtain the target score for each candidate expression, including: determining the target score of the candidate expression based on the keyword base score, integrity coefficient, context coefficient, and source coefficient of any candidate expression in the second candidate expression set.
[0068] The keyword base score refers to the basic target score of the candidate emoji corresponding to the emoji keyword. The completeness coefficient refers to the influence coefficient of the completeness of the text to be processed or the complete caption on the target score; the completeness coefficient calculated based on the staged text is less than the completeness coefficient calculated based on the complete caption. The context coefficient refers to the influence coefficient of the degree of matching of the candidate emoji's context on the target score. The source coefficient refers to the credibility coefficient of the emoji data source; server-sent emoji hints (emotionHint) can be used as a source to add points, but cannot exceed the negation rule.
[0069] Specifically, for each candidate emoji in the second candidate emoji set, the receiving end obtains the keyword base score, integrity coefficient, context coefficient, and source coefficient of the candidate emoji, and calculates the target score of the candidate emoji by comprehensively calculating the above scoring parameters.
[0070] Alternatively, the target score can be calculated as follows: Target Score = Keyword Base Score × Completeness Coefficient × Context Coefficient × Source Coefficient Negation Punishment Ambiguity penalty. Specifically, the negation penalty and ambiguity penalty are used to quantify the impact of negation structure correction and semantic ambiguity on candidate expressions, respectively.
[0071] In this embodiment, the target score of a candidate expression is determined based on the keyword base score, integrity coefficient, context coefficient, and source coefficient of any candidate expression in the second candidate expression set. This allows for the comprehensive quantification of the semantic confidence of the candidate expression by considering multiple scoring parameters, thus providing a comparable basis for determining expression data.
[0072] In some embodiments, determining full-sentence expression data based on a target score includes the following steps: When the target scores of multiple candidate expressions all reach the preset score threshold, the priority of each candidate expression is compared. When multiple candidate expressions have the same priority, compare the target scores of each candidate expression; Based on the target score, determine the full sentence's facial expression data.
[0073] The preset score threshold is used to determine whether a candidate emoji meets the conditions for formal output; for example, it can be set to 0.70. Candidate emojis whose target scores do not reach the preset score threshold are not included in determining the full sentence emoji data, and the device retains the current output emoji.
[0074] Priority is a parameter used to indicate the importance or safety level of candidate expressions, and can be ranked from high to low as follows: safe / warning, soothing / apology, sad / caring, encouraging / happy, and neutral.
[0075] Specifically, the receiving end compares the target score of each candidate emoji with a preset score threshold. When the target scores of multiple candidate emojis all reach the preset score threshold, the receiving end obtains the priority of each candidate emoji, first comparing the safety priority, and then comparing the target scores. When multiple candidate emojis have the same priority, the receiving end compares the target scores of these candidate emojis. The receiving end determines the full-sentence emoji data based on the comparison result of priority and target score. For example, it prioritizes determining the full-sentence emoji data corresponding to the candidate emoji with the highest priority. When multiple candidate emojis have the same priority, it prioritizes determining the full-sentence emoji data corresponding to the candidate emoji with the highest target score.
[0076] Optionally, when determining the full sentence expression data, if the target scores are the same, the most recent hit position of each candidate expression can be compared, and the most recent hit position can be used as a reference for determining the full sentence expression data.
[0077] For example, if a complete caption simultaneously hits both warning and happy emoji keywords, and the target scores for both candidate emojis reach the preset score threshold, then warning candidate emojis have a higher priority than happy candidate emojis. Therefore, the full-sentence emoji data corresponding to the warning emoji is output. For example, after negative mode correction, the happy candidate emoji for "Great, but watch out for the steps" is reduced, while the warning candidate emoji is increased. Ultimately, the full-sentence emoji data corresponding to the warning emoji is output, instead of the happy emoji.
[0078] In this embodiment, by comparing the priority of each candidate expression when the target scores of multiple candidate expressions all reach a preset score threshold, comparing the target scores of each candidate expression when the priorities of multiple candidate expressions are the same, and determining the full sentence expression data based on the target scores, it is possible to prioritize the safer and more important expression when multiple candidate expressions coexist, thus avoiding semantic conflicts in expressions.
[0079] In some embodiments, temporary emoticon data carries a first version number, and full-sentence emoticon data carries a second version number. Sending full-sentence emoticon data to the device includes the following steps: When the full-sentence emoji data is inconsistent with the temporary emoji data, the full-sentence emoji data carrying the second version number is sent to the device to instruct the device to call local animation resources to update the non-verbal interaction emojis based on the full-sentence emoji data carrying the second version number. The second version number is greater than the first version number.
[0080] Specifically, the device receives emoticon data with a version number greater than the version number carried by the currently received emoticon data. Inconsistency between full-sentence emoticon data and temporary emoticon data means that the emoticon corresponding to the full-sentence emoticon data is different from the emoticon corresponding to the temporary emoticon data. The version number is a sequence number used to identify the newness of emoticon data. The first version number is the version number carried by the temporary emoticon data. The second version number is the version number carried by the full-sentence emoticon data. The version number carried by the currently received emoticon data refers to the version number carried by the most recently received emoticon data by the device.
[0081] Specifically, the receiving end compares the full-sentence emoji data with the temporary emoji data. When the full-sentence emoji data and the temporary emoji data are inconsistent, the receiving end sets the version number carried by the full-sentence emoji data to a second version number and sends the full-sentence emoji data carrying the second version number to the device. When the device receives emoji data, it compares the version number carried by the received emoji data with the version number carried by the currently received emoji data. When the version number carried by the received emoji data is greater than the version number carried by the currently received emoji data, the device accepts the emoji data and updates the non-verbal interactive emojis by calling local animation resources based on the emoji data. When the version number carried by the received emoji data is not greater than the version number carried by the currently received emoji data, the device ignores the emoji data to prevent out-of-order old emoji data from overwriting new emoji data.
[0082] For example, temporary emoji data carries a first version number, revision=1, corresponding to a sad (SAD) emoji. Full-sentence emoji data carries a second version number, revision=2, corresponding to a neutral (NEUTRAL) emoji. The device first outputs a sad emoji based on the temporary emoji data. Upon receiving full-sentence emoji data carrying revision=2, since 2 is greater than 1, the device accepts the full-sentence emoji data and updates the emoji to neutral. If it subsequently receives older emoji data carrying revision=1, since 1 is not greater than the version number 2 carried by the currently received emoji data, the device ignores the older data.
[0083] In this embodiment, when the full-sentence emoji data and the temporary emoji data are inconsistent, the full-sentence emoji data carrying the second version number is sent to the device, so that the second version number is greater than the first version number, and the device only accepts emoji data with a version number greater than the version number carried by the currently received emoji data. This can prevent old emoji data arriving out of order from overwriting new emoji data and ensure that the emoji state on the device is consistent with the complete semantics.
[0084] In some embodiments, the method further includes the following steps: Within the minimum retention time corresponding to non-verbal interaction expressions, when the device receives new expression data, if the priority of the new expression data is not higher than the priority of the currently output expression, the device is instructed to add the new expression data to the processing queue and not perform expression switching. After the minimum hold time ends, when the difference between the target score of the new expression data and the target score of the current output expression reaches a preset hysteresis difference, the device is instructed to switch the current output expression to the new expression data.
[0085] The minimum duration of the expression state refers to the minimum time it must remain after it is officially displayed. For example, 1500 milliseconds for a normal emotion, 2500 milliseconds for a calming emotion, and 3000 milliseconds for a warning emotion. This is to prevent short clips from causing animation flickering. The pending queue is a queue used to temporarily store data for expressions to be switched. Safety warning expression data refers to expression data with higher priority than the currently output expression, used for safety warnings.
[0086] The target score refers to the semantic confidence score of the candidate expression. The preset hysteresis difference is a switching hysteresis value set to avoid fluctuations between similar target scores, such as 0.10.
[0087] Specifically, after outputting a non-verbal interactive emoticon, the device starts a minimum hold timer for that emoticon. During this minimum hold time, when the device receives new emoticon data, it compares the priority of the new emoticon data with the priority of the currently output emoticon. If the priority of the new emoticon data is not higher than that of the currently output emoticon, the device adds the new emoticon data to the processing queue without performing an emoticon switch. If the new emoticon data is a security warning type emoticon with a higher priority than the currently output emoticon, the device immediately switches the currently output emoticon to the new emoticon data. After the minimum hold time ends, the device compares the new emoticon data in the processing queue with the currently output emoticon. When the difference between the target score of the new emoticon data and the target score of the currently output emoticon reaches a preset hysteresis difference, the device switches the currently output emoticon to the new emoticon data. If the difference does not reach the preset hysteresis difference, the device does not perform an emoticon switch.
[0088] Optionally, the method can also set a cooldown period, during which re-entry of an exited emoticon state is prohibited. The cooldown period refers to the shortest interval between exiting and re-entering the same low-priority emoticon, such as 2000 milliseconds, to suppress repeated triggering.
[0089] For example, if the full subtitle is "Don't worry, I'll help you finish your homework," the emoji matching function determines a reassuring emoji, the device outputs the reassuring emoji and sets a minimum holding time to prevent the next short subtitle from immediately switching the emoji away.
[0090] In this embodiment, by adding new emoticon data with a priority no higher than the current output emoticon to the processing queue without performing emoticon switching within the minimum holding time corresponding to non-verbal interaction emoticons, and performing switching on safety warning emoticon data with a priority higher than the current output emoticon, and only performing switching when the difference between the target score of the new emoticon data and the target score of the current output emoticon reaches a preset hysteresis difference after the minimum holding time ends, it is possible to suppress frequent emoticon switching and flashing, while ensuring timely output of safety warning emoticons.
[0091] In some embodiments, the server organizes interactive data in layers according to device, agent, channel, and message. The device layer identifies the device sending the subtitle segment using a device identifier, such as the device's MAC address or deviceId. The agent layer identifies the agent performing the broadcast or text generation using an agent identifier, such as the agentId. The channel layer identifies the communication channel between the device and the agent using a channel identifier, such as the channel itself. The message layer identifies the subtitle message generated by the same agent's response using a message identifier. The server associates each streaming response with information such as message identifier, segment sequence number, total number of segments or end identifier, version number, and timestamp, ensuring a one-to-one correspondence between subtitle segments and emoticon data within the same device and session.
[0092] Specifically, when generating streaming subtitles, the server writes the device identifier, agent identifier, channel identifier, and message identifier into each subtitle segment. The receiving end locates the segment buffer based on the device identifier, session identifier, and message identifier, reassembles the subtitles based on the segment sequence number and the total number of segments, distinguishes between new and old emoji data based on the version number, and sorts or determines timeout based on the timestamp.
[0093] In this embodiment, by organizing interactive data hierarchically according to device, agent, channel and message, and associating information such as message identifier, segment number, total number of segments, version number and timestamp with each streaming response, it is possible to make the subtitle segments and emoticon data in the same device and the same session correspond one-to-one, providing a reliable data foundation for disordered reordering and emoticon correction.
[0094] In some embodiments, timeout cleanup is performed on individual subtitles that have not been fully received. Timeout cleanup refers to clearing the fragment buffer corresponding to a single subtitle when it is not fully received within a preset time period. The fragment buffer records the creation time and last update time. The receiving end periodically scans each fragment buffer; when the difference between the current time and the creation time or last update time exceeds a preset threshold, it is considered that the subtitle message cannot be fully received, and cleanup is performed. The preset threshold can be set according to network quality and subtitle length, for example, 10 to 30 seconds.
[0095] Specifically, after cleanup, the incomplete subtitle fragments corresponding to the message identifier will no longer participate in splicing or trigger emoji data generation. The cleanup action only applies to specific message identifiers and does not affect other currently receiving single subtitles. If the device is already displaying part of the subtitle during cleanup, you can choose to maintain the current display, display an error message, or wait for the next complete subtitle to overwrite it. If the connection is lost, the receiving end cleans up the active fragment buffer associated with that connection to avoid the old state of the disconnected device remaining. If a subtitle fragment arrives late after cleanup, it will be discarded as an expired fragment and will not trigger emoji data again.
[0096] In this embodiment, by performing timeout cleanup on individual subtitles that have not been fully received and only applying it to specific message identifiers, it is possible to avoid long-term unreceived fragment buffers occupying memory and to prevent old subtitles from interfering with subsequent subtitles and emoticons.
[0097] In some embodiments, facial expression matching is performed when the received text reaches a stable window. A stable window refers to the received text reaching a preset stable length or covering a text range that includes key semantic elements. If the receiving end has not received all subtitle segments, when the received text reaches the stable window, facial expression matching is performed based on the text within the stable window, and facial expression data is output. The final facial expression state is still based on the complete subtitle or the stable window result. When more subtitle segments or complete subtitles are subsequently received, facial expression matching is performed again with the more complete text, correcting or reversing the previously output facial expression data.
[0098] Specifically, during streaming reception, the receiving end continuously evaluates the length or semantic coverage of the received text. When the received text reaches a stable window, the text within the stable window is used as input for facial expression matching to generate facial expression data and send it to the device. The stable window setting can strike a balance between the real-time performance of facial expression response and the accuracy of facial expression.
[0099] In this embodiment, by performing facial expression matching when the received text reaches a stable window, and then re-performing facial expression matching with more complete text when more complete text becomes available, it is possible to ensure the real-time performance of facial expression responses while maintaining consistency between the final facial expression and the complete semantics.
[0100] In some embodiments, during agent-based broadcasting or text generation, the server splits continuously generated subtitles into multiple subtitle segments. Each subtitle segment carries a message identifier (messageId), segment sequence number (partIndex), total number of segments (totalParts), and text content (text), and is associated with information such as device identifier (deviceId), session identifier (sessionId), timestamp, end identifier (isFinal), and revision number. Optionally, subtitle segments may also carry information such as message type (msgType), emotion hint (emotionHint), and trace identifier (traceId). The server sends the subtitle segments to the device via a persistent connection. Due to network jitter, different subtitle segments may arrive out of order, be duplicated, or be missing.
[0101] The device establishes a fragment buffer based on the device identifier, session identifier, and message identifier. The fragment buffer records the total number of fragments, the set of received fragments, the creation time, and the last update time. The device stores arriving subtitle fragments in the corresponding positions within the fragment buffer according to their fragment sequence numbers. Each time a new subtitle fragment is received, the device checks if the number of received valid fragment sequence numbers equals the total number of fragments and determines if there are duplicate or invalid subtitle fragments. Duplicate subtitle fragments are identified based on the first received content; subsequent duplicates are ignored and logged. Invalid subtitle fragments are discarded. When the number of received valid fragment sequence numbers in the fragment buffer equals the total number of fragments, and all valid fragment sequence numbers are within the range of 1 to the total number of fragments, the device concatenates the subtitle fragments into a complete subtitle in ascending order of fragment sequence number. For missing subtitle fragments, the device waits until a timeout threshold is reached without immediately reporting an error. The device periodically scans each segment buffer. When the difference between the current time and the creation time or last update time exceeds a preset threshold (e.g., 10 to 30 seconds), the corresponding segment buffer is cleared. After clearing, incomplete subtitle segments corresponding to that message identifier will no longer participate in splicing or generate emoji data. If the connection is broken, the device clears the active segment buffer associated with that connection, and late subtitle segments are discarded as expired segments. In scenarios where multiple devices or multiple sessions send subtitle segments simultaneously, the device uses the device identifier, session identifier, and message identifier to jointly locate the segment buffer, and uses thread-safe structures or equivalent concurrency control to avoid counting errors caused by two subtitle segments being written simultaneously.
[0102] Before all subtitle segments corresponding to a single subtitle are received, the device updates the current continuous stable prefix according to the segment sequence number for each received subtitle segment. It only analyzes continuous and uninterrupted text starting from the initial subtitle segment and concatenates these continuous and uninterrupted subtitle segments into the text to be processed, avoiding treating isolated, out-of-order subtitle segments as complete semantics. The device compares the text to be processed with the emoji keyword table. The emoji keyword table can be configured with parameters such as keyword text (pattern), emoji type (emotion), base score (baseScore), priority (priority), and required context (requiredContext). For example, the keyword "don't worry" corresponds to the comforting (CALM) emoji, with a base score of 0.88 and a priority of 60. When the text to be processed matches an emoji keyword, and the context before and after the match meets the required context requirements, the device performs semantic parsing on the text to be processed, obtains the first set of candidate emojis, and matches negative language structures such as negation, rhetorical questions, transitions, and quotations within a preset window before and after the keyword, performing negative pattern correction on the scoring parameters of the candidate emojis. The device calculates the target score by combining the keyword base score, completeness coefficient, context coefficient, and source coefficient of each candidate emoji. Optionally, the target score can be calculated as: Target Score = Keyword Base Score × Completeness Coefficient × Context Coefficient × Source Coefficient Negation Punishment The ambiguity penalty is calculated as follows: When the target score reaches the prediction threshold (e.g., 0.85), the device generates correctable temporary emoji data carrying the provisional=true flag and sends the temporary emoji data to the device. Single words, ordinary interjections, or discontinuous segments do not trigger prediction. For example, if the received subtitle segment first shows "unhappy," the sad (SAD) temporary emoji is not immediately output. Emoji hints (emotionHints) explicitly sent by the server can be used as source bonuses, but cannot override the negation rule.
[0103] The device receives temporary emoticon data and calls local animation resources to output non-verbal interactive emoticons corresponding to the temporary emoticon data. When the subtitle segments corresponding to a single subtitle are received, the device re-executes emoticon keyword matching, semantic parsing, negative language structure correction, semantic confidence scoring, and candidate comparison on the sequentially reassembled complete subtitles to obtain full-sentence emoticon data carrying the provisional=false flag. When determining the full-sentence emoticon data, if the target scores of multiple candidate emoticons all reach a preset score threshold (e.g., 0.70), the safety priority is compared first, then the target scores are compared, and the most recent hit position of each candidate emoticon can also be considered for determination. Emoticon priorities from high to low can be: safety / warning, soothing / apology, sadness / care, encouragement / happiness, and neutral (NEUTRAL). For example, after negative mode correction, the "happy" candidate emoticon is reduced, the "warning" candidate emoticon is increased, and the "WARNING" emoticon is finally output.
[0104] The full-sentence emoji data carries a second version number, while the temporary emoji data carries a first version number, with the second version number being greater than the first. When the full-sentence emoji data and the temporary emoji data are inconsistent, the device sends the full-sentence emoji data carrying the second version number to the device. The device only accepts emoji data with a version number greater than the version number carried by the currently received emoji data, and calls local animation resources to update the non-verbal interaction emojis to prevent out-of-order old emoji commands from overwriting new results. When the full-sentence emoji data and the temporary emoji data are consistent, the device retains the currently output non-verbal interaction emojis, only extending the retention time. If there are no valid candidate emojis in the complete subtitles, the temporary state corresponding to the temporary emoji data is canceled, and the device smoothly returns to a neutral (NEUTRAL) emoji after the retention period ends.
[0105] After outputting non-verbal interactive emoticons, the device applies minimum hold time and cooldown time constraints. The minimum hold time can be set according to the emoticon type, such as 1500 milliseconds for normal emotions, 2500 milliseconds for soothing, and 3000 milliseconds for warnings, to prevent short clips from causing animation flickering. The cooldown time can be set as the shortest interval between the same low-priority emoticon exiting and re-entering, such as 2000 milliseconds, to suppress repeated triggering. During the minimum hold time, new emoticon data with a priority no higher than the currently output emoticon is added to the processing queue without switching, while safety warning emoticon data with a priority higher than the currently output emoticon can preempt the switching. After the minimum hold time ends, the emoticon is switched only if the difference between the target score of the new emoticon data and the target score of the currently output emoticon reaches a preset hysteresis difference (e.g., 0.10).
[0106] In this embodiment, before the subtitle segments corresponding to a single subtitle are fully received, correctable temporary emoticon data is generated and sent based on the received subtitle segments, enabling the device to output non-verbal interactive emoticons first. After the subtitle segments are received, the subtitle segments are concatenated into a complete subtitle, and emoticon matching is performed on the complete subtitle to obtain full-sentence emoticon data, which is used to update or maintain the non-verbal interactive emoticons already displayed on the device. Thus, even if temporary misjudgments occur during the streaming stage due to incomplete semantics of some subtitle segments, they can be corrected after the complete subtitle is determined, ensuring that the non-verbal interactive emoticons output by the device are consistent with the complete semantics, avoiding semantic mismatch caused by judging emoticons based only on some subtitle segments. At the same time, the device calls local animation resources to output non-verbal interactive emoticons, eliminating the need to download emoticon images or videos for each subtitle, thus reducing transmission and decoding overhead.
[0107] In one embodiment, a method for linking subtitles and emoticons includes: Step (a1): Receive the streamed subtitle fragments. Each subtitle fragment carries a message identifier, fragment sequence number, and total number of fragments.
[0108] Step (a2): Before the subtitle segment corresponding to a single subtitle is fully received, the continuous and uninterrupted portion of the subtitle segment starting from the starting subtitle segment is obtained as the text to be processed.
[0109] Step (a3): When the text to be processed matches the emoji keyword, perform semantic parsing on the subtitle segment to obtain the first candidate emoji set.
[0110] Step (a4) identifies negative language structures in the subtitle segments and, based on the negative language structures, performs negative pattern correction on the scoring parameters of at least some candidate expressions in the first candidate expression set.
[0111] Step (a5): Based on the scoring parameters after negative pattern correction, perform semantic confidence scoring on each candidate expression to obtain the target score for each candidate expression.
[0112] Step (a6): Determine the correctable temporary facial expression data based on the target score.
[0113] Step (a7) sends temporary emoticon data to the device to instruct the device to call local animation resources and output non-verbal interactive emoticons corresponding to the temporary emoticon data.
[0114] Step (a8): Establish the corresponding segment buffer according to the message identifier, and store the arriving subtitle segments into the segment buffer according to the segment sequence number.
[0115] Step (a9): When the number of valid segment numbers received in the segment buffer is equal to the total number of segments, and all valid segment numbers are within the range of the total number of segments, the subtitle segments are spliced into a complete subtitle in ascending order of segment number.
[0116] Step (a10) involves semantic parsing of the complete subtitles to obtain the second set of candidate expressions.
[0117] Step (a11) identifies negative language structures in the complete subtitle. Negative language structures include at least one of negation, double negation, and contrast structures.
[0118] Step (a12): When a negation structure is detected, the keyword base score of the candidate emoji matched by the negation structure is set to zero.
[0119] Step (a13): When a double negation structure is detected, restore the keyword base score of the candidate expression matched by the double negation structure before it is set to zero.
[0120] Step (a14): When a transition structure is detected, the context coefficient of the candidate emoji matched by the first half of the text of the transition structure is reduced, and the context coefficient of the candidate emoji matched by the second half of the text of the transition structure is increased.
[0121] Step (a15): Determine the target score of the candidate emoji based on the keyword base score, integrity coefficient, context coefficient, and source coefficient of any candidate emoji in the second candidate emoji set.
[0122] Step (a16): When the target scores of multiple candidate expressions all reach the preset score threshold, compare the priorities of each candidate expression.
[0123] Step (a17): When multiple candidate expressions have the same priority, compare the target scores of each candidate expression.
[0124] Step (a18): When the target scores of multiple candidate expressions all reach the preset score threshold, compare the priorities of each candidate expression.
[0125] Step (a19): When multiple candidate expressions have the same priority, compare the target scores of each candidate expression.
[0126] Step (a20): Determine the full sentence expression data based on the target score.
[0127] Step (a21): When the full-sentence emoji data and the temporary emoji data are inconsistent, send the full-sentence emoji data carrying a second version number to the device to instruct the device to update the non-verbal interaction emojis by calling local animation resources based on the full-sentence emoji data carrying the second version number. The second version number is greater than the first version number.
[0128] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.
[0129] Based on the same inventive concept, this application also provides a streaming subtitle and emoji linkage device for implementing the above-described streaming subtitle and emoji linkage method. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more streaming subtitle and emoji linkage device embodiments provided below can be found in the limitations of the streaming subtitle and emoji linkage method described above, and will not be repeated here.
[0130] In some embodiments, such as Figure 3 As shown, a streaming subtitle and emoticon linkage device is provided, including a segment receiving module 302, a subtitle splicing module 304, a temporary emoticon generation module 306, a full-sentence emoticon generation module 310, and an emoticon data distribution module 308, wherein: The segment receiving module 302 is used to receive subtitle segments output in a streaming manner; The subtitle splicing module 304 is used to splice the subtitle segments corresponding to a single subtitle into a complete subtitle when the subtitle segments corresponding to a single subtitle are received. The temporary emoticon generation module 306 is used to generate correctable temporary emoticon data based on the received subtitle segments when the subtitle segments corresponding to a single subtitle have not been fully received. The expression data sending module 308 is used to send temporary expression data to the device to instruct the device to call local animation resources and output non-verbal interactive expressions corresponding to the temporary expression data; The full-sentence expression generation module 310 is used to match expressions in complete subtitles to obtain full-sentence expression data; The emoticon data delivery module 308 is also used to send full-sentence emoticon data to the device to instruct the device to call local animation resources to update or maintain non-verbal interaction emoticons.
[0131] In this embodiment, correctable temporary emoticon data is generated and sent based on the received subtitle segments, enabling the terminal to output non-verbal interactive emoticons first. After the subtitle segments are received, each subtitle segment is spliced into a complete subtitle, and emoticon matching is performed on the complete subtitle to obtain full-sentence emoticon data, which is used to update or maintain the non-verbal interactive emoticons already displayed by the terminal. Thus, even if temporary misjudgments occur during the streaming stage due to incomplete semantics of some subtitle segments, they can be corrected after the complete subtitle is determined, ensuring that the non-verbal interactive emoticons output by the device are consistent with the complete semantics, avoiding semantic mismatch caused by judging emoticons based only on some subtitle segments. At the same time, the device calls local animation resources to output non-verbal interactive emoticons, eliminating the need to download emoticon images or videos for each subtitle, thus reducing transmission and decoding overhead.
[0132] In some embodiments, the temporary emoji generation module 306 is configured to: when an emoji keyword is matched based on a subtitle segment and the context corresponding to the keyword meets preset conditions, perform semantic parsing on the subtitle segment to obtain a first candidate emoji set; identify negative language structures in the subtitle segment, and perform negative pattern correction on the scoring parameters of at least some candidate emojis in the first candidate emoji set based on the negative language structures; perform semantic confidence scoring on each candidate emoji according to the scoring parameters after negative pattern correction to obtain a target score for each candidate emoji; and determine correctable temporary emoji data based on the target scores.
[0133] In some embodiments, the temporary emoji generation module 306 is further configured to: obtain a continuous and uninterrupted portion of the subtitle segment starting from the starting subtitle segment as the text to be processed; and match emoji keywords based on the text to be processed.
[0134] In some embodiments, the subtitle splicing module 304 is further configured to: establish a corresponding segment buffer according to the message identifier, store the arriving subtitle segments in the segment buffer according to the segment sequence number; when the number of valid segment sequence numbers received in the segment buffer is equal to the total number of segments, and each valid segment sequence number is within the range of 1 to the total number of segments, splice each subtitle segment into a complete subtitle in ascending order of segment sequence number; wherein, when a subtitle segment with the same message identifier and the same segment sequence number as a stored subtitle segment is received, the subtitle segment that arrives later is ignored; when the segment sequence number of a subtitle segment is less than 1 or greater than the total number of segments, the subtitle segment is discarded.
[0135] In some embodiments, the full-sentence expression generation module 310 is further configured to: perform semantic parsing on the complete subtitles to obtain a second candidate expression set; identify negative language structures in the complete subtitles, and based on the negative language structures, perform negative pattern correction on the scoring parameters of at least some candidate expressions in the second candidate expression set; perform semantic confidence scoring on each candidate expression according to the negative pattern corrected scoring parameters to obtain a target score for each candidate expression; and determine the full-sentence expression data based on the target scores.
[0136] In some embodiments, the negative language structure includes at least one of a negation structure, a double negation structure, and a contrast structure; the full-sentence expression generation module 310 is further configured to: perform negative pattern correction on the scoring parameters of at least some candidate expressions in the second candidate expression set based on the negative language structure, including: when a negation structure is detected, setting the keyword base score of the candidate expression matched by the negation structure to zero; when a double negation structure is detected, restoring the keyword base score of the candidate expression matched by the double negation structure before it was set to zero; when a contrast structure is detected, reducing the context coefficient of the candidate expression matched by the first half of the text of the contrast structure and increasing the context coefficient of the candidate expression matched by the second half of the text of the contrast structure.
[0137] In some embodiments, the full-sentence emoji generation module 310 is further configured to: determine the target score of a candidate emoji based on the keyword base score, integrity coefficient, context coefficient, and source coefficient of any candidate emoji in the second candidate emoji set.
[0138] In some embodiments, the full-sentence expression generation module 310 is further configured to: compare the priorities of each candidate expression when the target scores of multiple candidate expressions all reach a preset score threshold; compare the target scores of each candidate expression when the priorities of multiple candidate expressions are the same; and determine the full-sentence expression data based on the target scores.
[0139] In some embodiments, the expression data delivery module 308 is further configured to: when the full-sentence expression data is inconsistent with the temporary expression data, send full-sentence expression data carrying a second version number to the device to instruct the device to call local animation resources to update the non-verbal interaction expression based on the full-sentence expression data carrying the second version number; wherein, the second version number is greater than the first version number, and the device only accepts expression data whose version number is greater than the version number carried by the currently received expression data.
[0140] In some embodiments, the expression data delivery module 308 is further configured to: within the minimum retention time corresponding to a non-verbal interaction expression, when the device receives new expression data, if the priority of the new expression data is not higher than the priority of the current output expression, instruct the device to add the new expression data to the processing queue and not perform expression switching; if the new expression data is a security warning expression data with a higher priority than the current output expression, instruct the device to switch the current output expression to the new expression data; after the minimum retention time ends, when the difference between the target score of the new expression data and the target score of the current output expression reaches a preset hysteresis difference, instruct the device to switch the current output expression to the new expression data.
[0141] Each module in the aforementioned streaming subtitle and emoticon linkage device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0142] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a method for linking streaming subtitles and emoticons.
[0143] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for linking streaming subtitles and emoticons. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0144] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0145] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of various method embodiments.
[0146] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the streaming caption and emoji linkage method in various embodiments.
[0147] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the streaming caption and emoticon linkage method in various embodiments.
[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for linking streaming subtitles and emoticons, characterized in that, The method includes: Receive subtitle segments from streaming output; Before the subtitle segments corresponding to a single subtitle have been fully received, correctable temporary emoticon data is generated based on the received subtitle segments. Send the temporary emoticon data to the device to instruct the device to call local animation resources and output non-verbal interactive emoticons corresponding to the temporary emoticon data; When the subtitle segment corresponding to the single subtitle is received, the subtitle segments corresponding to the single subtitle are spliced together into a complete subtitle; Perform facial expression matching on the complete subtitle to obtain full-sentence facial expression data; The full-sentence emoticon data is sent to the device to instruct the device to call local animation resources to update or maintain the non-verbal interaction emoticons.
2. The method according to claim 1, characterized in that, The process of generating correctable temporary facial expression data based on the received subtitle segments includes: When an emoji keyword is matched based on the subtitle segment, semantic parsing is performed on the subtitle segment to obtain a first candidate emoji set; Identify negative language structures in the subtitle segments, and based on the negative language structures, perform negative pattern correction on the scoring parameters of at least some candidate expressions in the first candidate expression set; Based on the scoring parameters after negative pattern correction, semantic confidence scores are performed on each candidate expression to obtain the target score for each candidate expression; The correctable temporary facial expression data is determined based on the target score.
3. The method according to claim 2, characterized in that, The matching of emoji keywords based on the subtitle segments includes: The continuous, uninterrupted portion of the subtitle segment starting from the starting subtitle segment is selected as the text to be processed. Based on the text to be processed, emoji keywords were matched.
4. The method according to claim 1, characterized in that, The subtitle fragment carries a message identifier, fragment number, and total number of fragments; the step of concatenating the subtitle fragments corresponding to a single subtitle into a complete subtitle upon completion of receiving the subtitle fragments corresponding to that single subtitle includes: Establish a corresponding segment buffer based on the message identifier, and store the arriving subtitle segments into the segment buffer according to the segment sequence number; When the number of valid segment numbers received in the segment buffer is equal to the total number of segments, and all valid segment numbers are within the range of the total number of segments, the subtitle segments are spliced together into the complete subtitle in ascending order of the segment numbers.
5. The method according to claim 1, characterized in that, The process of performing facial expression matching on the complete subtitles to obtain full-sentence facial expression data includes: Semantic parsing is performed on the complete subtitles to obtain a second set of candidate expressions; Identify negative language structures in the complete subtitles, and based on the negative language structures, perform negative pattern correction on the scoring parameters of at least some candidate expressions in the second candidate expression set; Based on the scoring parameters after negative pattern correction, semantic confidence scores are performed on each candidate expression to obtain the target score for each candidate expression; Based on the target score, determine the full sentence expression data.
6. The method according to claim 5, characterized in that, The negative language structure includes at least one of the following: negation structure, double negation structure, and contrast structure; The step of performing negative pattern correction on the rating parameters of at least some candidate expressions in the second candidate expression set based on the negative language structure includes: When the negation structure is detected, the keyword base score of the candidate emoji matched by the negation structure is set to zero; When the double negation structure is detected, the keyword base score before the candidate expression matched by the double negation structure was set to zero is restored; When the transition structure is detected, the context coefficient of the candidate emoji matched with the first half of the text of the transition structure is reduced, and the context coefficient of the candidate emoji matched with the second half of the text of the transition structure is increased.
7. The method according to claim 5, characterized in that, The step of scoring each candidate expression based on the negative pattern-corrected scoring parameters to obtain a target score for each candidate expression includes: The target score of a candidate emoji is determined based on its keyword base score, integrity coefficient, context coefficient, and source coefficient.
8. The method according to claim 5, characterized in that, The process of determining the full-sentence expression data based on the target score includes: When the target scores of multiple candidate expressions all reach a preset score threshold, the priorities of each candidate expression are compared. When multiple candidate expressions have the same priority, compare the target scores of each candidate expression; Based on the target score, the full-sentence expression data is determined.
9. The method according to claim 1, characterized in that, The temporary emoticon data carries a first version number, and the full-sentence emoticon data carries a second version number; sending the full-sentence emoticon data to the device includes: When the full-sentence emoji data is inconsistent with the temporary emoji data, the full-sentence emoji data carrying the second version number is sent to the device to instruct the device to update the non-verbal interaction emoji by calling local animation resources based on the full-sentence emoji data carrying the second version number. The second version number is greater than the first version number.
10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: During the minimum retention time corresponding to the non-verbal interaction emoticon, when the device receives new emoticon data, if the priority of the new emoticon data is not higher than the priority of the currently output emoticon, the device is instructed to add the new emoticon data to the processing queue and not perform emoticon switching. After the minimum hold time ends, when the difference between the target score of the new expression data and the target score of the current output expression reaches a preset hysteresis difference, the device is instructed to switch the current output expression to the new expression data.
