A multimodal feedback-driven method, device, and medium based on emotion perception

Abstract

This invention discloses a multimodal feedback-driven method, device, and medium based on emotion perception, relating to the field of artificial intelligence technology. The method includes: responding to acquired user multimodal data, performing emotion inference to generate an emotion type; using the emotion type as an index, querying a preset animation mapping table, voice style mapping table, and prompt word prefix mapping table in parallel to generate corresponding animation instructions, voice style parameters, and prompt word prefixes, respectively; executing animation output based on the animation instructions, generating response text based on the prompt word prefixes, and synthesizing speech based on the response text and the voice style parameters. This invention eliminates the separation and mismatch between animation output, voice style, and dialogue content by driving them with the same emotional state, achieving synchronous linkage of visual, auditory, and linguistic outputs, and ensuring high consistency in multimodal feedback.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal feedback-driven method, device, and medium based on emotion perception. Background Technology

[0002] In existing technologies, smart terminals typically employ multimodal feedback methods, including visual, auditory, and verbal feedback, to respond to user emotions during emotion-aware interactions. However, in most solutions, animation, voice style, and dialogue content operate independently, lacking a unified driving mechanism. For example, when a user's sadness is detected, the voice module might output comforting statements, but the animation module might still play a generic smiling emoji, and the voice tone might not match the emotion, resulting in a noticeable disconnect between visual, auditory, and verbal responses and a jarring user experience.

[0003] The root cause of this problem is that existing technologies typically rely on large language models to generate all outputs simultaneously, and the instability of large model outputs further exacerbates the mismatch among the three. Summary of the Invention

[0004] This invention provides a multimodal feedback-driven method, device, and medium based on emotion perception. The technical problem it aims to solve is: how to provide an effective solution that can drive visual, auditory, and linguistic outputs with a unified emotional state, ensuring high consistency and synchronous linkage of multimodal feedback.

[0005] In a first aspect, embodiments of the present invention provide a multimodal feedback-driven method based on emotion perception, comprising: In response to the acquired user multimodal data, emotion inference is performed on the multimodal data to generate emotion types; Using the emotion type as an index, the preset animation mapping table, voice style mapping table, and prompt word prefix mapping table are queried in parallel to generate the animation command, voice style parameter, and prompt word prefix corresponding to the emotion type, respectively. Animation output is executed based on the animation instructions, response text is generated based on the prompt word prefix, and speech is synthesized based on the response text and the speech style parameters.

[0006] Optionally, the step of executing animation output based on the animation instruction includes: The animation instructions are encapsulated into UART instruction frames; The UART instruction frame is sent to the animation execution module via the UART bus, wherein the sending operation of the UART instruction frame is performed before the operation of generating reply text based on the prompt word prefix and synthesizing speech based on the reply text and the speech style parameters, and the UART instruction frame is given the highest priority in the sending queue; After receiving the UART instruction frame, the animation execution module parses the UART instruction frame to obtain the animation instruction, and starts playing the animation sequence corresponding to the animation instruction.

[0007] Optionally, generating the reply text based on the prompt word prefix includes: Retrieve the dialogue history in the current dialogue context and count the total number of rounds M in the dialogue history; If M is greater than the preset maximum number of rounds to retain K, then the dialogue statements of the most recent K rounds are selected as the valid dialogue history; if M is less than or equal to K, then all M rounds of dialogue statements are selected as the valid dialogue history. The prompt word prefix is ​​placed before the effective dialogue history and concatenated sequentially to construct a complete prompt word, wherein the prompt word prefix adopts the form of natural language instructions to specify the sentiment tendency, politeness level and tone intensity of the large language model's response; The complete prompt word is sent to the large language model, and the response text returned by the large language model is received.

[0008] Optionally, the process of synthesizing speech based on the reply text and the speech style parameters includes: The response text is subjected to text regularization and word segmentation to obtain the phoneme sequence and the original duration of each phoneme; The speech rate value in the speech style parameters is mapped to the duration scaling factor of each phoneme, the pitch value in the speech style parameters is mapped to the offset Hertz number of the fundamental frequency curve, and the emotional color label in the speech style parameters is mapped to the set of stress positions and the pause pattern encoding. The phoneme sequence, the duration scaling factor, the offset Hertz number, the accent position set, and the pause pattern encoding are input into the speech synthesis engine, and the speech data in waveform format returned by the speech synthesis engine is received.

[0009] Optionally, the emotion inference also produces a confidence value for the emotion type; the method further includes: Determine whether the confidence value is less than a preset confidence threshold; If the confidence value is less than the preset confidence threshold, the emotion type is forcibly corrected to a neutral state, and the neutral state is used as the emotion type for subsequent queries. If the confidence value is not less than the preset confidence threshold, the emotion type remains unchanged.

[0010] Optionally, the method further includes: When the emotion type is forcibly corrected to the neutral state, the default animation instruction corresponding to the neutral state is queried from the animation mapping table. The default animation instruction is the curiosity expression animation instruction. The animation output is executed based on the default animation instruction, while the query operations of the voice style mapping table and the prompt word prefix mapping table are blocked, so that only visual feedback is output in low confidence scenarios.

[0011] Optionally, the animation mapping table, the voice style mapping table, and the prompt word prefix mapping table are all static JSON configuration files stored on the local terminal; the method further includes: The update package is received via OTA firmware update, and the update package carries a new configuration file for the target mapping table, a format checksum of the new configuration file, and a version number. Verify whether the format check code of the new configuration file matches the preset JSON format template to obtain the first verification result; Determine whether the version number of the new configuration file is higher than the version number of the corresponding mapping table currently stored locally, and obtain a second determination result; If the first verification result is a match and the second judgment result is higher, then the target mapping table stored locally is atomically replaced with the new configuration file, and the mapping data already loaded into memory remains unchanged during the replacement process, without interrupting the currently ongoing emotion feedback process; If the first verification result is a mismatch or the second judgment result is not higher than, then the update package is discarded and the original mapping table is retained.

[0012] Optionally, the animation mapping table, the voice style mapping table, and the prompt word prefix mapping table are stored independently in different configuration files, and each has its own version number; the method further includes: When the first update instruction for the animation mapping table is received, only the configuration file of the animation mapping table is replaced, while the version number and content of the voice style mapping table and the prompt word prefix mapping table remain unchanged; When a second update instruction is received for the speech style mapping table, only the configuration file of the speech style mapping table is replaced, while the version number and content of the animation mapping table and the prompt word prefix mapping table remain unchanged; When a third update instruction is received for the prompt word prefix mapping table, only the configuration file of the prompt word prefix mapping table is replaced, while the version number and content of the animation mapping table and the voice style mapping table remain unchanged; When performing parallel queries, the animation instructions, the voice style parameters, and the prompt word prefixes are read independently from their respective configuration files. Failure to read any configuration file will only result in the missing output of the corresponding channel and will not affect the normal output of other channels. The method further includes: Acquire multiple consecutive frames of images captured by the camera and calculate the frame difference between adjacent frames; When the frame difference value is continuously lower than the preset static threshold, the inference frequency of the emotion inference of the multimodal data is reduced from the preset normal inference frequency to the preset power-saving inference frequency. When the frame difference exceeds a preset dynamic threshold, the inference frequency is restored to the normal inference frequency.

[0013] Secondly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Thirdly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0015] This invention provides a multimodal feedback-driven method, device, and medium based on emotion perception. The method includes: responding to acquired user multimodal data, performing emotion inference on the multimodal data to generate an emotion type; using the emotion type as an index, querying a preset animation mapping table, voice style mapping table, and prompt word prefix mapping table in parallel to generate animation instructions, voice style parameters, and prompt word prefixes corresponding to the emotion type; executing animation output based on the animation instructions; generating response text based on the prompt word prefixes; and synthesizing speech based on the response text and the voice style parameters. This invention uses the emotion type generated by emotion inference as a unique index and queries the animation mapping table, voice style mapping table, and prompt word prefix mapping table in parallel to obtain animation instructions, voice style parameters, and prompt word prefixes, ensuring that visual output, voice style, and dialogue content are all driven by the same emotional state, eliminating the separation and mismatch between the three. The animation execution, response text generation, and speech synthesis are processed in parallel. Based on a unified emotion index, the three-layer output is synchronized and linked, ensuring that the multimodal feedback is highly consistent in emotion. This avoids the incongruity of contradictory facial expressions and language in traditional solutions, thus providing an effective solution that drives visual, auditory, and linguistic output with a unified emotional state. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a multimodal feedback-driven method based on emotion perception provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] Please see Figure 1 This invention provides a multimodal feedback-driven method based on emotion perception. By driving the three-layer output of visual animation, speech style, and dialogue content with a unified emotional state, it maintains emotional consistency among the three, avoiding the instability and high computational cost of relying on large language models to directly generate multimodal output. This method enables synchronized linkage of animation, speech, and text, eliminating the disjointed feeling of facial expressions and speech, and providing rapid response on low-cost terminals. Specifically, the method includes the following steps: S1, in response to the acquired user multimodal data, performs emotion inference on the multimodal data and generates an emotion type.

[0024] In specific implementation, firstly, the terminal device acquires the user's facial image through a camera, and the user's voice signal through a microphone, or optionally, the user's touch signal through a touch sensor. These data are collectively referred to as user multimodal data. It should be noted that the inclusion of voice and touch signals in user multimodal data is merely an example; those skilled in the art can flexibly adjust it according to actual needs, and this invention is not specifically limited. The terminal device inputs the acquired multimodal data into a pre-trained lightweight convolutional neural network model. A convolutional neural network (CNN) is a deep learning model specifically designed for processing grid-like data (such as images). Its core operation is that the convolutional kernel slides across the input data to extract local features. A lightweight convolutional neural network refers to a CNN structure with a small number of parameters and computational cost, such as the MiniXception architecture. Its characteristic is that it can run in real time on a microcontroller or low-power embedded processor without requiring a high-performance graphics processor.

[0025] Furthermore, the model performs emotion inference on the multimodal data: for image input, the model extracts facial key point features and expression features; for voice input, the model extracts tone and energy features; for touch input, the model extracts operation force and frequency features.

[0026] Furthermore, the aforementioned features are fused through the fully connected layer of the model, ultimately outputting an emotion type. The emotion type is selected from a predefined set of emotions, such as happiness, sadness, anger, surprise, and neutrality. As a specific implementation, the MiniXception architecture can output seven emotion categories, including happiness, sadness, anger, fear, disgust, surprise, and neutrality.

[0027] Furthermore, the model also outputs a confidence score, representing the reliability of the sentiment inference result, ranging from 0 to 1. The confidence score is typically given by the maximum value in the probability distribution output by the model's last Softmax layer; a higher value indicates that the model is more confident in the judgment.

[0028] For example, the training method of the lightweight convolutional neural network model includes: supervising the MiniXception architecture on a publicly available facial expression dataset (such as FER2013 or AffectNet dataset), using the cross-entropy loss function as the optimization objective, employing the Adam optimizer, setting the initial learning rate to 0.001, the batch size to 32, and the training epochs to 100, and using data augmentation strategies (including random horizontal flipping, random rotation, and random pruning) during training to improve the model's generalization ability. After training, the model is deployed to the terminal device through quantization compression (quantized from FP32 to INT8).

[0029] It is important to emphasize that all data related to user identity in this application, including but not limited to facial feature vectors, facial key point data, and any biometric information that can be used to identify users, are stored only in the local non-volatile memory of the terminal device. This storage area does not include any remote storage based on the cloud or network servers. This application does not upload the aforementioned user identity data to the cloud or any network server; it is used only within the local device for emotion reasoning and feedback-driven processing. The dialogue text content can be selectively sent to a cloud-based large language model service as needed, but user identity feature data remains localized at all times, thereby achieving on-device privacy protection, which is significantly different from solutions that rely on the cloud to process sensitive user data.

[0030] S2, using the emotion type as an index, query the preset animation mapping table, voice style mapping table and prompt word prefix mapping table in parallel, and generate the animation command, voice style parameter and prompt word prefix corresponding to the emotion type respectively.

[0031] In practice, the terminal device uses the emotion type as an index to query three preset mapping tables in parallel: an animation mapping table, a voice style mapping table, and a cue word prefix mapping table. Furthermore, these three mapping tables are stored in the terminal device's local non-volatile memory as static JSON configuration files. JSON stands for JavaScript Object Notation, a lightweight data-interchange format that uses key-value pairs and array structures to represent data, making it easy for machines to parse and generate, and also easy for humans to read and edit.

[0032] For example, an entry in the animation mapping table can be represented as: {"emotion": "happy", "animation_id": 5}. The value of the "animation_id" field is the animation instruction. The animation mapping table records the correspondence between each emotion type and the animation instruction. An animation instruction is a predefined identifier used to index specific animation sequence files in the terminal device's animation library. Further, the speech style mapping table records the correspondence between each emotion type and speech style parameters, including speech rate, pitch, and emotional color label. The speech rate is a numerical value; for example, 1.2 represents 20% faster than normal speech, and 0.8 represents 20% slower. The pitch value can be "high," "medium," "low," or a specific frequency offset. The emotional color label is an enumerated value, such as "happy," "sad," "angry," "gentle," etc., used to guide the downstream speech synthesis engine in adjusting prosodic features. Furthermore, the prompt prefix mapping table records the correspondence between each emotion type and the prompt prefix. The prompt prefix is ​​a natural language string; for example, the prompt prefix for sadness is "You are a gentle friend, please reply in a comforting tone." This string will be directly concatenated into the input prompts of the large language model.

[0033] Furthermore, the terminal device uses the emotion type as an index to simultaneously query the three mapping tables mentioned above, obtaining the animation command, voice style parameters, and prompt word prefix corresponding to that emotion type. Since the three query operations are executed in parallel and based on the same emotion type, the obtained animation command, voice style parameters, and prompt word prefix are naturally consistent in emotion. Parallel queries can be implemented through multithreading or asynchronous input / output. In embedded systems, they can also be executed sequentially but logically considered parallel because there is no data dependency between the three queries.

[0034] S3, execute animation output based on the animation instruction, generate reply text based on the prompt word prefix, and synthesize speech based on the reply text and the speech style parameters.

[0035] In practice, the terminal device executes animation output based on the animation instructions. Specifically, the terminal device sends the animation instructions to the animation execution module, which reads the corresponding frame sequence data from the locally stored animation library according to the animation instructions and drives the display screen to render frame by frame to form a continuous animation effect.

[0036] Further, the terminal device generates a response text based on the prompt word prefix. The terminal device concatenates the prompt word prefix with the user's dialogue history to form a complete prompt word. The dialogue history records the most recent rounds of interaction between the user and the terminal, stored in natural language text format. This complete prompt word is sent to a cloud-deployed Large Language Model (LLM) server via Hypertext Transfer Protocol or Message Queuing Telemetry Protocol, or to a locally loaded LLM process. A Large Language Model (LLM) is a deep learning model based on the Transformer architecture. Through pre-training on massive amounts of text data, it gains the ability to understand natural language instructions and generate coherent, context-sensitive responses. Representative LLM models include the GPT series, ChatGLM, and LLaMA. The terminal device receives the response text returned by the LLM, which is a natural language sentence or paragraph.

[0037] Furthermore, the terminal device inputs the response text and the speech style parameters into the speech synthesis engine. A text-to-speech (TTS) engine is a technology that converts text into speech waveforms. It can use a locally deployed speech synthesis engine or call a cloud-based speech synthesis service through an application programming interface (API). When using a local engine, it typically includes a front-end text analysis module (converting text into phoneme sequences and prosodic tags) and a back-end acoustic model (converting phoneme sequences and prosodic tags into acoustic features, then synthesizing waveforms using a vocoder). The speech style parameters are directly used to control the duration, fundamental frequency, and spectral parameters during the synthesis process. When using a cloud-based speech synthesis service, the terminal device encapsulates the response text and the speech style parameters into an API request. The speech style parameters are mapped to the speech rate, pitch, and emotion style label parameters in the API call. The cloud service returns the synthesized speech waveform data. As a specific implementation, a cloud-based speech service that supports emotion synthesis (such as CosyVoice and other emotion-based TTS services) can be selected, directly mapping the emotion color labels in the speech style parameters to the emotion style labels defined by the service. Whether using a local engine or a cloud service, the final output is waveform-formatted voice data.

[0038] Understandably, the animation output, response text generation, and speech synthesis are executed in parallel. Since all three are driven by the same emotion type, the output animation, speech style, and dialogue content maintain a high degree of emotional consistency.

[0039] Furthermore, in some optional implementations, considering that the terminal device may be offline (e.g., without network connection or unavailable cloud service), a degradation processing mechanism is provided to ensure the continuity of user experience. Specifically, this includes: the terminal device detecting the current network connection status, and automatically switching to local feedback mode when it determines that it cannot connect to the cloud-based speech synthesis service. In local feedback mode, the step of generating reply text based on the prompt word prefix can be skipped, and instead, a pre-recorded audio file (e.g., .wav format) corresponding to the current emotion type is selected from a pre-set local audio library for playback. The pre-recorded audio file is pre-recorded according to each emotion type; for example, a comforting tone corresponds to a sad emotion, and a cheerful tone corresponds to a happy emotion. Alternatively, the terminal device can enable a locally deployed lightweight speech synthesis model (e.g., a lightweight TTS model that can run on embedded devices, such as Tacotron2) to directly synthesize speech locally based on the reply text and speech style parameters. Through this degradation mechanism, even in scenarios without cloud services, the terminal device can still output auditory feedback matching the emotion, maintaining the integrity of multimodal interaction.

[0040] This embodiment achieves emotional consistency in visual animation, voice style, and dialogue content by using a unified emotion type as an index to query three independent pre-set mapping tables in parallel. Compared to existing technologies where animation, voice, and text make independent decisions or rely on large language models for simultaneous generation, leading to mismatches, this embodiment eliminates the incongruity of disjointed facial expressions and language. Furthermore, since the three mapping tables are statically pre-configured, no complex online inference is required, making it suitable for low-cost terminal devices. The parallel execution of animation output, text generation, and speech synthesis improves response efficiency while ensuring emotional consistency.

[0041] In some preferred embodiments, the step of executing animation output based on the animation instruction includes: encapsulating the animation instruction into a UART instruction frame; sending the UART instruction frame to the animation execution module via the UART bus, wherein the sending operation of the UART instruction frame is performed before the operation of generating reply text based on the prompt word prefix and the operation of synthesizing speech based on the reply text and the speech style parameters, and the UART instruction frame is given the highest priority in the sending queue; after receiving the UART instruction frame, the animation execution module parses the UART instruction frame to obtain the animation instruction, and starts playing the animation sequence corresponding to the animation instruction.

[0042] In practice, this embodiment prioritizes animation output over speech output and utilizes hardware-level low-latency communication, enabling users to perceive visual feedback first and thus enhancing the immediacy of emotional connection. This embodiment can compress animation response time to the millisecond level without relying on the response speed of a large language model.

[0043] Specifically, after the terminal device queries the three mapping tables in parallel to obtain the animation instructions, voice style parameters, and prompt word prefixes, the terminal device executes the animation output process as follows.

[0044] First, the terminal device encapsulates the animation instructions into a UART instruction frame. UART stands for Universal Asynchronous Receiver / Transmitter, a widely used serial communication interface standard. UART communication requires only two signal lines (transmit and receive lines) and does not require a clock synchronization signal, thus resulting in extremely low hardware overhead. The animation instruction itself is a preset numerical identifier; for example, the animation identifier 0x01 represents a happy animation, and 0x02 represents a sad animation. The terminal device combines the animation instruction with a preset two-byte frame header (0xAA 0x55), a length byte (1 byte), a command word (1 byte), a payload (0 to 250 bytes), and a CRC8 checksum byte (1 byte) to form an instruction frame conforming to the UART protocol. The payload carries the animation instruction, the length byte indicates the number of bytes in the payload, and the CRC8 checksum byte is used to verify the correctness of the frame transmission. UART communication uses full-duplex mode, with a baud rate that can be set to 921600bps and big-endian byte order. This instruction frame does not include a separate frame trailer, nor does it include a retransmission mechanism requiring receiver acknowledgment, to ensure minimal transmission latency. In contrast, reliable transmission protocols such as Transmission Control Protocol (TCP) require a three-way handshake and acknowledgment retransmission, resulting in latency of tens or even hundreds of milliseconds.

[0045] Furthermore, the terminal device sends the UART instruction frames to the animation execution module via the UART bus. The animation execution module is a dedicated hardware module responsible for driving the display device to play animations, such as a microcontroller with a screen. This module internally maintains a UART receive buffer and continuously monitors data on the UART bus.

[0046] The key to this embodiment is that the transmission of the UART command frame must be performed before the generation of the response text based on the prompt word prefix and the synthesis of speech based on the response text and the speech style parameters. In other words, after completing the lookup of the three mapping tables, the main control unit of the terminal device immediately calls the UART driver function to write the animation command frame into the transmission buffer, and then begins to execute operations such as prompt word concatenation, calling the large language model, and speech synthesis.

[0047] Furthermore, the UART command frame is assigned the highest priority in the terminal's transmit queue. The terminal device's operating system or bare-metal program typically maintains a UART transmit queue, processing data according to priority when multiple tasks simultaneously write data to the UART. Setting the animation command frame to the highest priority means that even if the terminal device currently has other data (such as network heartbeat packets or log reporting data) waiting to be transmitted, the animation command frame will be placed at the front of the queue, preferentially occupying bus resources and thus minimizing transmission latency.

[0048] Furthermore, after receiving the UART instruction frame, the animation execution module parses the UART instruction frame. The parsing process includes: detecting the double-byte frame header (0xAA 0x55), reading the length byte, extracting the animation instruction from the command word and payload, and verifying the CRC8 checksum. Upon successful parsing, the animation instruction is extracted. The animation execution module searches for the corresponding animation sequence data in the locally stored animation library based on the animation instruction, and then immediately starts playing the animation sequence, refreshing the screen frame by frame starting from the first frame, typically at a frame rate of 30 frames per second. During or after animation playback, the animation execution module can send a playback status confirmation message to the main control module via the UART's reverse channel, achieving closed-loop control.

[0049] Furthermore, since the sending of animation command frames is prioritized and does not depend on the response time of the large language model (large language model calls typically take hundreds of milliseconds to several seconds), animation playback can usually begin within milliseconds after the UART command is sent. Real-world testing shows that the time from the completion of emotion recognition to the display of the first animation frame is less than 10 milliseconds.

[0050] It should be noted that the aforementioned main control module and animation execution module can be implemented using two independent processors, with the first processor handling emotion reasoning and mapping table lookup, and the second processor acting as a microcontroller to drive the display screen; alternatively, a single-chip solution can be used, where the tasks are divided into logically independent main control module and animation execution module within the same processor, communicating via an internal bus or shared memory, also following the UART instruction frame format. Furthermore, the UART instruction frame format and baud rate used in this embodiment are merely examples; in actual applications, the parameters can be flexibly adjusted according to specific hardware, as long as reliable transmission of animation instructions is achieved.

[0051] This embodiment encapsulates animation commands into UART command frames with a two-byte header, length field, and CRC8 checksum, assigning them the highest transmission priority. Furthermore, it forces the animation command transmission operation to execute before text generation and speech synthesis, enabling animation output to start playing within a very short time after emotion recognition (less than 10 milliseconds in actual testing). Moreover, users will first observe the changes in the terminal device's facial expressions before hearing the voice response; this vision-first timing design significantly enhances the user's immediate emotional connection to the device. Furthermore, full-duplex UART communication and CRC8 checksum ensure the reliability of command transmission and the possibility of feedback loop closure, while the high baud rate further reduces transmission latency. Compared to traditional solutions, this embodiment avoids latency caused by animation lag or complex protocol stacks, making it suitable for low-cost hardware implementation.

[0052] In some preferred embodiments, generating the response text based on the prompt word prefix includes: obtaining the dialogue history in the current dialogue context and counting the total number of rounds M in the dialogue history; if M is greater than the preset maximum number of rounds to retain K, then selecting the dialogue statements of the most recent K rounds as the valid dialogue history; if M is less than or equal to K, then selecting all M rounds of dialogue statements as the valid dialogue history; sequentially concatenating the prompt word prefix before the valid dialogue history to construct a complete prompt word, wherein the prompt word prefix adopts the form of natural language instructions to specify the sentiment tendency, politeness level and tone intensity of the large language model's response; sending the complete prompt word to the large language model and receiving the response text returned by the large language model.

[0053] This embodiment ensures that the emotional style of the response text generated by the large language model matches the current emotion type while maintaining the continuity of the dialogue context, thus avoiding the model ignoring the context or style inconsistency. This allows for adaptive pruning of excessively long dialogue history and precise guidance of the model output with natural language instructions.

[0054] In practice, the terminal device first obtains the dialogue history from the current dialogue context. This dialogue history records the most recent rounds of interaction between the user and the terminal device, including statements spoken by the user and previous responses from the terminal device. The dialogue history is typically stored as a list in memory or flash memory, with each record containing a speaker identifier (user or terminal), a timestamp, and text content. The terminal device then counts the total number of rounds M in the dialogue history. A round of dialogue is typically defined as a complete interaction involving one user input and one terminal response, but it can also include only user input (when the terminal has not yet responded). For example, if the dialogue history stores the three most recent user statements and their corresponding three terminal responses, then M = 3; if it only stores user statements without a terminal response, then M is the number of user statements.

[0055] Furthermore, the terminal device determines the valid dialogue history based on the relationship between M and the preset maximum number of rounds to retain, K. K is a configurable positive integer whose value depends on the context window size limit of the large language model and the performance of the terminal device. For example, the context window of mainstream large language models is typically 2048 tokens, while an average dialogue round consumes 50 tokens; therefore, K is usually set between 5 and 10. If M is greater than K, the terminal device selects only the most recent K rounds of dialogue as valid dialogue history, discarding earlier rounds. This discarding is achieved by truncating the first element of the list. This is done to avoid prompts exceeding the input length limit of the large language model and to reduce computational load. If M is less than or equal to K, the terminal device selects all M rounds of dialogue as valid dialogue history.

[0056] Furthermore, the terminal device sequentially concatenates the prompt word prefix before the valid dialogue history to construct a complete prompt word. Concatenation refers to linking strings in sequence to form a longer string. For example, if the valid dialogue history is "User: I'm very unhappy today. Terminal: What's wrong? User: I didn't do well on the exam.", and the prompt word prefix is ​​"Please reply in a comforting tone," then the complete prompt word is "Please reply in a comforting tone\nUser: I'm very unhappy today. Terminal: What's wrong? User: I didn't do well on the exam." Placing the prompt word prefix before the dialogue history allows the large language model to receive the emotional style instruction before understanding the dialogue context. The large language model generates output word by word from left to right when processing input, so instructions that come earlier have higher guiding weight. The prompt word prefix adopts the form of natural language instructions, such as "You are a gentle friend, please reply in a comforting tone," rather than discrete emotion labels (such as "sad"). Using natural language instructions can fully utilize the rich semantic understanding capabilities learned by the large language model during the pre-training stage, enabling the model to more accurately grasp the desired emotional style. The prompt word prefixes are used to specify the sentiment tendency (e.g., positive, negative, neutral), politeness level (e.g., formal, friendly), and tone intensity (e.g., strong, mild) of the large language model's response.

[0057] Further, the terminal device sends the complete prompt word to the large language model. This is typically done by calling the cloud-based large language model's application programming interface (API) via Hypertext Transfer Protocol (HTTP), specifying the model name, temperature parameters (to control randomness), and the maximum number of generated tokens in the request parameters. After the model's inference is complete, the terminal device receives the response text returned by the large language model. Based on the instructions in the complete prompt word and the dialogue history, the large language model generates a natural language response word by word, utilizing its massive internal parameters and self-attention mechanism. Because the prompt word prefix explicitly specifies the emotional style requirements, the generated response text maintains semantic and tonal consistency with the stated emotion type. For example, for a sad emotion, the model might generate "Don't be sad, one exam doesn't mean anything, just try harder next time."

[0058] This embodiment selects the most recent multi-turn dialogues as valid dialogue history through a sliding window, avoiding the problem of excessively long dialogues leading to exceeding the cue word limit, while preserving sufficient contextual information. Placing the cue word prefix before the dialogue history allows the large language model to prioritize receiving sentiment style instructions, thereby generating response text that matches the sentiment type. Furthermore, the cue word prefix in natural language instruction form has the advantages of high readability and ease of modification, allowing for adjustment of the response style without retraining the model. Compared to directly inputting sentiment labels as parameters, this embodiment fully utilizes the natural language understanding capabilities of the large language model, resulting in more natural and nuanced responses.

[0059] In some preferred embodiments, the step of synthesizing speech based on the response text and the speech style parameters includes: performing text regularization and word segmentation on the response text to obtain a phoneme sequence and the original duration of each phoneme; mapping the speech rate value in the speech style parameters to a duration scaling factor for each phoneme, mapping the pitch value in the speech style parameters to the offset Hertz number of the fundamental frequency curve, and mapping the emotional color label in the speech style parameters to a set of stress positions and a pause pattern encoding; inputting the phoneme sequence, the duration scaling factor, the offset Hertz number, the set of stress positions, and the pause pattern encoding into a speech synthesis engine, and receiving the waveform-formatted speech data returned by the speech synthesis engine.

[0060] In its specific implementation, this embodiment achieves richer and more natural emotional expression in synthesized speech by precisely controlling phoneme duration, fundamental frequency curve, and prosodic features in the speech synthesis parameters. This avoids the mechanical feel caused by simple overall speed and pitch changes, thereby generating expressive speech output that highly matches the emotions conveyed. The specific implementation method is as follows.

[0061] First, the terminal device performs text normalization and word segmentation on the reply text. Text normalization is to convert numbers, abbreviations, special symbols, etc. into natural language readings. For example, convert "2024" to "two thousand and twenty-four". Normalization is achieved through a rule base or a statistical model to ensure that the synthesized speech does not read out strange forms such as "two zero two four" or "ten US dollars". Further, word segmentation is to split continuous text into independent word sequences. For Chinese, word segmentation is a necessary step because there are no spaces between Chinese words. Commonly used word segmentation tools include the maximum matching algorithm based on a dictionary or the hidden Markov model based on statistics. After word segmentation, each word is further converted into a phoneme sequence through a phonetic conversion module. A phoneme is the smallest sound unit that constitutes speech. For example, the initials "b", "p", "m" and finals "a", "o", "e" in Chinese pinyin. Each Chinese character corresponds to one or more phonemes. Word segmentation also outputs the original duration of each phoneme, that is, the expected duration of the phoneme in the standard pronunciation (neutral tone, normal speed), usually in milliseconds. The original duration can be obtained by querying from a pre-established phoneme duration database.

[0062] Further, the terminal device maps the speech rate value in the speech style parameters to a duration scaling factor for each phoneme. The speech rate value is a relative value. For example, 0.8 represents slow speed (80% of the normal speed), 1.0 represents normal, and 1.2 represents fast. The duration scaling factor is a coefficient calculated from the speech rate value. For example, the duration scaling factor is equal to 1 divided by the speech rate value. When the speech rate value is 1.2, the duration scaling factor is approximately 0.833, meaning that the playback time of each phoneme is shortened to 83.3% of the original. The terminal device maps the pitch value in the speech style parameters to the offset Hertz number of the fundamental frequency curve. The pitch value is a relative value or an enumerated value. For example, "high pitch" corresponds to a 50-Hertz increase in the fundamental frequency offset, and "low pitch" corresponds to a 50-Hertz decrease. The fundamental frequency curve is the trajectory of the fundamental frequency of vocal cord vibration in the speech signal, which determines the pitch undulation of speech. The fundamental frequency range of human speech is usually between 80 - 200 Hertz for males and 150 - 400 Hertz for females. Further, by offsetting the fundamental frequency curve, intonations such as cheerful (higher fundamental frequency) or低沉 (lower fundamental frequency) can be simulated. The terminal device also maps the emotional color label in the speech style parameters to a set of stress positions and a pause pattern encoding. Different emotional colors correspond to different stress distributions and pause patterns. For example, in a sad mood, there are fewer stresses, usually falling on negative words or negative adjectives, and the pauses are longer, especially at the end of the sentence; in a happy mood, there are more stresses, usually falling on positive adjectives and interjections, and the pauses are shorter and lively. The set of stress positions is a data structure indicating which syllables need to be pronounced with emphasis, and the pause pattern encoding is a sequence indicating where to insert pauses of what length.

[0063] Furthermore, the terminal device inputs the phoneme sequence, the duration scaling factor, the offset Hertz number, the set of stress positions, and the pause pattern encoding into the speech synthesis engine. The speech synthesis engine is a deep learning-based text-to-speech model, such as Tacotron2, FastSpeech, or VITS. The engine's inputs are linguistic features (phoneme sequence) and prosodic parameters (duration, fundamental frequency, stress, pauses), and its output is waveform-formatted speech data (typically pulse code modulation).

[0064] Specifically, the engine's workflow is as follows: the encoder converts the phoneme sequence into a hidden state sequence; the duration predictor adjusts the duration of each phoneme according to the duration scaling factor; the fundamental frequency predictor adjusts the fundamental frequency curve according to the offset Hertz number; the acoustic model combines accent and pause information to generate a Mel spectrum; and the vocoder converts the Mel spectrum into the final waveform. The speech synthesis engine outputs waveform-formatted speech data, which can be directly converted into an analog signal by a digital-to-analog converter and played by a speaker.

[0065] This embodiment achieves emotional speech synthesis that far surpasses simple overall speed and pitch modulation by finely decomposing speech style parameters and mapping speech rate, pitch, and emotional tone to phoneme duration scaling, fundamental frequency offset, and prosodic feature control, respectively. For example, in the case of sadness, extending phoneme duration (duration scaling factor greater than 1), lowering the fundamental frequency (negative offset), and increasing pause length can naturally produce a choked or low tone; in the case of happiness, shortening phoneme duration (duration scaling factor less than 1), raising the fundamental frequency (positive offset), and increasing stress density can produce a light and rising tone. This fine control makes the emotional expression of synthesized speech richer and more natural, significantly enhancing the user's emotional immersion experience.

[0066] In some preferred embodiments, the emotion inference also produces a confidence value for the emotion type; the method further includes: determining whether the confidence value is less than a preset confidence threshold; if the confidence value is less than the preset confidence threshold, forcibly correcting the emotion type to a neutral state, and using the neutral state as the emotion type for subsequent queries; if the confidence value is not less than the preset confidence threshold, keeping the emotion type unchanged.

[0067] In practice, this embodiment avoids outputting inappropriate feedback when the emotion recognition result is unreliable. By forcibly correcting low-confidence emotions to a neutral state, it prevents erroneous animations, voice, or text outputs caused by misrecognition, thereby significantly improving the robustness and interactive security of the system. The specific implementation method is as follows.

[0068] Based on the method described in the above embodiments, the emotion inference also produces a confidence value for the emotion type. This confidence value is output by the Softmax layer of the emotion inference model. The Softmax function converts the raw scores (logits) of the last layer of the model into a probability distribution, where the maximum probability value is the confidence score. For example, if the model outputs [Happy: 0.7, Sad: 0.2, Neutral: 0.1], then the emotion type is Happy, and the confidence score is 0.7. This embodiment adds a decision step on the confidence value before parallel querying the three mapping tables.

[0069] Specifically, the terminal device acquires the confidence value generated by the emotion inference. Then, the terminal device determines whether the confidence value is less than a preset confidence threshold. The preset confidence threshold is a pre-defined value, such as 0.5. Furthermore, this threshold can be dynamically adjusted based on historical recognition accuracy. For example, when low confidence levels occur frequently over a period of time, the threshold can be appropriately lowered to reduce the frequency of neutral regression; conversely, when the accuracy of high-confidence recognition is very high, the threshold can be appropriately increased for more cautious handling.

[0070] Furthermore, if the confidence value is less than the preset confidence threshold, it indicates that the reliability of the emotion inference result is low. In this case, the terminal device forcibly corrects the emotion type to a neutral state and uses the neutral state as the emotion type used for subsequent queries of the animation mapping table, voice style mapping table, and prompt word prefix mapping table. If the confidence value is not less than the preset confidence threshold, it indicates that the emotion inference result is reliable, and the terminal device keeps the emotion type unchanged, that is, directly uses the original emotion type for subsequent queries.

[0071] For example, when a user's face is obscured (e.g., wearing a mask) or insufficient lighting prevents the emotion recognition model from clearly extracting facial features, the probabilities of various emotions output by the model may be very close, such as [happy: 0.35, sad: 0.33, neutral: 0.32], with a confidence level of only 0.35. In this case, the terminal device forces the emotion type to a neutral state. When the neutral state is used as an index to query the three mapping tables, the animation mapping table returns an animation instruction corresponding to a neutral expression (e.g., calm gaze), the voice style mapping table returns neutral tone parameters (speech rate 1.0, medium tone, emotional color label "neutral"), and the prompt word prefix mapping table returns a neutral style prompt word prefix (e.g., "Please answer normally"). This avoids outputting incorrect feedback when the user's actual emotion is unknown (e.g., incorrectly outputting a comforting or happy expression). If the user is actually sad, but the terminal incorrectly outputs a happy expression or cheerful voice, it will lead to serious user experience problems. The confidence level decision effectively prevents this situation.

[0072] This embodiment introduces a confidence-based decision mechanism to uniformly classify low-reliability emotion recognition results into a neutral state, avoiding inappropriate animation, voice, or text feedback due to misidentification. For example, when the user has no expression or the lighting is poor, the terminal will not incorrectly output comfort or congratulations, thus avoiding embarrassment or offense. This mechanism significantly improves the robustness and interactive security of the system, enabling the terminal device to work stably in complex real-world environments without requiring deliberate user cooperation.

[0073] In some preferred embodiments, the method further includes: when the emotion type is forcibly corrected to the neutral state, querying the animation mapping table for a default animation instruction corresponding to the neutral state, wherein the default animation instruction is a curiosity expression animation instruction; executing animation output based on the default animation instruction, while simultaneously blocking the query operations of the voice style mapping table and the prompt word prefix mapping table, thereby outputting only visual feedback in low confidence scenarios.

[0074] In specific implementation, this embodiment further reduces the risk of false feedback when emotion recognition is unreliable by retaining only visual feedback and completely blocking voice and text feedback to respond to the user in the safest way. This avoids any inappropriate voice content due to misrecognition, while conveying the terminal's attention through curious expressions. The specific implementation method is as follows.

[0075] First, the terminal device queries the animation mapping table for the default animation instruction corresponding to the neutral state. The default animation instruction is the curiosity expression animation instruction. The curiosity expression is a neutral, friendly expression, such as a slight tilt of the head, wide eyes, and slightly raised eyebrows, which does not convey strong positive or negative emotions. This expression will not offend or misinterpret the user, while attracting the user's attention and encouraging further interaction.

[0076] Furthermore, the terminal device executes animation output based on the default animation instruction. Simultaneously, the terminal device sets a disable trigger flag. This disable trigger flag is an internal semaphore or flag, implemented in memory as a Boolean variable with an initial value of false. In embedded systems, this can be achieved by defining a global variable, such as `booldisable_flag = false`. When the confidence value is lower than a preset confidence threshold, the system sets this flag to true, for example, by executing `disable_flag = true`. When this flag is true, the speech synthesis module and text generation module check this flag at the function entry point for querying the mapping table. If the flag is true, the query operations for the speech style mapping table and the prompt word prefix mapping table are skipped, and an empty string is returned directly or the system exits early, outputting only the curiosity expression animation corresponding to the neutral state. When the confidence value of the subsequent emotion inference output recovers to above the preset confidence threshold, the system resets this flag to false, for example, by executing `disable_flag = false`, restoring the normal three-layer query and output process. Through this mechanism, in low-confidence scenarios, the terminal device only outputs animated feedback, without outputting any voice or text feedback at all.

[0077] Furthermore, in this way, when the terminal device is unsure of the user's emotion, it only plays a curious emoji animation, without outputting either speech or text. This completely eliminates the risk of inappropriate speech content due to misrecognition (such as mistakenly saying "You look sad" or "You're so happy today"). After seeing the curious emoji, the user may actively interact with the terminal, such as by speaking or touching it. At this time, the terminal can obtain clearer multimodal data (close range, frontal view, well-lit conditions), thereby increasing the confidence of the next emotion recognition and entering a normal feedback mode.

[0078] This embodiment outputs only a curious emoji animation in low-confidence scenarios, while simultaneously blocking both voice and text channels, thus implementing a safe fallback mechanism. Compared to solutions that merely correct the emotion to neutral but still output voice, this embodiment completely eliminates the risk of generating incorrect voice content due to misrecognition. Furthermore, the curious emoji animation is neither offensive to the user nor does it disrespect the terminal; it conveys the terminal's attention, making it a safe and user-friendly default feedback method. This mechanism is particularly suitable for scenarios such as children's play toys, avoiding confusion or negative emotions in children due to incorrect feedback.

[0079] In some preferred embodiments, the animation mapping table, the voice style mapping table, and the prompt word prefix mapping table are all static JSON configuration files stored on the local terminal. The method further includes: receiving an update package via OTA firmware update, the update package carrying a new configuration file for the target mapping table, a format checksum of the new configuration file, and a version number; verifying whether the format checksum of the new configuration file matches a preset JSON format template to obtain a first verification result; determining whether the version number of the new configuration file is higher than the version number of the corresponding mapping table currently stored locally to obtain a second determination result; if the first verification result is a match and the second determination result is higher, then the target mapping table stored locally is atomically replaced with the new configuration file, and the mapping data already loaded into memory remains unchanged during the replacement process, without interrupting the currently ongoing emotion feedback process; if the first verification result is a mismatch or the second determination result is not higher, then the update package is discarded and the original mapping table is retained.

[0080] In practice, this embodiment can securely update three static mapping tables without recompiling firmware or interrupting device operation, achieving hot updates of configurations. This allows for remote adjustment of feedback strategies via OTA, reducing maintenance costs while ensuring the security and smoothness of updates. The specific implementation method is as follows.

[0081] The animation mapping table, the voice style mapping table, and the prompt word prefix mapping table are all static JSON configuration files stored in the local non-volatile memory of the terminal device. The non-volatile memory can be flash memory, electrically erasable programmable read-only memory, or an embedded multimedia card. This embodiment describes the specific process of remotely updating these configuration files via OTA firmware updates. OTA stands for Over-The-Air, which allows devices to receive firmware or configuration updates via wireless networks (such as Wi-Fi or cellular networks) without a physical connection.

[0082] First, the terminal device receives the update package via Over-The-Air (OTA) firmware update. The update package is typically transmitted using Hypertext Transfer Protocol (HTTP) or Message Queuing Telemetry (MQT) and is encrypted and compressed. The update package carries a new configuration file for the target mapping table, a format checksum for the new configuration file, and a version number. The target mapping table can be any one of an animation mapping table, a voice style mapping table, or a cue word prefix mapping table. The format checksum is used to verify the correctness of the new configuration file's format; for example, it can be a JSON-formatted cyclic redundancy checksum or a 256-bit hash value using a secure hash algorithm. The format checksum is pre-calculated by the update package publisher and included in the package.

[0083] Further, the terminal device verifies whether the format checksum of the new configuration file matches a preset JSON format template to obtain a first verification result. The preset JSON format template defines the field structure, data types, and value ranges that the mapping table file should have. For example, the template for the animation mapping table requires each record to contain "emotion" (string type) and "animation_id" (integer type, value range 1-100). The terminal device first calculates the hash value of the received configuration file using the same algorithm as the publisher (such as SHA-256) and compares it with the checksum in the packet. If they do not match, it indicates that the file has been corrupted or tampered with during transmission. If they match, the JSON content is then parsed to verify whether the field names, data types, and value ranges conform to the template. If the format checksum matches the template, it indicates that the structure of the new configuration file is correct; if they do not match, it indicates that the file is corrupted or has an incorrect format.

[0084] Furthermore, the terminal device determines whether the version number of the new configuration file is higher than the version number of the corresponding mapping table currently stored locally, obtaining a second determination result. Each mapping table maintains a monotonically increasing version number, for example, in the format "major version number.minor version number.revision number". The version number is stored in string or integer form. Only configuration files with higher version numbers are accepted to prevent older versions from overwriting newer versions. The version comparison function needs to be able to correctly parse the version number string and perform numerical comparisons.

[0085] Furthermore, if the first verification result is a match and the second judgment result is higher, the terminal device atomically replaces the target mapping table stored locally with the new configuration file. Atomic replacement means the replacement operation is indivisible: the terminal device first writes the new configuration file to a temporary file, then calls the file system's rename or replace interface to atomically overwrite the original file with the temporary file. In a transactional file system, this operation either succeeds completely or fails completely, preventing the configuration file from being corrupted due to partial replacement. During the replacement process, the terminal device maintains the mapping data already loaded into memory. Specifically, the terminal device loads all mapping tables into memory at startup to improve query speed. During OTA updates, the ongoing sentiment feedback process still uses the old mapping data in memory. After the update is complete, the terminal device can set a "configuration updated" flag, reload the configuration file into memory for the next query, or use a double-buffering mechanism (keeping both old and new data in memory and switching atomically via pointers). This ensures that the currently ongoing sentiment feedback process is not interrupted.

[0086] Furthermore, if the first verification result is a mismatch or the second judgment result is not higher than, the terminal device discards the update package and retains the original mapping table without performing any replacement operation. The terminal device can record an error log and report it to the operation and maintenance server via the network for troubleshooting.

[0087] This embodiment achieves remote updates of the three mapping tables via OTA (Over-The-Air) without recompiling firmware or restarting the device, significantly reducing maintenance costs. Furthermore, format verification and version number checks ensure the correctness and security of the update package, preventing device malfunctions caused by misconfigurations. Moreover, atomic replacement and memory data retention mechanisms ensure that the update process does not interrupt the currently ongoing emotion feedback process, providing a seamless user experience and achieving hot updates. Compared to traditional solutions that require power outages or full firmware upgrades, this embodiment significantly improves maintainability and user experience.

[0088] In some preferred embodiments, the animation mapping table, the voice style mapping table, and the prompt word prefix mapping table are stored independently in different configuration files, each with its own independent version number. The method further includes: when a first update instruction for the animation mapping table is received, only the configuration file of the animation mapping table is replaced, while keeping the version numbers and contents of the voice style mapping table and the prompt word prefix mapping table unchanged; when a second update instruction for the voice style mapping table is received, only the configuration file of the voice style mapping table is replaced, while keeping the version numbers and contents of the animation mapping table and the prompt word prefix mapping table unchanged; when a third update instruction for the prompt word prefix mapping table is received, only the configuration file of the prompt word prefix mapping table is replaced, while keeping the version numbers and contents of the animation mapping table and the voice style mapping table unchanged; when performing parallel queries, the animation instruction, the voice style parameter, and the prompt word prefix are read independently from their respective configuration files. Failure to read any configuration file only results in the loss of output for the corresponding channel and does not affect the normal output of other channels.

[0089] In specific implementation, the technical objective of this embodiment is to ensure complete decoupling of the visual, auditory, and language feedback layers during storage, updating, and querying, thereby improving the robustness and scalability of the system. Through this embodiment, the configuration of a single layer can be modified without affecting other layers, and the corruption of a single layer's configuration file will not cause the entire system to crash. The specific implementation method is as follows.

[0090] Based on the method described in the above embodiments, the animation mapping table, the voice style mapping table, and the prompt word prefix mapping table are stored independently in different configuration files. Each configuration file is stored separately in the terminal device's file system, for example, named animation_map.json, voice_style_map.json, and prompt_prefix_map.json, respectively. Each configuration file has its own independent version number; for example, the version number of animation_map is v1.2, the version number of voice_style_map is v2.0, and the version number of prompt_prefix_map is v1.0. The version number is also stored within each configuration file or as a separate file attribute.

[0091] This embodiment defines the independent update process for the three mapping tables. When the terminal device receives a first update instruction for the animation mapping table, it only replaces the configuration file of the animation mapping table, that is, replaces the old `animation_map.json` with the new one, while keeping the version numbers and contents of the voice style mapping table and the prompt word prefix mapping table unchanged. Similarly, when a second update instruction for the voice style mapping table is received, only the configuration file of the voice style mapping table is replaced, while keeping the version numbers and contents of the animation mapping table and the prompt word prefix mapping table unchanged. When a third update instruction for the prompt word prefix mapping table is received, only the configuration file of the prompt word prefix mapping table is replaced, while keeping the version numbers and contents of the animation mapping table and the voice style mapping table unchanged. This independent update capability allows operators to adjust the animation effects individually without affecting the voice and text; for example, simply changing the "happy" animation from "shaking its head and wagging its tail" to "jumping" without reconfiguring the voice style or prompt word prefix.

[0092] Furthermore, regarding query execution, when the terminal device performs parallel queries, it independently reads the animation instructions, the voice style parameters, and the prompt word prefixes from their respective configuration files. These three read operations are independent and do not block each other. In specific implementation, the terminal device maintains an independent file handle or memory cache area for each mapping table. When querying the animation mapping table, it only accesses the memory area corresponding to `animation_map.json` and does not hold locks on other tables. More importantly, a failure to read any configuration file only results in the missing output of the corresponding channel and does not affect the normal output of other channels. For example, if the animation mapping table configuration file is corrupted due to bad blocks in the storage medium, causing JSON parsing failure or a read timeout, the terminal device will output a default animation instruction (such as the system's built-in "default" animation) in the animation channel, or it may not output any animation, but the voice and text channels will still function normally. Fault detection can be implemented through an exception handling mechanism: a try-catch block is used to catch parsing exceptions when reading the configuration file; when an exception occurs, the output of that channel is set to a preset default value, and the error is recorded, but the program continues to execute queries for other channels.

[0093] This embodiment decouples the visual, auditory, and linguistic feedback layers by storing the three mapping tables independently in different configuration files and maintaining their own version numbers. Modifying the animation mapping table does not affect the voice and text configurations, and vice versa, greatly improving maintainability. Independent reading and fault isolation mechanisms ensure that even if one layer's configuration file is corrupted, other layers can still function normally, avoiding a single point of failure that could paralyze the entire system. This design is particularly suitable for consumer products that require frequent adjustments to feedback strategies. Operations personnel can update animated emoticons independently without reconfiguring the voice style, reducing the risk of update errors and testing costs.

[0094] Furthermore, in some preferred embodiments, the method further includes: acquiring multiple consecutive frames of images captured by a camera, calculating the frame difference between adjacent frames; when the frame difference is continuously lower than a preset static threshold, reducing the inference frequency of emotion inference on the multimodal data from a preset normal inference frequency to a preset power-saving inference frequency; and when the frame difference exceeds a preset dynamic threshold, restoring the inference frequency to the normal inference frequency.

[0095] In specific implementations, to further reduce the power consumption of terminal devices in non-interactive or static scenarios, especially to reduce the computational load of neural network processing units (NPUs) or embedded processors, this invention also provides a dynamic adjustment mechanism for emotion recognition frequency. Specifically, this includes: The terminal device continuously captures multiple frames of images through a camera and calculates the frame difference between adjacent frames. This frame difference can be obtained by calculating the absolute difference and / or mean square error of corresponding pixels in two frames, and is used to measure the dynamic range of the scene. When the frame difference of multiple consecutive frames is lower than a preset static threshold, the current scene is determined to be a static scene (e.g., the user has remained still for a long time or has left). In response to this static scene determination, the terminal device reduces the inference frequency of the multimodal data's emotion inference from a preset normal inference frequency to a preset power-saving inference frequency. For example, the normal inference frequency is 10 frames per second, and the power-saving inference frequency is 0.5 frames per second (i.e., inference once every two seconds). The above values ​​are only examples; in actual applications, they can be flexibly set according to hardware performance and power consumption requirements.

[0096] Furthermore, when the frame difference exceeds a preset dynamic threshold, the scene is determined to have reverted to a dynamic scene, and the terminal device restores the inference frequency to the normal inference frequency. This mechanism significantly reduces unnecessary computational overhead when the scene is static, extending the operating time of battery-powered devices, while simultaneously restoring full-speed emotion recognition in real-time when the scene is dynamically changing, ensuring real-time response performance. It should be noted that the preset static threshold and the preset dynamic threshold can be the same value, or different values ​​can be set to introduce hysteresis comparisons and avoid frequent frequency switching.

[0097] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0098] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0099] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a multimodal feedback-driven method based on emotion perception.

[0100] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500. The processor 502 may include a neural network processing unit (NPU) to accelerate AI model inference.

[0101] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a multimodal feedback-driven method based on emotion perception.

[0102] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0103] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of any of the above method embodiments.

[0104] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0105] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0106] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of any of the above-described method embodiments.

[0107] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0108] Furthermore, the above method can be stored in the Flash memory of the embedded device as firmware and executed collaboratively by the NPU and MCU.

[0109] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0110] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0111] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0113] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0114] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0115] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal feedback-driven method based on emotion perception, characterized in that, include: In response to the acquired user multimodal data, emotion inference is performed on the multimodal data to generate emotion types; Using the emotion type as an index, the preset animation mapping table, voice style mapping table, and prompt word prefix mapping table are queried in parallel to generate the animation command, voice style parameter, and prompt word prefix corresponding to the emotion type, respectively. Animation output is executed based on the animation instructions, response text is generated based on the prompt word prefix, and speech is synthesized based on the response text and the speech style parameters.

2. The multimodal feedback-driven method based on emotion perception according to claim 1, characterized in that, The execution of animation output based on the animation instructions includes: The animation instructions are encapsulated into UART instruction frames; The UART instruction frame is sent to the animation execution module via the UART bus, wherein the sending operation of the UART instruction frame is performed before the operation of generating reply text based on the prompt word prefix and synthesizing speech based on the reply text and the speech style parameters, and the UART instruction frame is given the highest priority in the sending queue; After receiving the UART instruction frame, the animation execution module parses the UART instruction frame to obtain the animation instruction, and starts playing the animation sequence corresponding to the animation instruction.

3. The multimodal feedback-driven method based on emotion perception according to claim 1, characterized in that, The process of generating reply text based on the prompt word prefix includes: Retrieve the dialogue history in the current dialogue context and count the total number of rounds M in the dialogue history; If M is greater than the preset maximum number of rounds to retain K, then the dialogue statements of the most recent K rounds are selected as the valid dialogue history; if M is less than or equal to K, then all M rounds of dialogue statements are selected as the valid dialogue history. The prompt word prefix is ​​placed before the effective dialogue history and concatenated sequentially to construct a complete prompt word, wherein the prompt word prefix adopts the form of natural language instructions to specify the sentiment tendency, politeness level and tone intensity of the large language model's response; The complete prompt word is sent to the large language model, and the response text returned by the large language model is received.

4. The multimodal feedback-driven method based on emotion perception according to claim 1, characterized in that, The speech synthesis based on the reply text and the speech style parameters includes: The response text is subjected to text regularization and word segmentation to obtain the phoneme sequence and the original duration of each phoneme; The speech rate value in the speech style parameters is mapped to the duration scaling factor of each phoneme, the pitch value in the speech style parameters is mapped to the offset Hertz number of the fundamental frequency curve, and the emotional color label in the speech style parameters is mapped to the set of stress positions and the pause pattern encoding. The phoneme sequence, the duration scaling factor, the offset Hertz number, the accent position set, and the pause pattern encoding are input into the speech synthesis engine, and the speech data in waveform format returned by the speech synthesis engine is received.

5. The multimodal feedback-driven method based on emotion perception according to claim 1, characterized in that, The emotion inference also produces a confidence value for the emotion type; the method further includes: Determine whether the confidence value is less than a preset confidence threshold; If the confidence value is less than the preset confidence threshold, the emotion type is forcibly corrected to a neutral state, and the neutral state is used as the emotion type for subsequent queries. If the confidence value is not less than the preset confidence threshold, the emotion type remains unchanged.

6. The multimodal feedback-driven method based on emotion perception according to claim 5, characterized in that, The method further includes: When the emotion type is forcibly corrected to the neutral state, the default animation instruction corresponding to the neutral state is queried from the animation mapping table. The default animation instruction is the curiosity expression animation instruction. The animation output is executed based on the default animation instruction, while the query operations of the voice style mapping table and the prompt word prefix mapping table are blocked, so that only visual feedback is output in low confidence scenarios.

7. The multimodal feedback-driven method based on emotion perception according to claim 1, characterized in that, The animation mapping table, the voice style mapping table, and the prompt word prefix mapping table are all static JSON configuration files stored on the local terminal; the method further includes: The update package is received via OTA firmware update, and the update package carries a new configuration file for the target mapping table, a format checksum of the new configuration file, and a version number. Verify whether the format check code of the new configuration file matches the preset JSON format template to obtain the first verification result; Determine whether the version number of the new configuration file is higher than the version number of the corresponding mapping table currently stored locally, and obtain a second determination result; If the first verification result is a match and the second judgment result is higher, then the target mapping table stored locally is atomically replaced with the new configuration file, and the mapping data already loaded into memory remains unchanged during the replacement process, without interrupting the currently ongoing emotion feedback process; If the first verification result is a mismatch or the second judgment result is not higher than, then the update package is discarded and the original mapping table is retained.

8. The multimodal feedback-driven method based on emotion perception according to claim 1, characterized in that, The animation mapping table, the voice style mapping table, and the prompt word prefix mapping table are stored independently in different configuration files, and each has its own version number; the method further includes: When the first update instruction for the animation mapping table is received, only the configuration file of the animation mapping table is replaced, while the version number and content of the voice style mapping table and the prompt word prefix mapping table remain unchanged; When a second update instruction is received for the speech style mapping table, only the configuration file of the speech style mapping table is replaced, while the version number and content of the animation mapping table and the prompt word prefix mapping table remain unchanged; When a third update instruction is received for the prompt word prefix mapping table, only the configuration file of the prompt word prefix mapping table is replaced, while the version number and content of the animation mapping table and the voice style mapping table remain unchanged; When performing parallel queries, the animation instructions, the voice style parameters, and the prompt word prefixes are read independently from their respective configuration files. Failure to read any configuration file will only result in the missing output of the corresponding channel and will not affect the normal output of other channels. The method further includes: Acquire multiple consecutive frames of images captured by the camera and calculate the frame difference between adjacent frames; When the frame difference value is continuously lower than the preset static threshold, the inference frequency of the emotion inference of the multimodal data is reduced from the preset normal inference frequency to the preset power-saving inference frequency. When the frame difference exceeds a preset dynamic threshold, the inference frequency is restored to the normal inference frequency.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images