Emotion recognition and feedback system and method

Abstract

Provided in the present invention are a system and method for giving feedback on a recognized emotion by means of a virtual human and for interacting with the recognized emotion. The method comprises: a user inputting text information into an emotion recognition system; performing system integration on the inputted text information; giving a prompt instruction for the integrated text information, and controlling a large language model of a character by means of the prompt instruction; outputting a text response by means of the large language model, wherein the text response controls both outputted speech and a body; and performing sentiment analysis or opinion mining on a result of the text response, and also controlling the outputted speech and the body by means of an analyzed emotional state, and giving feedback in the form of a virtual human. In the emotion recognition, expression and interaction system and method of the present invention, a text input of a user is received, the emotion of the user is recognized by means of the text input, and comprehensive feedback in terms of emotion, life, work, etc., is given by means of a virtual human, such that the user obtains a comprehensive pleasant experience of vision and feeling.

Description

System and method for emotion recognition and feedback TECHNICAL FIELD

[0001] The present invention belongs to the field of systems and methods for emotion recognition, in particular, it relates to a system and method for recognizing emotions through the input of user text conversation, giving feedback to the recognized emotions through virtual human / bionic / Simulacra, and interacting with the recognized emotions. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, in the field of emotion recognition, "data" and "model" show a spiral promotion relationship. After the model progresses, the amount of data needed gradually increases, and after the data reaches a new level in some way, a new model is needed to improve recognition. In the development process, knowledge-guided data is the basis, and better understanding of data can better collect and label data. When artificial intelligence processes data and models, knowledge fusion using knowledge-driven is more effective, and the three form a complementary relationship. In addition, in emotion recognition, multi-modal recognition can greatly improve recognition performance, different input signals lead to different model selection, and different application data, hardware, interpretability, speed and accuracy requirements need to be considered, and the appropriate method is selected according to different needs. The bottleneck of emotion recognition landing is that the model training and the resources needed for arrangement are more, the universality of affective computing is poor, and in addition, data sources, data security and model interpretability are also one of the bottlenecks of emotion recognition landing. To truly land, the emotion recognition field needs to develop unified evaluation standards, combine technology development direction with real application scenarios, and collect and label higher-quality data sets.

[0003] Because of the above-mentioned multiple factors in the emotion recognition, the existing emotion recognition technology mainly includes: emotion recognition based on voice, various changes in the autonomic nervous system can indirectly change a person's speech, and emotion recognition technology can use this information to recognize emotions, for example, in the state of fear, anger or joy, the speech will be quickly, loudly, accurately pronounced, the tone is higher, the range is wider, and emotions such as fatigue, boredom, sadness, etc. often produce slow, low and unclear language. Emotion recognition based on facial expressions, facial expressions of human faces are analyzed to recognize emotions such as happiness, anger, surprise, fear, etc. Emotion recognition based on body movements, automatic motion analysis can be effectively used to detect the user's specific emotional state, for example, when you don't know the answer to a question, raise your shoulders, when using an object, point to them with your fingers, etc. These actions can be captured and recognized. Emotion recognition based on body data monitoring, detect and analyze user's body data signals to detect user's emotional state, such as heart rate, blood pressure, brain wave data, etc.

[0004] CN112926525A of the Construction Bank discloses a method for encoding a face image of a to-be-identified person to obtain an image encoding result, and encoding voice information output by the to-be-identified person to obtain a voice encoding result, wherein the voice encoding result is obtained according to text encoding and concept encoding of the voice information, so as to improve the accuracy of the voice encoding result; and then performing emotion recognition on the to-be-identified person according to a fusion encoding result of the image encoding result and the voice encoding result, so as to improve the accuracy of the emotion recognition of the person.

[0005] PCT / US2020 / 065680 of Sony Interactive Entertainment discloses a method for training an intelligent agent, which includes forming a personality matrix, combining a cognitive bias matrix with the personality matrix, and generating a behavior function for a scenario based on the combined cognitive bias matrix and personality matrix. The inputs from all aspects of vision, hearing, touch, taste and smell are analyzed in real time by the intelligent system, and the corresponding responses are calculated and executed using touch, voice generation and robot technology. The responses constructed by the intelligent system should be able to imitate human responses in a way that other humans cannot distinguish them from humans and potentially even more empathetic (or Machiavellian).

[0006] These prior art studies show that with the continuous development of artificial intelligence technology, more and more technologies related to emotion recognition are emerging in terms of both principles and applications, making the technology more close to human performance, thereby forming a true artificial intelligence.

[0007] There is a need in the prior art for a more optimized method and system of emotion recognition from the human emotional aspect through text input and feedback through virtual people / bionics / Simulacra according to the recognized emotion, so as to obtain user-satisfactory feedback from all aspects of vision and feeling.

[0008] SUMMARY

[0009] The present application relates to a system and method for recognizing emotions through user text conversation input, giving feedback to the recognized emotions through virtual people / bionics / Simulacra, and interacting with the recognized emotions.

[0010] (This part is consistent with the content of the claims, and will be supplemented after the content of the claims is determined)

[0011] Since the emotion recognition, expression and interaction system and method of the present application receives the input of user text, recognizes the emotion of the user through the text input, and gives comprehensive feedback in the aspects of emotion, life and work through the virtual person / bionic person / simulacra mode, so that the user can obtain a full range of visual and sensory pleasure experience. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some examples of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0013] Fig. 1 is a flow chart of the emotion recognition system of the present application.

[0014] Fig. 2 is a flow chart of the text information input step example of the emotion recognition system of the present application.

[0015] Fig. 3 is a flow chart of the giving prompt instruction step example of the emotion recognition system of the present application.

[0016] Fig. 4 is a system structure diagram of the emotion recognition system of the present application.

[0017] Fig. 5 is a computer product diagram of the portable or fixed storage unit of the emotion recognition system of the present application. DETAILED DESCRIPTION

[0018] The specific embodiments of the present application will be described in conjunction with the corresponding drawings. However, the present application can be implemented in various different forms, and should not be interpreted as being limited to the embodiments shown here. These embodiments are provided only to make the present application more thorough and comprehensive, so as to fully describe the scope of the present application to those skilled in the art. The language used in the detailed description of the embodiments illustrated in the drawings should not limit the present application.

[0019] The virtual person / simulacrum / bionic person / Simulacra in the present application, from the Latin simulacrum, means "similar, appearance", refers to the substitution of appearance for reality, the reproduction or imitation of people or things, and has a wide range of applications and meanings in philosophy, entertainment, cartoons, iconography, architecture, etc. For example, the gingerbread man, the thumbelina and even the terracotta warriors are all virtual people / simulacrum / bionic people / Simulacra, regardless of the external form of expression, the inner part is the human thinking. In Jean Baudrillard's (Selected Writings, Stanford University Press, 1988) article, the definition of the word simulacrum itself is that the simulacrum will never hide the truth - the truth will hide the non-existence of the truth, and the simulacrum is real. The virtual person / simulacrum / bionic person / Simulacra referred to in the present application refers to an object in a virtual community that can have natural characteristics such as specific user age, identity, gender, appearance, etc. It also has social characteristics such as personality and work, and is closely linked to the identity of a specific user. It is a virtual person / simulacrum / bionic person / Simulacra of a real user.

[0020] This invention relates to a rough classification of emotions, including but not limited to anxiety, restlessness, true feelings, love, judgment, rationality, emotion, craving, exploration, rudeness, fishing, comfort, fashion, hysteria, etherealness, cursing, and curiosity. According to Tiffany Watt Smith's *The Book of Human Emotions*, human emotions can be categorized into 156 types. Specifically, these include: love turned to hate, hurt self-esteem (Abhiman), laziness (Acedia), clinginess and affection (Amae), need for clarification (Ambiguphobia), anger (Anger), anticipation (Anticipation), anxiety (Anxiety), indifference (Apathy), and a hollow cry (L'appel du...). vide), separation and emptiness (Awumbuk), confusion (Bafflement), longing for a kiss (basorexia), bewilderment (Befuddlement), confusion (Bewilderment), boredom (Boredom), bottom line (Brabant), contemplation (Broodiness), calm (Calm), carefree (Carefree), happiness (Cheerfulness), boredom (Cheesed(off)), claustrophobia (Claustrophobia), stomachache caused by anxiety (Collywobbles), comfort (Comfort), compassion (Compassion), multifaceted love (Compersion), confidence (Confidence), contempt (Contempt), contentment (Contentment), courage (Courage), curiosity (Curiosity), cyberchondria (Cyberchondria), joy (Delight), disappointment (Dépaysement), desire (Desire), despair (Despair), disappearance, desire (Disappear, the desire) Disappointment, Disgruntlement, Disgust, Dissay, Dolcefar niente, Dread, Ecstasy, Embarrassment, Empathy, Envy, Euphoria, Exasperation, Excitement, Fear, Feeling good (about yourself), Formal feeling, Fraudfeeling like a), Frustration, Gezelligheid, Gladsomeness, Glee, Gratitude, Grengjai, Grief, Guilt, Han, Happiness, Hatred, Heebie-Jeebies, hiraeth, Hoard, the urge to, Homefulness, Homesickness, Hopefulness, Huff, in a, Humble, feeling, Humiliation, Hunger, Hwyl, Ijirashi, Ilinx, Impatience, Indignation, Inhabitiveness, Insulted, feeling, Irritation, Jealousy, Joy, Kaukokaipuu, Liget, Litost, Loneliness, Love, Malu, Man, Matutolypea, Mehameha, Melancholy, Miffed, a bit, Mono no aware, Morbid curiosity, Nakhes, Nginyiwarrarringu, Nostalgia, Oime, Overwhelmed, feeling, Panic, Paranoia, Perversity, Peur des espaces, Philoprogenitiveness, Pique, a fit of, Pity, Postal,going), Pride, Pronoia (the peculiar and perverse feeling that everything is conspiring to help you), Rage, Regret, Relief, Reluctance, Remorse, Reproachfulness, Resentment, Ringxiety, Rivalry, Road rage, Ruinenlust, Sadness, Satisfaction, Saudade, Schadenfreude, Self-pity, Shame, Shock, Smugness, Song, Surprise, Suspicion, Technostress, Terror, Torschlusspanik, Toska, Triumph, Vengefulness, Vergüenza ajena, Viraha, Vulnerability, Wanderlust, Warm glow, Wonder, Worry, Wistfulness, and Woe is me, The emotions involved in the emotion recognition system of the present application include a mixture of one or more of the above-mentioned emotions. FIG. 1 is a flowchart of the emotion recognition system of the present application. Step 101, text information is input into the emotion recognition system. The text information input includes input methods such as Chinese and English keyboard input, voice input, etc. Step 102, the input text information is subjected to system integration (SI). In this step, the SI can be composed of multiple entities to represent different perspectives of the role: LoRA / QLorA or checkpoints are the results of SFT / DPO, and the perspectives are integrated into a single output in the subsequent role LLM step. Step 103, the integrated text information is given a prompt instruction. Step 104, the given prompt instruction is input into the role large language model (LLM). Step 105, the output of the role large language model LLM is given a text response; and in steps 106 and 107, the results of the text response are input into the text-to-speech TTS step (step 106) and the body control step (step 107), respectively; in step 108, the results of the text response are subjected to text sentiment analysis / opinion mining; and in step 109, the emotional state of the text is obtained; the emotional state of the emotion is input into the text-to-speech TTS step (step 106) and the body control step (step 107), respectively, and feedback is given in the form of a virtual person.

[0021] In the system integration step 102, the fine-tuning technology of quantized large language model is adopted, wherein the LoRA technology is a low-rank adaptation technology (Low-rank adaptation, LoRA) which is an effective fine-tuning model technology based on adapter. A low-rank matrix is added to the original matrix. The adapter is a set of low-rank matrices, which produces a fine-tuning model when added to the base model. The QLorA technology is a quantized low-rank adaptation technology (Quantized Low-Rank Adaptation, QLorA), which realizes efficient fine-tuning of quantized LLMs by reducing the complexity and parameter number of the model. Since both low-rank decomposition and quantization reduce the parameter number and computational demand, the fine-tuning process of QLorA is more efficient. The SFT technology is a supervised fine-tuning technology (Supervised Fine-Tuning, SFT) which pre-trains a neural network model, i.e. a source model, on a source dataset. Then a new neural network model, i.e. a target model, is created. The target model replicates all model designs and their parameters of the source model except the output layer. These model parameters contain the knowledge learned on the source dataset, and these knowledge are also applicable to the target dataset. The output layer of the source model is closely related to the labels of the source dataset, so it is not used in the target model. When fine-tuning, an output layer with an output size equal to the number of classes of the target dataset is added to the target model, and the model parameters of this layer are randomly initialized. When training the target model on the target dataset, the output layer is trained from scratch, and the parameters of the remaining layers are fine-tuned based on the parameters of the source model; including pre-training, fine-tuning and evaluation process. The DPO technology is a direct preference optimization technology (Direct Preference Optimization, DPO), unlike traditional alignment methods based on reinforcement learning, DPO redefines the alignment formula as a simple loss function that can be directly optimized on the preference dataset {(x, y_w, y_l)}.

[0022] In one embodiment, in addition to the text input, the system integration step 102 can also receive the emotional state of the user. The emotional state is derived from an emotional analysis of the video state, including: a video input step, the video input is video information obtained by various camera devices; a facial landmark detection step on the video information, the facial landmark detection step locates the feature points on the face in the video information, such as the corners of the mouth, the corners of the eyes, etc., and is robust to rigid and non-rigid facial deformations caused by head motion and facial expressions; based on the 68 feature points of the Dlib library for face detection, 13 feature points are added (a total of 81 feature points), making the head detection and image operation more accurate; and a facial sentiment recognition step, which determines the facial expression in the video to infer the emotional state, usually containing 7 emotions (happy, sad, angry, disgusted, afraid, surprised and calm).

[0023] In another embodiment, the text response content in the text response step 105 can be used to control the retrieval-augmented generation (RAG) database step 300, which is described in detail below. The content of the text response is time-stamped and matched with the information in the RAG database to coordinate with the role large language model LLM.

[0024] In step 103 of giving a prompt instruction, the prompt instruction can be a question, a piece of text description, or a piece of text description with a bunch of parameters. In order to provide a specific prompt or question, the AI model can generate more accurate related text, pictures, or tables, etc. The prompt instruction is the starting point for user interaction with the language model, telling the model the user's intention, and expecting the model to respond in a meaningful and relevant way. In an embodiment, the prompt instruction can be a piece of handcrafted, pre-defined text with discrete input tokens, or a hard prompt of a text template. Static prompts can be incorporated into programs for programming, storage and reuse, and large model-based applications can have multiple prompt templates for use. In another embodiment, the prompt instruction can be a soft prompt created in the prompt tuning process. Soft prompts cannot be viewed and edited in the text, containing an embedding or a string of numbers representing knowledge obtained from a large model. In still another embodiment, the prompt instruction can be an online prompt provided in real-time interaction with the model, for immediate interactive applications. Such prompts are provided when the user is actually talking to the model, and the user can input, edit or change the prompt step by step, which is common in online chat, voice assistant, real-time question answering and other applications. The prompt instruction can contain 6 elements: task, context, example, role, format and tone, and these elements are arranged in descending order of importance.

[0025] In the LLM step 104, the training parameter quantity is significantly reduced due to the use of the above-mentioned quantization fine-tuning technology of the large language model, and since the LLM is suitable for different tasks, the representative model adopts different neurons / features to process for different tasks, finds the features suitable for the downstream task from the numerous features, and strengthens these features, that is, has better results for specific tasks. The underlying converter of the LLM described in the present application is a set of neural networks composed of an encoder and a decoder with self-attention function. The encoder and the decoder extract meaning from a series of texts and understand the relationship between the words and phrases in them. The neural network based on the converter is very large. These networks contain multiple nodes and layers. Each node in the layer has a connection to all the nodes in the subsequent layer, and each node has a weight and a bias. The weight and bias and the embedding are called model parameters. The large neural network based on the converter can have billions of parameters. The size of the model is usually determined by the empirical relationship between the model size, the number of parameters, and the size of the training data. Training is performed using a large amount of high-quality data. During the training process, the model iteratively adjusts the parameter values until the model can correctly predict the next token according to the previous input token sequence. To this end, the model uses self-learning techniques that teach the model to adjust the parameters to maximize the likelihood of correctly predicting the next token in the training examples.

[0026] In one embodiment, the optional role LLM can be controlled by the above-mentioned LoRA step, or can be controlled by the RM / PPO / RLHF step, which includes three stages, the Reward Model (RM) stage is used to evaluate the quality of the text output by the model; the Proximal Policy Optimization (PPO) stage is used to update the parameters of the model, optimize the reward function; the Reinforcement Learning from Human Feedback (RLHF) stage includes two steps of training model learning and human feedback, through human observation of the behavior of the machine and giving feedback, the human will give whether the answer of the machine is appropriate, whether it is helpful, and according to the evaluation, give good and bad evaluation.

[0027] In one embodiment of the present invention, the AI agents driven by LLMs can be extended to the multimodal domain. This extension enables agents to interpret and respond to diverse multimodal user queries, thereby handling more complex and nuanced tasks, and is referred to as Large Multimodal Agents (LMAs). The LLM step in the emotion recognition system of the present invention can be replaced by an LMA step, which comprises four modules, a perception module, a planning module, an action module, and a memory module.

[0028] The perception module, LMAs in different tasks involve various modalities, and they need to extract key information from these different modalities that are most beneficial to completing the task, thereby facilitating more effective planning and task execution. The perception module in the present invention senses key information most relevant to emotion perception from videos, texts, and images.

[0029] The planning module, LMAs operate in more complex environments, conduct in-depth reasoning on the current task, and formulate corresponding plans. Models such as GPT-3.5 or GPT-4 [43, 41, 9, 30, 57, 51] are used in the form of natural language or programs, such as calling the ImagePatch function to execute the planning in a formatted manner. For LMAs, during the planning process, relevant experiences are first retrieved to help the planning module think carefully to reduce uncertainty. In addition, plans formulated by humans in different states when performing the same task are utilized. Alternatively, when encountering similar states, the planner can refer to these “standard answers” for thinking, thereby forming a more reasonable plan. Alternatively, more complex planning methods, such as the Monte Carlo method, are used to expand the scope of planning search to find the optimal planning strategy. In the planning method step, dynamic or static planning can be used, the dynamic planning decomposes the goal into a series of sub-plans based on the initial input, similar to the Chain of Thought (CoT), even if errors occur during the process, the plan will not be re-formulated; the static planning refers to each plan is formulated based on the current environmental information or feedback. If an error is detected in the plan, it will return to the original state for re-planning.

[0030] The action module in a multimodal agent system is responsible for executing plans and decisions made by the planning module. It translates these plans into concrete actions, such as using tools, making physical movements, or interacting with interfaces, ensuring that the agent can accurately and efficiently achieve its goals and interact with the environment. It is divided into tool use (T), embodied action (E), and virtual action (V) steps, where tools include visual foundation models (VFMs), APIs, Python, etc. (as listed in Table 2); embodied action is performed by physical entities such as virtual characters; virtual action (V) includes web tasks (e.g., clicking links, scrolling, and keyboard usage). In terms of methods, the first involves using prompts to provide the agent with information about executable actions, such as currently available tools and their functions; the second involves collecting action data and using this information for a self-guided fine-tuning process of open-source large models, such as LLaVA. This data is typically generated by high-level models, such as GPT-4. Compared to agents that only use language, the complexity of action-related information and data requires more complex methods to optimize learning strategies.

[0031] The memory module in the agent is divided into long-term memory and short-term memory; in simple environments, short-term memory is sufficient for the agent to handle the task at hand. However, in more complex and realistic environments, long-term memory becomes crucial. Only a few LMAs integrate long-term memory. Unlike agents that only use language, these multimodal agents need long-term memory that can store information across various modalities. All modalities are converted to text format for storage. One multimodal long-term memory system is specifically designed to archive previous successful experiences. These memories are stored as key-value pairs, where the key is the multimodal state and the value is the successful plan.

[0032] In one embodiment, closed-source LLMs without long-term memory serve as the planning module. Prompts are used to leverage closed-source large language models (such as GPT-3.5) as planners for reasoning and planning. Depending on the specific environmental or task requirements, the execution of these plans can be done through a downstream toolkit or directly using physical devices (such as a mouse or robotic arm) to interact with the environment. Such LMAs typically operate in simpler environments, performing traditional tasks such as image editing, visual localization, and visual question answering (VQA).

[0033] In another embodiment, fine-tuned LLMs without long-term memory serve as the planning module. This involves collecting multimodal instruction execution data or using self-instructions to fine-tune open-source large language models (such as LLaMA) or multimodal models (such as LLaVA). This enhancement not only enables the model to serve as the central "brain" for reasoning and planning, but also enables it to execute these plans. The environment and tasks faced by LMAs in this embodiment are similar to those faced by closed-source LLMs without long-term memory, typically involving traditional visual or multimodal tasks.

[0034] In another embodiment, planning modules with indirect long-term memory are employed. LLMs serve as central planning modules and are equipped with long-term memory. These planning modules access and retrieve long-term memory by invoking relevant tools, which utilize these memories to enhance reasoning and planning. The multi-modal agent framework is suitable for dynamic tasks such as video processing. The framework consists of a planning module, a toolkit, and a task-dependent repository that catalogues spatial and temporal attributes. The planning module queries spatial and temporal attributes related to video content using specialized subtask tools, thereby inferring task-dependent temporal and spatial data. Each tool is stored in the toolkit and is designed specifically for a particular type of spatial-temporal reasoning and functions as an executor within the framework.

[0035] In another embodiment, planning modules with local long-term memory are employed. LLMs interact directly with long-term memory, bypassing the need to access long-term memory using tools. In the multi-modal agent design, the interactive planning module merges multi-modal base models with LLMs, first converting multi-modal inputs of the environment into text. The planning module further uses a self-checking mechanism to predict and evaluate each step in execution, proactively discovering potential flaws and, in combination with environmental feedback and self-explanation, quickly correcting and refining the plan without additional information. Furthermore, this multi-modal agent framework includes a novel multi-modal memory. Successful task plans and their initial multi-modal states are stored, and the planning module retrieves similar states from this database for new tasks, leveraging accumulated experience to complete tasks more quickly and efficiently.

[0036] In another embodiment, the LLM step in the emotion recognition system of the present invention can be replaced by a SSM step, which can employ a base model pre-trained to adapt to downstream tasks, the base model typically being a sequential model operating on arbitrary input sequences from various domains; the base model is based on a sequential model: Transformer and its core attention layer. The effectiveness of self-attention is attributed to its ability to densely route information within a context window, allowing it to model complex data; variants of the attention are able to model arbitrary content outside the window, for example, employing structured state space sequential models (SSM) for recursion or convolution, scaling the sequence length linearly or nearly linearly.

[0037] The SSM includes a selection module, a hardware-aware module, and an architecture module. The selection module, first, identifies a key limitation of previous models: the ability to effectively select data in an input-dependent manner (i.e., focusing on or ignoring specific inputs). Based on intuition for important compositional tasks (e.g., selective copying and sensory head), a simple selection mechanism is designed by parameterizing SSM parameters based on inputs. This allows the model to filter out irrelevant information and remember relevant information indefinitely. The hardware-aware module uses scan instead of convolutional recurrent computation models, but does not implement extended states to avoid IO access between different levels of the GPU memory hierarchy. The architecture module combines the design of previous SSM architectures with the MLP block of Transformers into a block, forming a simple and homogeneous architecture design (Mamba) that includes a selective state space. The selective SSM, as well as the extension of the Mamba architecture, are fully recurrent models with key properties that make them suitable as a backbone for a general-purpose model for operating on sequences.

[0038] In the text response step 105, since the LLM has the ability to make inferences from the environment, the text response step 105 is used to generate a coherent and environment-related response. The extraction of entities and relationships from the text response of the LLM can enhance the communication between the user and the LLM by, for example, converting the text response into a graphical chart.

[0039] In the text-to-speech (TTS) step 106, text-to-speech converts text into speech signals and generates audible speech through a speech synthesizer; including (1) text preprocessing step, the input text is segmented, part-of-speech tagged, and parsed to identify words and phrases in the text. (2) Speech synthesis step, the preprocessed text is converted into speech signals, and the speech synthesizer generates speech. In this process, the speech synthesizer uses different speech libraries and algorithms to generate speech in different voices and languages. The synthesis network converts text into time-aligned features such as spectrograms or fundamental frequencies, which are the frequencies at which the vocal cords vibrate in speech. The vocoder network converts time-aligned features into audio waveforms. (3) Speech post-processing step, the generated speech is post-processed, including pitch adjustment, voice quality improvement, noise removal, etc., to improve the quality of the generated speech.

[0040] In the limb control step 107, local control steps can be used to control individual limbs, such as FLAME (face) and AMASS (limb) techniques; or whole-limb control steps can be used, such as EMAGE technology.

[0041] As an example of the FLAME model, a linear blend skinning technique is used to deform the face mesh with the pose. The steps taken include, establishing a face mesh template; determining key face joints, such as neck, chin, etc. joints; assigning skinning weights W to each mesh point, indicating the degree of influence of each joint; when receiving the output deformation of the text response, performing rigid body transformation on each joint according to the pose rotation parameter θ to obtain the transformed joint position J'; using the skinning weight W to calculate the interpolation of each mesh vertex relative to each joint to obtain the deformed mesh vertex position T'.

[0042] As an example of the EMAGE technology, prior knowledge of the body action mask is used in the training process to improve the inference performance. EMAGE uses an audio and action mask converter to effectively improve the joint training efficiency of audio generating action and action reconstruction under the action mask, and effectively encodes the prompt frame of audio and body action into the network. The body prompt frame of the action mask is used to generate face and body action respectively. In addition, EMAGE adaptively combines the rhythm and content of the audio speech features, and uses the combined VQ-VAEs of each part of the body to enhance the authenticity and diversity of the results.

[0043] The limb control step 107 and the text-to-speech step 106 also receive input from a text sentiment analysis / opinion mining step 108 and a step 109 of obtaining an emotional state of the text. In the text sentiment analysis / opinion mining step 108 and the step 109 of obtaining an emotional state of the text, subjective information in the text response output is identified and extracted in order to find the subjective attitude of the text information input, including emotional state or intentional emotional communication; to determine whether the opinion of the text information input is positive, negative, or neutral, and to determine more complex emotional states, including, for example, "angry," "sad," "happy," and so on. Emotion can be identified using keyword recognition, lexical association, statistical methods, or concept-level techniques: keyword recognition techniques use keywords appearing in the text response, such as "happy," "sad," "sorrow," "fear," "bored," to classify; lexical association assigns a "connection" value between a particular keyword and an emotion in addition to determining the keyword; statistical methods analyze latent semantic analysis, support vector machines, and bag-of-words to regulate learning elements; to mine opinions in a certain context or to obtain a function to which an opinion is given, the relationship between syntax needs to be used. The relationship between syntax often needs to be obtained by deep parsing of the text. Concept-level techniques weigh elements of knowledge representation, such as ontologies and semantic networks, so the algorithm can also explore subtle emotional expressions between words. For example, some concepts do not explicitly express related information, but they can be obtained by their subtle connection to explicit concepts. Sentiment analysis can be divided into three levels of analysis: document level, sentence level, and aspect level. In the document level, only one sentiment is extracted from a document. In the sentence level, the sentiment analysis method extracts a corresponding sentiment from each sentence in the document. Aspect level is quite different from the above two. In the aspect level, the unit of sentiment analysis is a different aspect of an entity, and an aspect extracts a sentiment. The purpose of aspect-level sentiment analysis is to obtain more granular analysis results.

[0044] Figure 2 is a flow chart of an example of a text information input step of the emotion recognition system. In the text information input step 101, text input can be directly in Chinese or English. In one embodiment, speech input can also be used, and the obtained speech input is converted into a text file that can be recognized by the emotion recognition system through a speech-to-text (SST) step 200, and the text file is input. In one embodiment of the present application, the speech-to-text step 200 is divided into the following sub-steps: step 201, obtaining a user's speech signal by using a microphone or other speech collection device; optionally, step 202, digitizing the collected speech signal to remove noise, echo, and other interference signals; step 203, mapping the speech signal to a feature vector space, and modeling the speech signal using, for example, a Gaussian Mixture Model (GMM), a Hidden Markov Model (HMM), or the like; step 204, mapping the text information to a statistical space, and modeling the text information using an n-gram language model; step 205, combining the speech signal modeling and the language modeling, and decoding the speech information using a maximum likelihood or maximum a posteriori probability method; and step 206, post-processing the decoding result, such as grammar correction, part-of-speech tagging, and the like. In another embodiment, a tone emotion analysis step can also be added, and the analyzed information is input to the system integration step 102. The tone emotion analysis step includes classifying emotions according to certain characteristic standards, and then classifying emotions into "happy", "angry", "surprised", "fearful", and "sad", and the like. A feature tone extraction algorithm derived from a multivariate analysis method, such as PCA technology, is used to convert a set of correlated vectors in a high-dimensional space into a set of uncorrelated vectors arranged in descending order of energy, extract new uncorrelated features from the original correlated variable set, analyze all tones, obtain a set of basic tones, and make the basic tones approximate to the original tones of all original speech inputs within an error tolerance range. An emotion tone model is established with the help of the feature tones. The obtained tone emotion / emotion is used to control the system integration step 102.

[0045] Figure 3 is a flow chart of an example of the prompt instruction step of the emotion recognition system of the present application. Between the text input step 101 and the system integration step 102 and the prompt instruction step 103, a retrieval-augmented generation (RAG) database step 300 can be added, in which the information from the text information input step 101 and the system integration step 102 can first pass through the retrieval-augmented generation database step 300 for optimization before entering the prompt instruction step 103. The purpose of adding the RAG step 300 is to optimize the large language model output to enable it to reference authoritative knowledge bases outside the training data sources before generating a response. The sub-steps of the retrieval-augmented generation database step 300 include: step 301, creating external data using the information input in the text information input step or the integrated information input in the system integration step, the external data including from multiple data sources, such as APIs, databases or document repositories. The data can exist in various formats, such as files, database records or long texts. In one embodiment of the external data creation step 301, the data can be converted into a digital representation using the technology of embedding language models and stored in a vector database. Step 302, performing a relevance search. The user query will be converted into a vector representation and matched with the vector database. Relevance is calculated and established using mathematical vector calculations and representations. Step 303, augmenting the prompt instruction, the RAG database augments the user input or prompt by adding the retrieved relevant data in context. The augmented prompt allows the large language model to generate accurate answers for the user query. Step 304, updating the external data, continuously updating the documents and updating the embedded representation of the documents. The result of the retrieval-augmented generation database step 300 is used to control the prompt instruction step 103.

[0046] In one embodiment of the emotion recognition system of the present invention, an example is illustratively given of the virtual person interacting with the user by obtaining the emotional state of the text step 109. For example, the virtual person gives the answer to the question "Would you like to be a romantic friend?" as "As a law school student, I am focused on my studies and personal growth. I would appreciate if we could maintain a friendly and platonic interaction. Let's keep a positive and respectful atmosphere!" and accordingly continues further interaction by the emotion recognition system of the present invention, for example: "Oh, so you're not ready to engage in a romantic relationship yet." Virtual person: "Ah, that's right! I am not interested in finding a romantic partner right now. I prioritize my studies and personal growth, and I want to focus on becoming the best version of myself before exploring relationships. Also, I am still figuring out who I am and what I want!" "So what is your ideal type?" Virtual person: "Ah, that's an interesting question! I am a spiritual person, so I like someone who shares similar values and is kind. Intelligent, witty, and good at conversation are definitely pluses! Plus, if they appreciate my love for dance and music, that's even better!" "I think you're cute, and I don't see why we can't be friends." Virtual person: "Ah, thank you for the compliment! I am honored, but I think we can definitely be friends without any romantic implications. I value our conversation and would like to explore common interests and support each other as friends!" "You must know how to reject boys, I bet you have experience" Virtual person: "Ah, no way! I am just a law student trying to focus on my studies and goals. I don't have time for dating or playing games. I am more interested in building meaningful connections and helping others."

[0047] Fig. 4 is a system architecture diagram of the emotion recognition system of the present application. The system server 401 expresses the entity item, for example, by a non-homogeneous proof of stake. The emotion recognition system server comprises a processor 410, which can be a general or special purpose chip (ASIC / eASIC) or FPGA or NPU, etc., and a computer program product or computer readable medium in the form of a memory 420. The memory 420 can be an electronic memory such as a flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk or ROM. The memory 420 has a storage space 430 for program code for executing any of the method steps in the above methods. For example, the storage space 430 for program code can comprise individual program codes 431 for implementing the various steps in the above methods, respectively. These program codes can be read out or written into the processor 410. These computer program products comprise program code carriers such as hard disks, compact discs (CDs), memory cards or floppy disks. Such computer program products are typically portable or stationary storage units as described with reference to Fig. 5. Fig. 5 is a computer product diagram of a portable or stationary storage unit of the emotion recognition system of the present application. The storage unit can have a storage section, storage space, etc., arranged similarly to the memory 420 in the server of Fig. 4. The program codes can be compressed, for example, in a suitable form. Typically, the storage unit comprises computer readable codes 431', i.e. codes that can be read by a processor such as 410, which, when run by the server, cause the server to perform the various steps in the above described methods. The codes, when run by the server, cause the server to perform the various steps in the above described methods. The term "one embodiment", "an embodiment" or "one or more embodiments" as referred to herein means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0048] The above description is merely intended to illustrate the technical solutions of the present application, and any person skilled in the art can modify and change the above embodiments without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application should be subject to the scope of claims. The present application has been described above in conjunction with examples. However, other embodiments beyond the above embodiments within the scope of the present application are also feasible. Different features and steps of the present application can be combined in other methods different from those described. The scope of the present application is limited only by the appended claims. More generally, all parameters, dimensions, materials, and arrangements described herein are meant to be exemplary and actual parameters, dimensions, materials, and / or arrangements will depend upon specific application or applications for which the present application is used.

Claims

1. A method of emotion recognition, comprising the steps of: a user inputting text information; system integration of the inputted text information to represent different perspectives of a character, integrating multiple perspectives; providing a prompt instruction for the integrated text information; and controlling a large language model (LLM) of the character using the prompt instruction; providing a text response to the large language model of the character; controlling the conversion of the text into speech of a virtual person and controlling the limbs of the virtual person using the text response, respectively; performing text sentiment analysis / opinion mining on the text response to obtain an emotional state of the text information input; further controlling the conversion of the text into speech of a virtual person and controlling the limbs of the virtual person using the emotional state, respectively; displaying the recognized emotion by the virtual person and providing feedback.

2. The method of emotion recognition of claim 1, wherein the system integration step employs a fine-tuning technique of a quantized large language model to supervise a low-rank adaptation technique or a direct preference optimization technique.

3. The method of emotion recognition of claim 1, wherein the system integration step further receives emotional state information of a user derived from video state emotion analysis, comprising: obtaining video information from various camera devices; performing facial landmark detection on the video information, which is robust to rigid and non-rigid facial deformations caused by head motion and facial expressions; performing facial emotion recognition to determine the facial expressions in the video to infer the emotional state. 4.The emotion recognition method of claim 1, wherein the integrated text information is given a hint instruction; wherein, The prompt instruction is a question, a piece of text description, or a text description with a bunch of parameters, so as to provide a specific prompt or question to make the AI model more accurately generate related text, pictures, or tables, etc.

5. The method of emotion recognition of claim 4, wherein the prompt instruction is a handcrafted, pre-defined text with discrete input tokens, or a hard prompt of a text template.

6. The method of emotion recognition of claim 4, wherein the prompt instruction is a soft prompt created in a prompt tuning process.

7. The method of emotion recognition of claim 4, wherein the prompt instruction is an online prompt provided in real-time interaction with the model for immediate interactive applications.

8. The emotion recognition method of claim 1, wherein, In the step of controlling the large language model of the character using the prompt instruction, the underlying transformer of the large language model is a set of neural networks composed of an encoder and a decoder with self-attention capabilities; Further comprising, the encoder and the decoder extract meaning from a sequence of text and understand the relationships between the words and phrases therein; the training model is determined by the empirical relationship between the model size, the number of parameters, and the scale of the training data; During the training process, the model iteratively adjusts the parameter values until the model can correctly predict the next token based on the previous input token sequence.

9. The emotion recognition method of claim 8, wherein, The step of controlling the role of the large language model by the prompt instruction is controlled by optimizing the low rank adaptation technology or the RM / PPO / RLHF technology, and the RM / PPO / RLHF technology includes a training reward model for evaluating the text quality of the model output; a proximal policy optimization of updating the parameters of the model and optimizing the reward function; and a reinforcement learning stage based on human feedback.

10. The emotion recognition method of claim 1, wherein the text response step is used to generate a coherent and environment-related response due to the ability of the LLM to make inferences from the environment, entities and relationships are extracted from the text response of the LLM, and communication between the user and the LLM is enhanced by converting the text response into a graph chart.

11. The emotion recognition method of claim 1, wherein in the text-to-speech step, the text is converted into a speech signal, and an audible speech is generated through a speech synthesizer; comprising: a text preprocessing step, in which the input text is subjected to word segmentation, part-of-speech tagging, and syntax analysis operations to identify words and phrases in the text; a speech synthesis step, in which the preprocessed text is converted into a speech signal, and a speech is generated through a speech synthesizer; a synthesis network converts the text into time-aligned features such as a spectrogram or a fundamental frequency, which is the frequency at which the vocal cords vibrate in speech; a vocoder network converts the time-aligned features into an audio waveform; and a speech post-processing step, in which the generated speech is post-processed, including pitch adjustment, sound quality improvement, and noise removal operations to improve the quality of the generated speech.

12. The emotion recognition method of claim 1, wherein in the limb control step, a local limb control step is used, such as the FLAME (face) and AMASS (limb) techniques; or a global limb control step is used, such as the EMAGE technique.

13. The emotion recognition method of claim 12, wherein the FLAME technique is to deform a face mesh with pose using linear blend skinning, including establishing a face mesh template; determining key face joints, such as neck, chin, and the like joints; Each mesh point is assigned a skinning weight W, indicating the degree of influence of each joint; when receiving the output deformation of the text response, each joint is subjected to a rigid transformation according to the pose rotation parameter θ to obtain the transformed joint position J'; Using the skinning weight W, the interpolation of each mesh vertex relative to each joint is calculated to obtain the deformed mesh vertex position T'.

14. The emotion recognition method of claim 12, wherein the EMAGE technique uses prior knowledge of the limb motion mask during the training process to improve inference performance; EMAGE uses an audio-to-motion mask converter to improve the joint training efficiency of audio-generated motion and motion reconstruction under the motion mask, and encodes the prompt frames of audio and limb motion into the network; the limb prompt frames of the motion mask are used to generate face and limb motions, respectively.

15. The method of claim 1, wherein the step of performing text sentiment analysis / opinion mining on the text response to obtain the emotional state of the text information input; includes identifying and extracting subjective information from the text response output to find out the subjective attitude of the text information input, including emotional state or intended emotional communication; using keyword recognition, lexical association, statistical method or concept level technology to identify emotion; assigning an "association" value between a particular keyword and an emotion; using statistical method to analyze latent semantic analysis, support vector machine, bag of words to regulate learning elements; using the relationship between the grammar to mine the opinion in a certain context, or to obtain a function given the opinion.

16. The method of claim 15, wherein the sentiment analysis includes a document level where a document only extracts one emotion, a sentence level where a sentiment analysis method extracts one corresponding emotion for each sentence in a document, and different perspectives where the unit of sentiment analysis is an entity, and an aspect level where a perspective extracts one emotion.

17. The emotion recognition method of claim 1, wherein the text information input step receives a text input in Chinese or English, or receives a voice input, converts the obtained voice input into a text file that can be recognized by the emotion recognition system of the present invention, and inputs the text file. including obtaining the user's speech signal using a microphone or other speech collection device; digitizing the collected speech signal, removing noise, echo and other interference signals; mapping the speech signal to a feature vector space, and modeling the speech signal using, for example, Gaussian Mixture Model (GMM), Hidden Markov Model (HMM), etc. mapping the text information to a statistical space, and modeling the text information using an n-gram language model; combining the speech signal modeling and the language modeling, and decoding the speech information using maximum likelihood or maximum a posteriori probability, etc. performing a post-processing step on the decoding result, such as grammar correction, part-of-speech tagging, etc.

18. The method of claim 17, further comprising a tone of voice emotion analysis step; including making an effective and reasonable classification of the emotion according to certain characteristic standards, and then classifying the emotion into "joy", "anger", "surprise", "fear", and "sadness", etc. and a feature tone extraction algorithm derived from the PCA technique, a multivariate analysis method, which converts a set of correlated vectors in a high-dimensional space into a set of uncorrelated vectors arranged in descending order of energy, extracts new uncorrelated features from the original correlated variable set, analyzes all tones of voice, and obtains a set of basic tones of voice that approximate the original tones of voice of all original speech inputs within an error tolerance range; establishing an emotional tone model with the help of feature tones; and controlling the system integration step by obtaining the tone of voice emotion / sentiment. 19.The emotion recognition method of claim 1, wherein a retrieval-augmented generation (RAG) database step is added between the system integration (SI) step and the prompt instruction step, wherein the information of the text information input step and the system integration step are both first passed through the step of the RAG database for optimization before entering the prompt instruction step. The search enhancement generation database step includes: creating external data using the information input by the text information input step or the integrated information input by the system integration step; performing a relevance search, converting the user query into a vector representation, and matching it with the vector database; Augmented prompt instruction, the RAG database enhances user input or prompts by adding retrieved relevant data in context; augmented prompts allow large language models to generate accurate answers for user queries; Update external data.

20. An emotion recognition system comprising a text information input module, in which a user inputs text information; a system integration module, which integrates the input text information to represent different perspectives of a character, and integrates multiple perspectives; a prompt instruction control character large language model module, which gives a prompt instruction to the integrated text information; and controls a large language model (LLM) of a character using the prompt instruction; a text response module, which gives a text response to the large language model of the character; a virtual human voice and body control module, which respectively controls the conversion of text into the voice of a virtual human and controls the body of a virtual human using the text response; an emotion analysis / opinion mining module, which performs text emotion analysis / opinion mining on the text response to obtain the emotional state of the text information input; a voice conversion module, which further controls the conversion of text into the language of a virtual human and controls the body of a virtual human using the emotional state; a feedback module, which displays the recognized emotion through the virtual human and gives feedback.

21. The emotion recognition system of claim 20, wherein the system integration module employs a fine-tuning technique of a quantized large language model to supervise a fine-tuning technique or a direct preference optimization technique to supervise or optimize a low-rank adaptation technique or a quantized low-rank adaptation technique.

22. The emotion recognition system of claim 20, wherein the system integration further receives user emotional state information derived from video state emotion analysis, comprising: a video information acquisition module, which obtains video information from various camera devices; a face feature detection module, which detects face feature points of the video information, and is robust to rigid and non-rigid facial deformations caused by head motion and facial expressions; a facial emotion recognition module, which performs facial emotion recognition to determine the facial expression in the video to infer the emotional state.

23. The emotion recognition system as claimed in claim 1, wherein the integrated text information is given a hint instruction; wherein, The prompt instruction is a question, a text description, or a text description with a bunch of parameters, so as to provide a specific prompt or question, and let the AI model generate more accurate text, pictures, or tables, etc.

24. The emotion recognition system of claim 23, wherein the prompt instruction is a handcrafted, pre-defined text with discrete input tokens, or a hard prompt of a text template.

25. The emotion recognition system of claim 23, wherein the prompt instruction is a soft prompt created in a prompt tuning process.

26. The emotion recognition system of claim 23, wherein the prompt instruction is an online prompt, which is provided in real-time interaction with the model, for immediate interactive applications.

27. The emotion recognition system of claim 20, wherein, In the prompt instruction control character large language model module, the underlying converter of the large language model is a set of neural networks composed of an encoder and a decoder with self-attention function. Also included, the encoder and decoder extract meaning from a series of texts and understand the relationships between the words and phrases therein; the training model is determined by the empirical relationship between model size, number of parameters, and the scale of training data; During the training process, the model iteratively adjusts the parameter values until the model can correctly predict the next token according to the previous input token sequence.

28. The emotion recognition system of claim 27, wherein, The prompt instruction controls the character large language model module, which is controlled by optimizing the low-rank adaptation technology or the RM / PPO / RLHF technology, which includes a training reward model for evaluating the quality of the text output of the model; updating the parameters of the model, optimizing the reward function of the proximal policy optimization; the reinforcement learning stage based on human feedback.

29. The emotion recognition system of claim 20, due to the ability of LLM to make inferences from the environment, the text response module is used to generate coherent and environment-related responses, extract entities and relationships from the text response of LLM, and enhance communication between users and LLM by converting text responses into graphical charts.

30. The emotion recognition system of claim 20, wherein in the text-to-speech module, the text is converted into a speech signal and an audible speech is generated through a speech synthesizer; comprising: The text preprocessing module performs word segmentation, part-of-speech tagging, and syntax analysis on the input text to identify words and phrases in the text; The speech synthesis module converts the preprocessed text into a speech signal and generates speech through a speech synthesizer; the synthesis network converts the text into time-aligned features such as spectrograms or fundamental frequencies, which are the frequencies at which the vocal cords vibrate in speech; the vocoder network converts the time-aligned features into audio waveforms; the speech post-processing step performs operations such as pitch adjustment, voice quality improvement, and noise removal to improve the quality of the generated speech.

31. The emotion recognition system of claim 20, in the limb control module, local control modules are used to control individual limbs, such as FLAME (facial) and AMASS (limb) technologies; or whole-body limb control modules are used, such as EMAGE technology.

32. The emotion recognition system of claim 31, wherein the FLAME technique is to utilize linear blend skinning to deform a face mesh with pose, including establishing a face mesh template; determining key face joints, such as neck, chin, etc. joints; Each grid point is assigned a skinning weight W, indicating the degree of influence of each joint; when receiving the output deformation of the text response, each joint is rigidly transformed according to the pose rotation parameter θ to obtain the transformed joint position J'; Using the skinning weight W, the interpolation of each grid vertex relative to each joint is calculated to obtain the deformed grid vertex position T'.

33. The emotion recognition system of claim 31, wherein the EMAGE technology uses prior knowledge of the limb motion mask to improve inference performance during the training process; EMAGE uses an audio-to-motion mask converter to improve the joint training efficiency of audio-generated motion and motion reconstruction under the motion mask, and encodes the prompt frames of audio and limb motion into the network; the limb prompt frames of the motion mask are used to generate facial and limb motions respectively.

34. The emotion recognition system of claim 20, wherein the module of text sentiment analysis / opinion mining of the text response to obtain the emotional state of the text information input; includes identifying and extracting subjective information of the text response output to find out the subjective attitude of the text information input, including emotional state or intentional emotional communication; using keyword recognition, lexical association, statistical method or concept level technology to identify emotion; giving the "association" value between special keywords and certain emotion; using statistical method to analyze latent semantic analysis, support vector machine, bag of words to regulate learning elements; using the relationship between syntax to mine opinions in a certain context or to obtain a certain function given opinions.

35. The emotion recognition system of claim 34, wherein the sentiment analysis includes a document level in which a file is extracted with only one emotion, a sentence level in which a sentence in a file is extracted with a corresponding emotion, and different aspects in which the unit of sentiment analysis is an entity, and an aspect level in which an aspect is extracted with one emotion.

36. The emotion recognition system as claimed in claim 20, wherein the text information input module receives text input in Chinese or English, or receives voice input, converts the obtained voice input into a text file which can be recognized by the emotion recognition system of the present application, and performs text file input. including a voice signal acquisition module that acquires a voice signal of a user using a microphone or other voice acquisition device; a digital processing module that performs digital processing on the acquired voice signal to remove noise, echo, and other interference signals; a voice signal modeling module that maps the voice signal to a feature vector space and models the voice signal using, for example, a Gaussian Mixture Model (GMM), a Hidden Markov Model (HMM), or the like; a language modeling module that maps text information to a statistical space and models the text information using an n-gram language model; a decoding module that combines the voice signal modeling and the language modeling and decodes the voice information using a maximum likelihood or maximum a posteriori probability method or the like; a post-processing module that performs a post-processing step on the decoding result, such as syntax correction, part-of-speech tagging, or the like.

37. The emotion recognition system of claim 36, further comprising a tone emotion analysis module; including making an effective and reasonable classification of emotion according to certain characteristic standards, and then classifying the emotion into "happy", "angry", "surprised", "fearful", and "sad", etc. and a feature prosody extraction algorithm derived from the PCA technique, which is a multivariate analysis method, converts a set of correlated vectors in a high-dimensional space into a set of uncorrelated vectors arranged in descending order of energy, extracts new uncorrelated features from the original correlated variable set, analyzes all prosodies, and obtains a set of basic prosodies that approximate the original prosodies of all original voice inputs within an error tolerance range; an emotion prosody model is established using the feature prosody, and the prosody emotion / sentiment obtained is used to control the system integration step. 38.The emotion recognition system of claim 20, wherein a Retrieval-Augmented Generation (RAG) database module is added between the System Integration (SI) module and the prompt instruction module, wherein the text information input step and the information of the system integration step are both first input into the RAG database module for optimization, and then input into the prompt instruction module. The search enhancement generation database module includes: an external data creation module that creates external data using information input in the text information input step or integrated information input in the system integration step; a search matching module that performs a correlation search, converts a user query into a vector representation, and matches the vector representation with a vector database; and a result output module that outputs a result of the search matching. An enhanced prompt instruction module, the RAG database enhances the user input or prompt by adding the retrieved relevant data in the context; the enhanced prompt allows the large language model to generate accurate answers for the user query; An update external data module.

Images