Multimodal affective interaction system and method

Abstract

The embodiment of the application discloses a multi-modal emotional interaction system and method, the system comprises a client and a server, through the cooperation of the client and the server, the application behavior analysis (ABA) theory is used as a basic framework, the behavior of the child is understood and intervened by the parents, and through the simulation scene practice, the parents are guided to understand the details of the intervention behavior, a structured roadmap can be provided for the parents, the behavior of the child is helped to be understood and managed, and meanwhile, the theoretical concept is combined with the practical application, the ability of the parents to provide customized intervention for the needs of the child is improved. Can be used for solving the problems that the information source of the current autism child care is unreliable, personalized content is insufficient, and there is a gap between theoretical knowledge and actual operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of assisted rehabilitation training, and in particular to a multi-modal emotional interaction system and method. BACKGROUND

[0002] Current technologies mainly focus on interventions implemented by parents in the home environment for children with autism, which involve addressing complex behavioral challenges such as social development, communication disorders, and emotional management. Parents often rely on information provided by social media and professional educators, such as WeChat, Zhihu, and Douyin platforms, to obtain detailed content and case-based explanations to help them understand and implement interventions.

[0003] However, the existing technology has many problems and shortcomings. First, the interventions lack standardized implementation, and inconsistencies often occur when different personnel perform them. Second, it is difficult to extract practical intervention methods from theoretical knowledge because of the lack of comprehensive details, making theoretical knowledge often unable to effectively guide practical intervention practice. In addition, relying on information sources from social media and other platforms may have variations in reliability and content quality, and these platforms lack personalized filtering capabilities. Existing technologies are not sufficient to fully meet the diverse needs of parents in the family, social, and educational environments. At the same time, parents are concerned about their mental health status due to long-term contact with children with autism. The root cause of these problems lies in the limitations of existing technologies, such as the reliability of information sources, the lack of personalized content, and the gap between theoretical knowledge and practical application. In order to provide customized, comprehensive, easy-to-understand, and applicable intervention strategies, it is necessary to further research multi-modal emotional interaction technology to address the current problems of unreliable information sources, lack of personalized content, and the gap between theoretical knowledge and practical operation for the care of children with autism. SUMMARY

[0004] The embodiments of the present application provide a multi-modal emotional interaction system and method to solve the problems of unreliable information sources, lack of personalized content, and the gap between theoretical knowledge and practical operation for the care of children with autism.

[0005] In one aspect, the present application provides a multi-modal emotional interaction system, which includes a client and a server, the client includes a front end and a back end;

[0006] The front end includes:

[0007] The acquisition module is configured to acquire user information, the user information including audio data and image data;

[0008] The output module is configured to display an agent image analyzed based on the user information, the agent image including a virtual scene and character dynamics;

[0009] The back end includes:

[0010] receive a behavior language data sent by a server, the behavior language data being obtained according to user information analysis;

[0011] generate a corresponding virtual human control information according to the behavior voice data;

[0012] a scene control module, configured to manage scene information of an application according to the behavior voice data;

[0013] an interaction module, configured to generate an interactive answer agent image according to the virtual human control information and the scene information;

[0014] The server end comprises:

[0015] an emotion recognition module, configured to analyze user information collected by a front end to obtain recognition information;

[0016] a retrieval module, configured to retrieve retrieval information according to the recognition information;

[0017] a pre-training module, configured to generate training information based on a large language model according to the recognition information and the retrieval information, the training information comprising a question and a dialogue;

[0018] an emotion fusion module, configured to generate the behavior language data according to the recognition information and the training information.

[0019] Further, the server end and the client end communicate through a WebSocket tunnel.

[0020] Further, the back end further comprises a storage module, configured to store user interaction and application data, and log records.

[0021] Further, the collection module is further configured to:

[0022] capture video and audio through a Micro_Camera class, and capture a user's face image using a front camera;

[0023] process the face image using a Texture2D, including resizing, rotating, and converting to a JPEG format, to obtain image data;

[0024] send the image data to the server end in the form of a Base64 string using a WebSocket;

[0025] capture audio from a microphone of the device using a Microphone class, and record the audio as audio data;

[0026] send the audio data to the server end in the form of a Base64 string.

[0027] Further, the collection module is further configured to:

[0028] The audio data is sliced into fixed size to obtain audio slices;

[0029] The audio slices are transmitted in a streaming manner so that the server side restores the audio slices into audio data;

[0030] The image data acquisition action is controlled by using a coroutine, and the next image data is transmitted to the backend for calculation after confirming that the emotional result of the previous image data calculated by the system backend is received, so as to control the sending and receiving of the communication protocol.

[0031] Further, the emotion recognition module is further configured to perform facial feature recognition, speech conversion, text emotion recognition, and audio emotion recognition on the image data and the audio data to obtain recognition information, the recognition information including BML marks obtained by encoding facial expressions, mouth shape actions, and body postures into a behavior markup language (BML), the BML marks being used to control the performance of a virtual character.

[0032] Further, the recognition information further includes an emotional calculation output value, and the emotion recognition module is further configured to:

[0033] perform speech recognition on the audio data by using an emotion model to obtain an emotional label and a first emotional probability, the emotional label including one of negative, neutral, or positive, and the first emotional probability including probabilities of various emotional classifications such as anger, fear, happiness, neutrality, sadness, and surprise;

[0034] perform recognition on the image data by using a lightweight facial expression recognition open source model HSEmotion to obtain a second emotional probability, the second emotional probability including probabilities of various emotional classifications such as anger, fear, happiness, neutrality, sadness, and surprise;

[0035] obtain the emotional calculation output value according to the emotional label, the first emotional probability, and the second emotional probability.

[0036] Further, the generation module is further configured to map the emotional calculation output value to obtain a body language animation of the virtual human control information.

[0037] Further, the generation module is further configured to adjust facial expressions, mouth shape actions, and body postures of the virtual human by using Avatar_Animation and Animation_Controller classes.

[0038] In order to solve the problems of unreliable information sources, lack of personalized content, and gap between theoretical knowledge and practical operation in the current autism child care, the purpose of the present application is to provide a multi-modal emotional interaction system, including a client and a server, the client includes a front end and a back end; the front end includes: a collection module for collecting user information, the user information including audio data and image data; an output module for displaying the agent image obtained by analyzing the user information, the agent image including a virtual scene and a dynamic character; the back end includes: a receiving module for receiving the behavioral language data sent by the server, the behavioral language data being obtained by analyzing the user information; a generating module for generating corresponding virtual person control information according to the behavioral voice data; a scene control module for managing the scene information of the application according to the behavioral voice data; an interaction module for generating the interactive answer agent image according to the virtual person control information and the scene information; the server end includes: an emotion recognition module for analyzing the user information collected by the front end to obtain recognition information; a retrieval module for retrieving the retrieval information according to the recognition information; a pre-training module for generating training information based on a large language model according to the recognition information and the retrieval information, the training information including questions and dialogues; an emotion fusion module for generating the behavioral language data according to the recognition information and the training information.

[0039] Compared with the prior art, the present application has the following advantages:

[0040] Through the combination of the client and the server, a multi-modal emotional interaction assistant is created. In the server, the pre-trained large language model (GPT) or database model (raw) receives the user's query to generate dialogue and voice output. These outputs pass through the emotion recognition module, including facial, voice and text emotion recognition, and the emotion generation module, to finally form the behavior markup language. These will be sent to the client. The client receives the user's information through video and audio input, and then generates video and audio output, including the agent's actions and scenes, etc. That is, the application behavior analysis (ABA) theory is used as the basic framework to guide parents to understand and intervene in the child's behavior, and through the simulation of the scene practice, the parents are guided to understand the details of the intervention behavior. It can provide a structured roadmap for parents to help them understand and manage the child's behavior. At the same time, the theoretical concepts are combined with practical application to improve the parents' ability to provide customized intervention for the child's needs. In this way, the present application will solve the deficiencies of the prior art in standardization, combination of theory and practice, information reliability and individualization, and practicality and ease of learning.

[0041] On the other hand, the present application provides a multi-modal emotional interaction method, which uses the above multi-modal emotional interaction system, including:

[0042] Collecting user information, the user information including audio data and image data;

[0043] analyze the user information to obtain identification information;

[0044] retrieve search information according to the identification information;

[0045] generate training information based on a large language model according to the identification information and the search information, the training information including a question and a dialogue;

[0046] generate behavioral language data according to the identification information and the training information;

[0047] receive the behavioral language data sent by the server end, the behavioral language data being obtained according to the analysis of the user information;

[0048] generate corresponding virtual human control information according to the behavioral language data;

[0049] manage scene information of an application according to the behavioral language data;

[0050] generate an interactive answer agent image including a virtual scene and character dynamics according to the virtual human control information and the scene information;

[0051] display the agent image. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0053] Figure 1 is a schematic diagram of a multi-modal emotional interaction system of an embodiment of the present application.

[0054] Figure 2 is a flowchart of a multi-modal emotional interaction method of another embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0056] Please refer to Figure 1 the multi-modal emotional interaction system 10 of an embodiment of the present application, which includes a client end 11 and a server end 12, the client end 11 including a front end 111 and a back end 112.

[0057] The front end 111 includes a collection module 1111 and an output module 1112. The collection module 1111 is configured to collect user information, which includes audio data and image data.

[0058] As one of the implementations, the collection module 1111 is further configured to:

[0059] capture video and audio through the Micro_Camera class, and capture the facial image of the user using the front camera;

[0060] process the facial image using Texture2D, including resizing, rotating, and converting to JPEG format, to obtain image data;

[0061] send the image data to the server 12 in the form of a Base64 string using WebSocket;

[0062] capture audio from the microphone of the device using the Microphone class, and record the audio as audio data;

[0063] send the audio data to the server 12 in the form of a Base64 string.

[0064] That is, the collection module 1111 can capture video and audio through the `Micro_Camera` class. Micro_Camera is a software class that is responsible for calling and controlling the camera of the device. This class has the function of starting the camera and capturing audio and video data, and can adjust the parameters of the camera (such as resolution, frame rate, etc.) as needed. Through the Micro_Camera class, high-quality audio and video data can be effectively obtained from the camera of the device, which is particularly important for real-time video communication and monitoring systems. For video, we use the front camera to capture the facial image of the user, and then use Texture2D to process it, including resizing, rotating, and converting to JPEG format. The captured image is sent to the server for sentiment analysis through the sending interface in the form of a Base64 string using WebSocket. After the server analyzes, it returns the sentiment data (such as the emotional value of facial expressions) to the Unity client 11.

[0065] For audio, the client 11 uses the `Microphone` class to capture audio from the device's microphone. The recorded audio is saved as an AudioClip and converted to WAV format using the `SavWav` conversion interface, and then converted to a Base64 string for transmission as a binary data stream. Converting audio data to a string format using Base64 encoding protects the integrity of the data during transmission. Base64 encoded data can also be sent through any text transmission protocol, providing high flexibility and compatibility for data exchange over multiple network protocols. This Base64 string is then sent to the server for voice emotion analysis through the `WsClient` send interface.

[0066] To reduce system transmission and operation pressure, the acquisition module 1111 is further used for:

[0067] Slicing the audio data into fixed-size slices to obtain audio slices;

[0068] Transmitting the audio slices in a streaming manner to enable the server side 12 to restore the audio slices to audio data;

[0069] Controlling the acquisition action of image data using a coroutine, and transmitting the next image data to the backend 112 for calculation after confirming that the emotional result of the previous image data calculated by the system backend 112 is received, to control the sending and receiving of the communication protocol.

[0070] The output module 1112 is used to display the agent image obtained according to the user information analysis, and the agent image includes a virtual scene and a dynamic character.

[0071] After the server side 12 is processed, the returned data (including the emotional value of facial expression and the voice analysis result) are returned to the client 11 through the `WsClient` class. The client 11 adjusts the facial expression, mouth shape action and body posture of the virtual character according to these data using the `Avatar_Animation` and `Animation_Controller` classes. These adjustments are made according to the emotional data returned by the server and the voice content to ensure that the performance of the virtual character matches the actual emotional state of the user and the voice content, and finally obtains the agent image, which is displayed to the user through the output module 1112.

[0072] The backend 112 includes a receiving module 1121, a generating module 1122, a scene control module 1123 and an interaction module 1124.

[0073] The receiving module 1121 is used to receive the behavioral language data sent by the server side 12, and the behavioral language data is obtained according to the user information analysis.

[0074] The generating module 1122 is configured to generate corresponding virtual human control information according to the behavioral voice data.

[0075] As one of the embodiments, the generating module 1122 is further configured to use Avatar_Animation and Animation_Controller classes to adjust facial expressions, mouth shape actions and body postures of the virtual human.

[0076] As one of the embodiments, the generating module 1122 is further configured to map the virtual human control information limb language animation according to the emotion calculation output value.

[0077] The scene control module 1123 is configured to manage scene information of the application according to the behavioral voice data.

[0078] The interaction module 1124 is configured to generate an interactive answer agent image according to the virtual human control information and the scene information.

[0079] As one of the embodiments, the backend 112 further includes a storage module configured to store user interaction and application data, and log records.

[0080] The server end 12 includes an emotion recognition module 121, a retrieval module 122, a pre-training module 123 and an emotion fusion module 124.

[0081] The emotion recognition module 121 is configured to analyze the user information collected by the front end 111 to obtain recognition information.

[0082] In the embodiments of the present application, the emotion recognition module 121 can be used for facial feature recognition, speech conversion, text emotion recognition, and audio emotion recognition.

[0083] As one of the embodiments, the emotion recognition module 121 is further configured to perform facial feature recognition, speech conversion, text emotion recognition, and audio emotion recognition on the image data and the audio data to obtain the recognition information, the recognition information including BML marks for encoding facial expressions, mouth shape actions and body postures into behavioral markup language, the BML marks being used to control performances of virtual characters.

[0084] In the embodiments of the present application, the system integrates a pre-trained large language model on the server side 12 for processing user queries and generating dialogues and speech outputs. These outputs are processed by the sentiment recognition module 121 and finally converted into behavior markup language. These animated performances (facial expressions, lip movements and body postures) are encoded into behavior markup language (BML), which is a coding system for describing human behavior and actions. Our system dynamically generates BML tags according to the sentiment recognition results to control the performance of virtual characters, so that the actions and expressions of the characters are synchronized with the emotional state and speech content of the user. Through such a processing flow, a expressive virtual character is realized, which can respond and interact according to the emotional state and speech content of the user, creating a more natural and immersive interactive experience.

[0085] In addition, the recognition information can also include an emotional computing output value, and the sentiment recognition module 121 is further used to:

[0086] performing speech recognition on the audio data using the emotion model to obtain an emotional label and a first emotion probability, the emotional label including one of negative, neutral or positive, and the first emotion probability including probabilities of various emotion classifications such as anger, fear, happiness, neutral, sadness and surprise;

[0087] performing recognition on the image data using a lightweight facial expression recognition open source model HSEmotion to obtain a second emotion probability, the second emotion probability including probabilities of various emotion classifications such as anger, fear, happiness, neutral, sadness and surprise;

[0088] obtaining an emotional computing output value according to the emotional label, the first emotion probability and the second emotion probability.

[0089] The retrieval module 122 is used to retrieve retrieval information according to the recognition information.

[0090] In the embodiments of the present application, the retrieval module 122 retrieves retrieval information according to the recognition information, and the retrieval information can include follow-up questions and dialogues.

[0091] The pre-training module 123 is used to generate training information based on a large language model according to the recognition information and the retrieval information, and the training information includes questions and dialogues.

[0092] The sentiment fusion module 124 is used to generate behavior language data according to the recognition information and the training information.

[0093] In the embodiments of the present application, the sentiment fusion module 124 fuses different sentiment recognition results and generates behavior language data, and the behavior language data includes behavior markup language.

[0094] As one of the embodiments, the server side 12 and the client side 11 communicate through a WebSocket tunnel. The user's interaction with the front end 111 of the client side 11 is realized through video and audio input, and the front end 111 transmits the user input to the server side 12 and the back end 112 for processing. The back end 112 processes the user input and generates corresponding actions and feedback through scene control, question generator, and audio, body gesture, and facial expression modules, and then the front end 111 displays them to the user through video and audio output.

[0095] The existing technology has a number of problems in the field of autism children's home intervention, including:

[0096] 1. Lack of standardization of intervention measures: existing intervention methods lack standardized implementation, and different implementers may lead to inconsistent results.

[0097] 2. Disconnection between theoretical knowledge and practical application: in the existing technology, theoretical knowledge is often insufficient to guide practical intervention practice, especially without comprehensive and detailed guidance.

[0098] 3. Reliability and individualization of information sources: parents currently rely mainly on social media and other platforms to obtain intervention information, but the reliability and content quality of these information sources vary, and there is a lack of personalized content for individual families and children.

[0099] 4. Limitations of practicality and ease of learning: existing intervention strategies are often not practical or easy to understand, especially for parents of different levels who need simpler and easier-to-remember effective guidance.

[0100] The multi-modal emotional interaction system of the embodiments of the present invention has significant improvements in many aspects, including:

[0101] 1. More professional content planning: compared with general search engines and platforms, the present invention provides more carefully planned content, reducing users' concerns about the authenticity and relevance of the content.

[0102] 2. More flexible accessibility: the present invention alleviates the time and space constraints of consulting with professional educators, providing flexible learning possibilities.

[0103] 3. Efficiency improvement: the present invention promotes the absorption of comprehensive knowledge through repeated learning and practical case analysis, thereby improving efficiency.

[0104] 4. Emotional support: the present invention provides emotional comfort to novice parents through the role of virtual agents, reducing negative emotions such as anxiety.

[0105] 5. Enhanced trust and willingness to use: Due to the aforementioned advantages, user trust in the system and willingness to use it significantly increase.

[0106] 6. Personalized intelligent QA response: The invention can provide refined intelligent question and answer feedback based on the personalized needs of users, making the user experience more personalized and accurate.

[0107] 7. Fine-tuned case selection for autonomous learning: The invention carefully plans cases in the autonomous learning module to meet the needs of different parents and children, enhancing the relevance and practicality of educational content.

[0108] 8. Continuous optimization of database and user interface (UI): Through interaction with parents and clinicians, the invention continuously optimizes and expands in terms of content and interface design, improving user experience.

[0109] 9. Clarity of knowledge framework: The invention enhances the clarity of the knowledge framework by providing structured intervention steps, enabling parents to more effectively understand and manage their children's behavior.

[0110] 10. Enhanced ease of learning and intuitiveness: The invention emphasizes providing intuitive and easy-to-understand guidance content, making it easier for parents of various professional levels to adopt intervention methods through imitation learning and quick practice.

[0111] See Figure 2 Another embodiment of the invention is a multi-modal emotional interaction method using the above multi-modal emotional interaction system, which includes:

[0112] Step 201, collecting user information, including audio data and image data.

[0113] Step 202, analyzing user information to obtain identification information.

[0114] Step 203, retrieving search information according to the identification information.

[0115] Step 204, generating training information based on a large language model according to the identification information and the search information, the training information including questions and dialogues.

[0116] Step 205, generating behavior language data according to the identification information and the training information.

[0117] Step 206, receiving server-side sent behavior language data, which is obtained by analyzing user information.

[0118] Step 207, generating corresponding virtual human control information according to the behavior voice data.

[0119] Step 208, managing application scenario information according to the behavior voice data.

[0120] Step 209, generating an interactive answer agent image according to the virtual human control information and the scene information, the agent image including a virtual scene and character dynamics.

[0121] Step 210, displaying the agent image.

[0122] As one of the embodiments of the present application, the client (Unity end) collects audio and image data of the user through the microphone and camera respectively. In the data collection process, the timestamp when the user starts using the microphone is taken as the start of the collection period, and the timestamp when the user stops using the microphone is taken as the end of the collection period. The audio data is transmitted in a streaming manner, which is limited by the data size of a single transmission protocol. The system front end slices the audio data into fixed size, sends it to the system back end, and performs streaming segmental speech recognition to reduce the delay. When the audio data stops collecting, the system back end splices the segmented audio data and the corresponding speech recognition result to obtain the text result of speech recognition, and saves the audio data. For image data, the system front end uses coroutine to control the collection action of image data. After the system front end confirms that the emotional result of the previous image data calculated by the system back end is received, the next image data is transmitted to the back end for calculation to control the sending and receiving of the communication protocol. These inputs are transmitted to the server end.

[0123] The server-side processes the input, runs the emotion recognition module, and transmits the generated behavior markup language back to the Unity side. On the server side, the system backend performs speech recognition on the received complete audio data and uses a dictionary-based three-class emotion model on the text result after speech recognition to obtain the "negative", "neutral", or "positive" emotion label. For speech emotion recognition, a Resnet34-based speech emotion model is used, and the final output is the probability of "anger", "fear", "joy", "neutral", "sadness", and "surprise" emotion classification. For facial expression recognition, a lightweight facial expression recognition open-source model HSEmotion is used to output the probability of "anger", "fear", "joy", "neutral", "sadness", and "surprise" emotion classification on the images collected during the collection process, and a data structure is used to accumulate the values of each probability during the collection process, and finally the number of pictures is averaged in space and time. In order to fuse the six-dimensional speech emotion and facial expression with the text emotion, a certain weighting coefficient is used to convert the six emotions into "positive" and "negative" calculation values, such as "positive calculation value" equals to "1.2*joy+0.4*neutral+1*surprise", while "negative calculation value" equals to "0.8*anger+0.6*fear+1.2*sadness". Finally, the calculation value of "positive" minus the calculation value of "negative" is used as the single-modal emotion output of speech emotion / facial expression. In terms of emotion fusion, different weights are linearly weighted for text, speech, and face to obtain the final emotion calculation output value and transmit it to the Unity side.

[0124] The Unity side generates corresponding actions and feedback according to the received behavior markup language and outputs them to the user through video and audio. After the server-side processing is completed, a sentiment value is returned, which is the 1:1 weighted average of the text emotion and expression emotion, ranging from -1 to 1. According to this integrated value, different expression animations are used to reflect the emotional state of the character. The Avatar_Animation class maps these data to a series of predefined facial expressions (such as "Sympathy", "Sad", "Serious", etc.) and mouth movements after obtaining the facial expression analysis results from the server. There are 9 types of expressions, including 4 positive expressions, 4 negative expressions, and 1 neutral expression. Each expression corresponds to a specific value range:

[0125] Most negative expression: integrated_val ∈ [-1, -0.875]

[0126] Second negative expression: integrated_val ∈ (-0.875, -0.625]

[0127] Mildly negative expression: integrated_val ∈ (-0.625, -0.375]

[0128] Mildly negative expression: integrated_val ∈ (-0.375, -0.125]

[0129] Neutral expression: integrated_val ∈ (-0.125, 0.125]

[0130] Mildly positive expression: integrated_val ∈ [0.125, 0.375]

[0131] Mildly positive expression: integrated_val ∈ [0.375, 0.625]

[0132] Mildly positive expression: integrated_val ∈ [0.625, 0.875]

[0133] Most positive expression: integrated_val ∈ [0.875, 1]

[0134] These expression animations will be played dynamically according to the integrated value, in order to express the emotional state of the character. In addition, the mouth animation of the character will continue to play, simulating the action of speaking, which is mainly manifested as the opening and closing of the mouth. In addition, these emotional data can also be mapped to a series of body gestures and gesture animations through the Animation_Controller class, and the body language animation will randomly select five of them to play, increasing the naturalness and diversity of expression. To enhance the expressiveness of the character. At the same time, the server will generate the required audio and return the streaming audio to unity, which is responsible for processing and playing the audio through the PlaySound class in unity.

[0135] Testing and feedback:

[0136] User testing was conducted in a laboratory environment, with each user session lasting approximately 30 minutes.

[0137] The test included a pre-test involving three scenario-based queries, followed by the participant interacting with the intelligent dialogue and autonomous practice modules.

[0138] The session ended with an individual interview, and the content of the test came from the integrated database of the system.

[0139] In addition, other personnel can directly use the interface provided by the research project to achieve targeted feedback answers with emotional values.

[0140] Those skilled in the art can clearly understand the unit and algorithm steps of each example described in combination with the embodiments disclosed in the embodiments of the present application, which can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multimodal emotional interaction system, characterized in that, It includes a client and a server, wherein the client includes a front-end and a back-end; The front end includes: The acquisition module is used to collect user information, which includes audio data and image data; The output module is used to display the proxy image obtained by analyzing the user information, the proxy image including virtual scenes and character dynamics; The backend includes: A receiving module is used to receive behavioral language data sent by the server, the behavioral language data being obtained by analyzing the user information; The generation module is used to generate corresponding virtual human control information based on the behavioral language data; The scene control module is used to manage the scene information of the application based on the behavioral language data; The interaction module is used to generate the proxy image for interactive responses based on the virtual human control information and the scene information; The server includes: The emotion recognition module is used to analyze the user information collected by the front end to obtain recognition information; The retrieval module is used to retrieve retrieval information based on the identification information; The pre-training module is used to generate training information based on the recognition information and retrieval information, according to a large language model. The training information includes questions and dialogues. The emotion fusion module is used to generate the behavioral language data based on the recognition information and the training information.

2. The multimodal emotional interaction system according to claim 1, characterized in that, The server and the client communicate via a WebSocket tunnel.

3. The multimodal emotional interaction system according to claim 1, characterized in that, The backend also includes a storage module for storing user interaction and application data, as well as log records.

4. The multimodal emotional interaction system according to claim 1, characterized in that, The acquisition module is also used for: Capture video and audio using the Micro_Camera class, and capture the user's facial image using the front-facing camera; The facial image is processed using Texture2D, including resizing, rotating, and converting to JPEG format, to obtain the image data. The image data is sent to the server in Base64 string format using WebSocket; Use the Microphone class to capture audio from the device's microphone and record it as audio data; The audio data is sent to the server in Base64 string format.

5. The multimodal emotional interaction system according to claim 4, characterized in that, The acquisition module is also used for: The audio data is sliced ​​into fixed-size segments to obtain audio slices; The audio slices are transmitted in a streaming manner so that the server can restore the audio slices into the audio data. The coroutine is used to control the acquisition of image data. Once the sentiment result of the previous image data calculated by the system backend is confirmed to be received, the next image data is transmitted to the backend for calculation, thereby controlling the sending and receiving of the communication protocol.

6. The multimodal emotional interaction system according to claim 4, characterized in that, The emotion recognition module is also used to perform facial feature recognition, speech conversion, text emotion recognition, and audio emotion recognition on the image data and the audio data to obtain the recognition information. The recognition information includes BML tags that encode facial expressions, lip movements, and body postures into behavior markup language. The BML tags are used to control the performance of the virtual character.

7. The multimodal emotional interaction system according to claim 6, characterized in that, The recognition information also includes emotion calculation output values, and the emotion recognition module is further used for: The audio data is used to perform speech recognition using an emotion model to obtain emotion labels and a first emotion probability. The emotion label includes one of negative, neutral, or positive. The first emotion probability includes the probability of multiple emotion categories such as anger, fear, happiness, neutrality, sadness, and surprise. The image data is identified using the lightweight open-source facial expression recognition model HSEmotion to obtain a second emotion probability, which includes the probabilities of multiple emotion categories such as anger, fear, happiness, neutrality, sadness, and surprise. The emotion calculation output value is obtained based on the emotion tag, the first emotion probability, and the second emotion probability.

8. The multimodal emotional interaction system according to claim 7, characterized in that, The generation module is also used to map the body language animation of the virtual human control information based on the emotion calculation output value.

9. The multimodal emotional interaction system according to claim 8, characterized in that, The generation module is also used to adjust the virtual human's facial expressions, lip movements, and body posture using the Avatar_Animation and Animation_Controller classes.

10. A multimodal emotion interaction method, characterized in that, The multimodal emotion interaction system according to any one of claims 1-9 includes: Collect user information, which includes audio data and image data; Analyze the user information to obtain identification information; The search information is obtained based on the identification information; Based on the identification information and retrieval information, training information is generated based on a large language model, and the training information includes questions and dialogues. The behavioral language data is generated based on the recognition information and the training information; Receive behavioral language data sent by the server, the behavioral language data being obtained through analysis of the user information; Generate corresponding virtual human control information based on the behavioral language data; Based on the scenario information of the behavioral language data management application; Based on the virtual human control information and the scene information, an interactive response proxy image is generated, wherein the proxy image includes a virtual scene and human dynamics; The proxy image is displayed.

Images