Emotion recognition method based on large model and related device

Abstract

The invention relates to the field of data processing, in particular to an emotion recognition method based on a large model and a related device. The method comprises the following steps: firstly, acquiring to-be-processed multi-modal data; the multi-modal data at least comprises text information, audio information and image information input by a user; the image information at least comprises a facial expression image and / or a limb action image. And inputting the multi-modal data into an emotion recognition model for emotion state analysis to obtain a predicted emotion state corresponding to the user. And finally, performing confidence evaluation based on the predicted emotional state, and generating a user emotion recognition result based on the predicted emotional state and a corresponding evaluation result. According to the method, richer emotion details are captured from the multi-modal data through the emotion recognition model, cross-modal emotion intelligent recognition is realized, the accuracy of emotion recognition is further improved by combining confidence evaluation, the emotion recognition efficiency is improved, and the application range of emotion recognition is expanded.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to an emotion recognition method based on a large model and related devices. Background Art

[0002] Emotion recognition is an important research direction in the field of artificial intelligence and computer vision. It can be applied to many scenarios, such as intelligent assistants, sentiment analysis, mental health diagnosis, etc. Traditional emotion recognition methods are mainly based on a single information source, such as speech, text or video images, but these methods often have difficulty in accurately capturing the complex emotional expressions of humans.

[0003] In the related art, emotion recognition is usually performed using only voice information or text information. The effective information contained in other forms of information (such as image information) is ignored. The lack of such information makes the accuracy of emotion recognition low. Therefore, it is urgent to propose a new technical solution to solve at least one of the above technical solutions. Summary of the invention

[0004] In response to the technical problems existing in the related art, the present application provides an emotion recognition method and related devices based on a large model to solve at least one technical problem existing in the related art.

[0005] In a first aspect, an embodiment of the present application provides an emotion recognition method based on a large model, the method comprising:

[0006] Acquire multimodal data to be processed; the multimodal data at least includes: text information, audio information, and image information input by a user; the image information at least includes: facial expression images and / or body movement images;

[0007] Inputting the multimodal data into an emotion recognition model to perform emotion state analysis to obtain a predicted emotion state corresponding to the user;

[0008] A confidence assessment is performed based on the predicted emotional state, and a user emotion recognition result is generated based on the predicted emotional state and the corresponding assessment result; the user emotion recognition result includes at least: an attention heat map for displaying the user's emotional state from multiple dimensions, and / or visual auxiliary analysis information; the auxiliary analysis information includes at least: voice emotion analysis, text emotion analysis, and facial expression emotion analysis.

[0009] In a second aspect, an embodiment of the present application provides an emotion recognition device based on a large model, the device comprising at least the following units:

[0010] An acquisition unit is configured to acquire multimodal data to be processed; the multimodal data at least includes: text information, audio information, and image information input by a user; the image information at least includes: facial expression images and / or body movement images;

[0011] An analysis unit configured to input the multimodal data into an emotion recognition model to perform emotion state analysis and obtain a predicted emotion state corresponding to the user;

[0012] The output unit is configured to perform a confidence assessment based on the predicted emotional state, and generate a user emotion recognition result based on the predicted emotional state and the corresponding assessment result; the user emotion recognition result includes at least: an attention heat map for displaying the user's emotional state from multiple dimensions, and / or visual auxiliary analysis information; the auxiliary analysis information includes at least: voice emotion analysis, text emotion analysis, and facial expression emotion analysis.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0014] at least one processor, memory, and input-output unit;

[0015] The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the large model-based emotion recognition method of the first aspect.

[0016] In a fourth aspect, a computer-readable storage medium is provided, which includes instructions, and when the instructions are executed on a computer, the computer executes the large model-based emotion recognition method of the first aspect.

[0017] The beneficial effects of the embodiments of the present application are: first, the multimodal data to be processed is obtained; the multimodal data at least includes: text information, audio information, and image information input by the user; the image information at least includes: facial expression images and / or body movement images. Then, the multimodal data is input into the emotion recognition model for emotional state analysis to obtain the predicted emotional state corresponding to the user. Finally, a confidence assessment is performed based on the predicted emotional state, and a user emotion recognition result is generated based on the predicted emotional state and the corresponding evaluation result. The user emotion recognition result at least includes: an attention heat map for displaying the user's emotional state from multiple dimensions, and / or visual auxiliary analysis information; the auxiliary analysis information at least includes: voice emotion analysis, text emotion analysis, and expression emotion analysis. In the embodiments of the present application, the emotion recognition model is used to capture richer emotional details from the multimodal data, realize cross-modal emotional intelligent recognition, and further improve the accuracy of emotion recognition in combination with confidence assessment, improve the efficiency of emotion recognition, and expand the application scope of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of an emotion recognition method based on a large model according to an embodiment of the present application;

[0019] Figure 2 is a structural schematic diagram of an emotion recognition device based on a large model according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0021] Figure 4 It is a structural schematic diagram of a medium device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0023] In order to solve at least one technical problem in the related art, the embodiment of the present application provides an emotion recognition method based on a large model and a related device. In this technical solution, first, the multimodal data to be processed is obtained; the multimodal data at least includes: text information, audio information, and image information input by the user; the image information at least includes: facial expression images and / or body movement images. Then, the multimodal data is input into the emotion recognition model for emotional state analysis to obtain the corresponding predicted emotional state of the user. Finally, a confidence assessment is performed based on the predicted emotional state, and a user emotion recognition result is generated based on the predicted emotional state and the corresponding evaluation result. The user emotion recognition result includes at least: an attention heat map for displaying the user's emotional state from multiple dimensions, and / or visual auxiliary analysis information; the auxiliary analysis information includes at least: speech emotion analysis, text emotion analysis, and expression emotion analysis.

[0024] In the technical solution provided by the present application, on the one hand, by acquiring multimodal data such as text information, audio information and image information (such as facial expression images and / or body movement images) input by the user, it is no longer limited to a single modality to analyze user emotions. For example, relying solely on text may not accurately judge the user's true emotions. If combined with audio information of tone and intonation and image information such as facial expressions, the user's actual emotional state can be captured more comprehensively from multiple angles, reducing the emotion recognition bias caused by insufficient or inaccurate single modality information. In different application scenarios, users have different emphases on expressing their emotions. In some online communication scenarios, text may be the majority, while audio and image information is more critical in video call scenarios. This method can adapt to these diverse scenarios. No matter which modality data is dominant, it can be comprehensively analyzed, so that it can be widely used in various occasions such as psychological counseling, online customer service, social media interaction, remote conferencing, etc., to improve the universality of emotion recognition.

[0025] Secondly, multimodal data is input into a special emotion recognition model for emotional state analysis. With the powerful learning and feature extraction capabilities of the model, the deep emotional features in different modal data and the relationship between modalities can be excavated, and relatively accurate predictions of emotional states can be obtained. For example, it can distinguish subtle emotional differences, such as in the "happy" emotion category, further determine whether it is excited happiness or peaceful happiness, etc., to improve the accuracy of emotion recognition. Considering that emotions are often not static and will change dynamically with time and other factors, the model can combine relevant ideas such as time series in the analysis process (such as capturing the dynamic transfer rules between emotions mentioned in the article, etc.), better insight into the evolution of emotions in continuous scenes, and better grasp the short-term fluctuations and long-term trends of user emotions, which is very helpful for understanding the emotional trends of users in a longer communication process.

[0026] Thirdly, confidence assessment based on predicted emotional states can provide a reliable basis for the final emotion recognition results. By evaluating uncertainty through methods such as Bayesian deep learning and quantifying reliability using the evidence theory framework, users can clearly understand the credibility of each emotion recognition result and avoid blindly relying on inaccurate prediction results, especially in some key decision-making scenarios (such as providing targeted services based on user emotions). It is more instructive. The generated user emotion recognition results include attention heat maps and visual auxiliary analysis information (such as voice emotion analysis, text emotion analysis, expression emotion analysis, etc.) used to display the user's emotional state from multiple dimensions, making the emotion recognition results more intuitive and easy to understand. For example, the attention heat map can clearly show which factors play a key role in emotional judgment in different modes or at different times; the visualized emotional analysis of various types allows users to see at a glance the user's emotional situation reflected in different aspects, which is convenient for subsequent more in-depth analysis and targeted response measures.

[0027] In summary, the emotion recognition method based on the large model of this solution can identify and display user emotions more comprehensively, accurately, reliably and intuitively, which plays a positive and important role in improving user experience, optimizing service quality and assisting various business decisions that require understanding user emotions. This solution captures richer emotional details from multimodal data through the emotion recognition model, realizes cross-modal intelligent emotion recognition, and combines confidence assessment to further improve the accuracy of emotion recognition, improve emotion recognition efficiency, and expand the application scope of emotion recognition.

[0028] The technical solution of the present application and the emotion recognition scheme based on the large model provided in the embodiment of the present application can also be executed by an electronic device, which can be a server, a server cluster, or a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a dedicated device (such as a dedicated terminal device with an emotion recognition method system based on a large model, etc.). These electronic devices can also be equipped with the chips or other hardware processing units introduced in the above embodiments. Alternatively, these electronic devices can also be installed with a service program for executing the emotion recognition scheme based on the large model.

[0029] Figure 1 A flowchart of an emotion recognition method based on a large model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0030] 101, obtaining multimodal data to be processed;

[0031] 102, inputting the multimodal data into an emotion recognition model to perform emotion state analysis to obtain a predicted emotion state corresponding to the user;

[0032] 103 , performing confidence evaluation based on the predicted emotional state, and generating a user emotion recognition result based on the predicted emotional state and the corresponding evaluation result.

[0033] In an embodiment of the present application, the image information includes at least: facial expression images and / or body movement images. Specifically, the image information has a clear coverage, which includes at least two categories: facial expression images and body movement images. Facial expression images can intuitively present a person's emotional states such as joy, anger, sorrow, and happiness. For example, expressions such as smiling, frowning, and staring can reflect different moods. Body movement images are also critical. For example, stretched body movements may imply relaxed and pleasant emotions, while curled or tense body postures may reflect anxiety, uneasiness, and other emotions. By incorporating these facial expression images and / or body movement images into the category of image information, it can provide important visual basis for subsequent related work such as emotion recognition based on multimodal data, and assist in more comprehensive and accurate grasp of the user's emotional state.

[0034] In an embodiment of the present application, the user emotion recognition result includes at least: an attention heat map for displaying the user's emotional state from multiple dimensions, and / or visual auxiliary analysis information. Specifically, the attention heat map is a way to display the user's emotional state from multiple dimensions. It presents the key areas or key information that the model focuses on in the process of emotion recognition in the form of a heat map by means of visualization. For example, when comprehensively analyzing multimodal data, for the text part, the heat map may highlight those words that play a key role in judging emotions; in terms of images, it will focus on showing the facial features (such as eyes, mouth, etc.) that best reflect emotions in facial expressions or the most representative parts of body movements. From different modes and different feature dimensions, the importance of each part to the overall emotional state judgment is intuitively displayed using changes in color depth, etc., so that users can quickly understand the focus of the model during emotion analysis and the contribution of each part of the information.

[0035] Visual auxiliary analysis information is content that displays and supplements the user emotion recognition results from another perspective, and at least covers voice emotion analysis, text emotion analysis, and facial expression emotion analysis. Further optionally, the auxiliary analysis information at least includes: voice emotion analysis, text emotion analysis, and facial expression emotion analysis.

[0036] For example, by professionally analyzing the collected user audio information, we can extract acoustic features such as pitch, speaking speed, and timbre, and then determine the emotional color contained therein. For example, a higher pitch and a faster speaking speed may mean excitement, while a low and slow voice may imply sadness or depression. The emotional analysis results reflected in these voices are presented in a visual form, such as using a bar chart to show the strength of voice features corresponding to different emotional categories, or using a line chart to show the trend of emotional changes in a voice over time, so that the user's voice emotional state is clear at a glance.

[0037] For example, natural language processing technology is used to analyze the words, sentence structure, and semantics in the text information input by the user to identify the emotional tendency conveyed by the text. For example, texts containing more positive words are more likely to reflect positive emotions, while texts containing a large number of negative words often reflect negative emotions. When presenting visually, a word cloud chart can be used to display the emotional keywords that appear frequently in the text, or a pie chart can be used to display the proportion of different emotional categories (such as positive, negative, and neutral) in the text, to help users clearly grasp the emotional status at the text level.

[0038] For example, based on the facial expression images mentioned above, the morphological changes of the facial features (such as a raised mouth corner indicates happiness, a frown indicates sadness, etc.) and the combined features of the overall facial expression are analyzed to determine the corresponding emotional state. Through visualization, such as displaying the emotional classification corresponding to different expressions with a picture set, or making a dynamic visualization chart of the emotional changes of expressions (showing the emotional transformation of expressions over time, etc.), the emotional trend of the user reflected by the facial expression is intuitively presented, which helps to more comprehensively understand the user's emotional expression.

[0039] By providing such user emotion recognition results, whether it is an attention heat map or visual auxiliary analysis information, users can understand the user's emotional state from multiple dimensions in an intuitive and clear manner, providing a strong basis for subsequent decision-making, interaction, or provision of targeted services.

[0040] As an optional embodiment, in 101, a text input module is used to receive and format text data to provide semantic input for subsequent processing. The voice input module captures and digitizes audio signals and converts them into acoustic data that can be processed by the system. The image input module acquires and standardizes visual data and provides facial expression and body language information. Exemplarily, multimodal data is received through three parallel input channels. In the preprocessing layer, the text data undergoes operations such as word segmentation, denoising, and standardization to clean and normalize the text data; the voice signal performs noise reduction, segmentation, and normalization on the audio signal to eliminate interference, extract key acoustic features, and use an adaptive segmentation algorithm to divide the voice segments; the image data uses a deep learning framework for denoising, illumination normalization, posture correction, and size unification. Finally, the emotional feature information corresponding to the three modalities is annotated to achieve intercommunication between the three modalities.

[0041] Specifically, in 101, the text input module undertakes the important task of receiving text data. It can connect to text information from multiple sources, such as text content directly entered by users in the interactive interface, such as sentences entered in the chat window, comments posted in the comment area, etc.; it can also receive text content imports from other documents and files, and is compatible with text input in different formats to ensure that text information related to the user is obtained as comprehensively as possible.

[0042] In 101, after receiving the text data, the text input module will format it. For example, the encoding format of the text is unified to avoid problems such as garbled characters due to inconsistent encoding that affect subsequent analysis; the punctuation marks of the text are standardized to make the sentence structure clearer, which is convenient for the subsequent natural language processing algorithm to understand its semantics; some extra spaces, line breaks and other irrelevant format elements are removed to organize the text into a neat and standardized format, so as to provide clear, accurate and easy-to-analyze semantic input for subsequent processing, and ensure that subsequent operations such as text-based sentiment analysis can be carried out smoothly.

[0043] In 101, the voice input module has the ability to capture audio signals, and it can obtain audio information in a variety of ways. For example, it can use the microphone built into the device, such as a mobile phone microphone, a computer microphone, etc., to collect the sound signals generated by the user's speech in real time; it can also receive the audio stream input by an external audio device, such as an audio file recorded by an external professional recording device, etc., to ensure that it can widely collect audio signals containing user voices generated in different scenarios.

[0044] In 101, after capturing the audio signal, the voice input module will digitize these analog audio signals. Through specific sampling, quantization and other technologies, the continuous audio waveform is converted into a discrete digital signal so that it can be recognized and processed by the computer system. After that, these digitized audio signals are further converted into acoustic data that can be processed by the system. This may involve extracting some basic acoustic features of the audio, such as Mel-frequency cepstral coefficients (MFCC), pitch, volume, etc., and converting the audio information into a specific data format suitable for subsequent voice emotion analysis, fusion with other modal data and other operations, laying the foundation for comprehensive emotion recognition work.

[0045] In 101, the image input module is responsible for obtaining visual data, and its acquisition methods are also diverse. On the one hand, it can capture images in real time through camera devices, such as using the front camera of a mobile phone to capture images of the user's facial expressions and body movements, or using an external camera on a computer to obtain images of people in a video conference scene, etc. On the other hand, it can also receive stored image files, such as user photos selected from an album, key frame images captured from a video, etc., to ensure that sufficient visual data related to the user's emotional expression can be obtained.

[0046] In 101, after acquiring the visual data, the image input module performs standardization operations on these image data. For example, the image size is unified to meet the requirements of subsequent model processing to avoid the impact of large image size differences on analysis efficiency and accuracy; the color mode of the image is adjusted to ensure the consistency and accuracy of color information; the image is normalized to keep the pixel value of the image in an appropriate range. After these standardized processes, the image data can better provide key information such as facial expressions and body language, which is convenient for subsequent sentiment analysis based on image features and multimodal fusion.

[0047] By performing their respective functions, these three modules can effectively process the multimodal data of text, voice and image, providing an accurate, standardized and appropriate data foundation for the entire emotion recognition system based on multimodal data, ensuring that all subsequent links can proceed smoothly and ultimately achieving accurate user emotion recognition.

[0048] As an optional embodiment, in 102, the multimodal data is input into an emotion recognition model to perform an emotion state analysis to obtain a predicted emotion state corresponding to the user, including:

[0049] Extracting corresponding emotion feature information under multimodality from the multimodal data; performing multimodal fusion on the emotion feature information to obtain emotion fusion features; inputting the emotion fusion features into an emotion recognition model to perform emotion state analysis to obtain the user's corresponding predicted emotion state.

[0050] In an embodiment of the present application, the emotion feature information includes at least one of the following: text emotion feature, acoustic emotion feature, and visual emotion feature.

[0051] In this optional embodiment, processing multimodal data through a series of steps to obtain the predicted emotional state of the user has many technical effects. First, emotional feature information including text emotional features, acoustic emotional features, visual emotional features, etc. is extracted from multimodal data, which makes it possible to fully explore the emotional content contained in different modal data and avoid relying on a single modality and missing important emotional clues. For example, text emotional features can show emotional tendencies from aspects such as word semantics, acoustic emotional features can reflect emotional states with the help of voice pitch, speech speed, etc., and visual emotional features can convey emotional information through facial expressions and body movements. Combined, it can capture the user's true emotions more completely.

[0052] Next, these emotional feature information are fused multimodally to obtain emotional fusion features. This can break the boundaries between different modalities, allowing the emotional features of each modality to complement and synergize each other, so that the fused features contain richer and more comprehensive emotional information, better reflect the performance of user emotions in different dimensions, and enhance the accuracy of the overall grasp of user emotions.

[0053] Finally, the emotion fusion features are input into the emotion recognition model for emotion state analysis to obtain the predicted emotion state. With the help of the powerful learning and analysis capabilities of the emotion recognition model, based on the comprehensive and integrated features after fusion, the user's specific emotional state can be accurately analyzed, the subtle differences between different emotions can be distinguished, and the dynamic changes of emotions over time and other factors can be taken into account, thereby providing a reliable and accurate foundation for subsequent confidence assessment and generating comprehensive user emotion recognition results, which helps to achieve more scientific and accurate recognition and presentation of user emotions, and has important value in many application scenarios that require understanding user emotions.

[0054] The following introduces the implementation method of extracting the corresponding multi-modal emotional feature information from the multi-modal data in the above steps from different modes.

[0055] For text information, the above step of extracting the corresponding emotion feature information under multimodality from the multimodal data can be implemented as follows:

[0056] A multi-level text sentiment feature vector is extracted from the text information through a generalized linear model (GLM); the text sentiment feature vector includes at least: a word sentiment feature vector at the word level, a sentence sentiment feature vector at the sentence level, and a paragraph sentiment feature vector at the context paragraph level; multi-task learning is performed on the text sentiment feature vector to obtain the micro-emotion change features implicit in the text sentiment feature vector; the micro-emotion change features include at least: modal particle features and implicit expression features; the text sentiment feature vector and the micro-emotion change features are used as the text sentiment features.

[0057] Specifically, the generalized linear model (GLM) is used to mine the emotional features in text information, and a multi-level text emotional feature vector is constructed through its powerful modeling capabilities. Specifically, it covers the word-level emotional feature vector, which can capture the emotional color contained in a single word. For example, words like "joy" and "sadness" have obvious emotional tendencies, and their corresponding vectors can reflect this emotional attribute; the sentence-level emotional feature vector focuses on the emotion conveyed by the entire sentence, and comprehensively considers the collocation of words in the sentence, grammatical structure, etc. to judge whether the sentence as a whole is positive, negative or neutral emotional state; there is also the context-level paragraph emotional feature vector, starting from a more macro level of the paragraph, considering the logical relationship between paragraphs, the context of the text, and other factors, to grasp the emotional trend of the entire text.

[0058] Then, based on the constructed text sentiment feature vector, multi-task learning is further carried out. This process aims to discover the subtle emotional change features hidden in the text sentiment feature vector. These subtle features are critical for accurately grasping the text sentiment. These include modal particles, such as "ah", "ya", "ne", etc., which can convey different emotions in different contexts. They may make the emotions softer and more intimate after being added to declarative sentences, or enhance the tone in exclamatory sentences to highlight strong emotions; there are also implicit expression features, such as the emotions contained in some rhetorical devices such as metaphors and symbols, such as using "dark days" to allude to difficult and unpleasant times. Through multi-task learning, these easily overlooked but important micro-emotional change features are excavated, and together with the previous text sentiment feature vector, they constitute a complete text sentiment feature, thereby reflecting the emotional information carried by the text in a more comprehensive and detailed manner.

[0059] For audio information, pre-trained models such as Wav2Vec and HuBERT are used to obtain multi-dimensional acoustic features, including acoustic parameters such as fundamental frequency, formant, energy envelope, spectral centroid, and prosodic features such as pitch curve, speech rate, and pauses. As an optional embodiment, the above steps of extracting the corresponding emotional feature information under multimodal conditions from the multimodal data can be implemented as follows:

[0060] The audio information is converted into an acoustic physical vector through the Wav2Vec model; the acoustic physical vector includes at least: spectral features and / or prosodic features; the spectral features include at least: Mel-frequency cepstral coefficients MFCC, spectral centroid feature information; the prosodic features include at least: pitch curve features, speech rate change features, pause features, and breathing features; the acoustic physical vector and the audio information are input into the HuBERT model to identify the emotional clues implicit in the user's voiceprint and obtain the acoustic emotional features.

[0061] Specifically, for audio information, the Wav2Vec model is first used to convert it to obtain an acoustic physical vector. This vector contains important acoustic information such as spectral features and rhythmic features. In terms of spectral features, there are Mel-frequency cepstral coefficients (MFCC), which can simulate the human ear's perception of sounds of different frequencies and reflect the characteristics of audio in the frequency dimension, as well as spectral centroid feature information, which reflects the center of gravity position of the sound spectrum. Rhythmic features include pitch curve features, that is, the curve formed by the change in the pitch of the sound, which can intuitively reflect the speaker's emotional changes. For example, the pitch often rises when excited, the speed of speech changes can reflect the fast and slow rhythm of speaking, the pause feature can imply thinking, emphasis, etc., and the breathing feature is also a reflection of the speaking rhythm and emotional state. These features together constitute the acoustic physical vector, presenting the basic characteristics of audio from an acoustic perspective.

[0062] Then, the acoustic physical vector obtained by the Wav2Vec model and the original audio information are input into the HuBERT model. With its advantages in speech processing, the HuBERT model can deeply analyze this information and identify the emotional clues hidden in the user's voiceprint. For example, when angry, the voiceprint may have specific frequency and rhythm changes, while when happy, it will show different characteristics. Finally, the acoustic emotional features are obtained to accurately capture the emotional content contained in the audio information.

[0063] It is further necessary to understand that when processing audio information to extract emotional features, related technologies may often only focus on some acoustic features, such as only focusing on one or two parameters in the spectral features, or simply considering a few prosodic features such as pitch and speaking rate. The embodiment of the present application first uses the Wav2Vec model to comprehensively obtain acoustic physical vectors including spectral features such as Mel-frequency cepstral coefficients (MFCCs), spectral centroid feature information, and prosodic features such as pitch curve features, speaking rate change features, pause features, and breathing features, and describes the basic characteristics of audio information from multiple dimensions, without missing any acoustic details that may contain emotional clues. In comparison, it achieves a more comprehensive and detailed audio feature extraction, laying a more solid foundation for subsequent accurate emotional analysis.

[0064] When further using the HuBERT model to identify emotional clues, it not only relies on the acoustic physical vectors extracted by the Wav2Vec model, but also inputs the original audio information. The related technology may only perform subsequent processing based on the extracted features alone, which is easy to lose some implicit information in the original audio that is difficult to fully reflect through features. However, this combination of embodiments of the present application enables the HuBERT model to fully tap into the information of both the original audio and the extracted features, making the most of the audio data to discover the emotional clues hidden in the user's voiceprint, and further improving the completeness and accuracy of the emotional feature extraction.

[0065] From the perspective of collaborative model utilization, related technologies may mostly use a single model to complete the entire process of audio emotion feature extraction, and the limitations of the model’s own capabilities will affect the final extraction effect. The embodiment of the present application first uses the Wav2Vec model, which is good at converting audio into rich feature vectors, for preliminary feature extraction, and then uses the HuBERT model, which has unique advantages in speech processing and can deeply mine emotional clues, for further analysis, thereby achieving the collaborative work and complementary advantages of the two advanced models. Wav2Vec provides high-quality feature input for HuBERT, which is based on which HuBERT can better identify emotions, making the entire audio emotion feature extraction process more efficient and accurate, which is a significant improvement over related technologies that rely solely on a single model.

[0066] From the perspective of the depth of emotional clue mining, in the key link of identifying the emotional clues hidden in the user's voiceprint, the relevant technology may be difficult to accurately capture the subtle frequency, rhythm and other changing characteristics of the voiceprint as the emotion changes due to insufficient features or limited model capabilities, resulting in inaccurate judgment of emotions. Through the above-mentioned comprehensive feature extraction and dual-model cooperation, the embodiment of the present application can keenly identify the specific frequency, rhythm changes and other characteristics of voiceprints under different emotions such as anger and joy, and more deeply mine the emotional clues contained in the voiceprint, and then accurately obtain the acoustic emotional features, which greatly improves the quality of emotional feature extraction of audio information, so that subsequent audio-based emotion recognition and other work can be carried out more reliably.

[0067] In summary, in terms of emotional feature extraction of audio information, through more comprehensive feature extraction, the synergistic advantages of dual models and deeper emotional clue mining, improvements have been achieved in multiple key links compared to related technologies, which helps to more accurately capture the emotional content contained in audio information.

[0068] For image information, visual feature extraction uses large models such as Vision Transformer and Swin Transformer to extract multi-scale visual features, including facial key points, micro-expressions, eye movement features, posture estimation, etc. As an optional embodiment, the above steps of extracting the corresponding emotional feature information under multimodal conditions from the multimodal data can be implemented as follows:

[0069] The Vision Transformer model is used to extract initial visual features at multiple scales from the image information; the initial visual features include at least facial key point features, facial contour features, limb contour features, posture features, motion change features, and color distribution features; the initial visual features are used to perform emotional state recognition and change trend analysis to obtain the visual emotional features.

[0070] Specifically, the powerful visual feature extraction capability of the Vision Transformer model is used to extract initial visual features at multiple scales from image information. These features are very rich, including facial key point features, such as the position and morphological changes of facial features such as eyes and mouth, which play a key role in judging facial expressions and inferring emotional states; facial contour features can assist in identifying changes in overall facial morphology; limb contour features can reflect the posture and amplitude of limb movements. Posture features are related to the overall posture state of the person. For example, standing straight may be related to emotions such as self-confidence, while hunching over may imply depression; movement change features can capture the emotional information contained in dynamic movements such as waving and nodding; color distribution features cannot be ignored either. Different color combinations will create different emotional atmospheres visually. For example, warm colors may convey positive and enthusiastic emotions, while cold colors may be associated with calmness, sadness and other emotions. Combining these multi-scale initial visual features provides rich visual materials for subsequent emotional analysis.

[0071] Then, the extracted initial visual features are further used to carry out emotional state recognition and change trend analysis. By analyzing the dynamic changes of facial key points, the expression is judged to be different states such as smiling and frowning. Combining features such as posture and body movements, the current emotional state presented by the image is identified, and the change trend of these features over time or between different images is observed, such as the change process from a smile to a calm expression, etc., in order to obtain visual emotional features, comprehensively and accurately grasp the emotional information contained in the image information, and lay the foundation for multimodal emotional fusion and subsequent emotion recognition work.

[0072] Through the above sentiment feature extraction steps for the three modal data of text, audio, and image, we can fully explore the sentiment information contained in each modality, and provide a comprehensive and high-quality data foundation for subsequent multimodal fusion and more accurate emotion recognition.

[0073] Exemplarily, in the feature extraction process of the above three modal data, the feature extractor in the embodiment of the present application can adopt domain adaptation and progressive transfer learning strategies to improve the discriminability and generalization ability of features through multi-task learning and contrastive learning.

[0074] Further optionally, before extracting the corresponding emotional feature information under multimodality from the multimodal data, the multimodal data can also be split according to the data type to obtain text preprocessing data, audio preprocessing data, and image preprocessing data. Then, the text preprocessing data is preprocessed to obtain preprocessed text information. Among them, the preprocessing operations according to the text data type include: word segmentation, denoising, and standardization. The audio preprocessing data is preprocessed to obtain preprocessed audio information. Among them, the preprocessing operations according to the audio data type include: noise reduction, segmentation, and normalization. The image preprocessing data is preprocessed to obtain preprocessed image information. Among them, the preprocessing operations according to the image data type include: denoising, resizing, and color standardization.

[0075] As an optional embodiment, in 103, multimodal fusion is performed on the emotion feature information to obtain an emotion fusion feature, including:

[0076] Through a nonlinear projection matrix, the emotional feature information is mapped into a high-dimensional semantic space according to different modalities to obtain a multimodal mapping feature matrix; through adversarial training, the multimodal mapping feature matrix is ​​subjected to modal difference elimination processing to obtain a first mapping feature matrix; a multi-layer multi-head cross-attention mechanism is adopted to establish fine-grained correlations between different modalities in the first mapping feature matrix to obtain a second mapping feature matrix; based on the Transformer encoder improved by the meta-learning strategy, the second mapping feature matrix is ​​adaptively evaluated according to the importance of different modalities, and dynamic weight allocation is performed according to the evaluation results to obtain the emotional fusion feature.

[0077] Specifically, first, we collect emotional feature information extracted from different modalities (such as text, speech, and images). These information have different forms of expression. For example, text features may be multi-level emotional feature vectors, speech features are data related to spectrum and rhythm, and image features are multi-scale visual features.

[0078] Next, the nonlinear projection matrix is ​​used to operate the emotional feature information of these different modalities separately. The purpose is to map all the modal features that were originally in different dimensions and representation forms into a unified high-dimensional semantic space. Through this mapping process, the features of each modality have corresponding representations in this new semantic space, and the feature representations of all modalities are combined to form a multimodal mapping feature matrix. For example, text features originally exist in the form of word vectors, etc. After being mapped by the projection matrix, they are converted into a new vector form that can reflect its semantic meaning in the high-dimensional semantic space; speech features are also transformed from spectrum-related data representations into corresponding vector forms in the semantic space; the same is true for image features, from the embodiment of visual features to vector representations in the high-dimensional semantic space, which is ready for the subsequent elimination of modal differences and further fusion.

[0079] After obtaining the multimodal mapping feature matrix, adversarial training is used for processing. In this process, a structure similar to the discriminator is set up.

[0080] Specifically, the features of different modalities begin to "compete against each other" under this adversarial training mechanism. The basic principle is to allow the model to learn to distinguish the features of different modalities while at the same time encouraging it to reduce the differences that naturally exist between the modalities. For example, the discriminator tries to identify whether a feature vector is from the text modality or the speech modality, etc., but the model will adjust the representation of the features of each modality to make them closer and closer at the semantic level, making it difficult for the discriminator to accurately distinguish them. After such a continuous adversarial learning process, the differences between the modalities gradually decrease, and the final feature matrix after the modal difference elimination process is the first mapping feature matrix, which makes the features of each modality more semantically unified and more conducive to subsequent fusion operations.

[0081] Based on the first mapping feature matrix, a multi-layer multi-head cross-attention mechanism is used to establish fine-grained correlations between different modalities. Under this mechanism, features of different modalities interact and cross-attend. For example, text features can pay attention to information related to speech features and image features. Similarly, speech features and image features will pay attention to corresponding information in other modalities. Through such mutual attention and interaction, not only a more detailed and in-depth association between modalities is established, but also in this process, the model can automatically learn the relative importance of each modal feature in the overall fusion context. For example, when analyzing the emotional content of a video containing voice explanations, text subtitles, and related display images, this mechanism can clearly clarify the key role played by each of the modalities of text, speech, and image in expressing a certain specific emotion. After such processing, the second mapping feature matrix is ​​obtained, which further enriches and refines the fusion relationship between modalities.

[0082] Finally, the second mapping feature matrix is ​​processed by the Transformer encoder improved based on the meta-learning strategy. This Transformer encoder will adaptively evaluate the importance of each modality feature based on the performance of different modalities in specific application scenarios, the data involved, and the comprehensive factors reflected in the previous steps. For example, in some specific emotional expression scenarios, the emotional information contained in the image modality may be more critical to the overall judgment, and the encoder can recognize this.

[0083] Then, based on the evaluation results of the importance of each modal feature, the weights corresponding to each modality are dynamically adjusted. If a certain modality is judged to be more important for emotional expression, the weight of the modal feature will be increased accordingly, otherwise its weight will be reduced. Through such a flexible weight adjustment method, the different contributions of each modality in actual situations are fully considered, and a more practical multimodal fusion is achieved. What is finally obtained is the emotional fusion feature after comprehensive fusion processing, which integrates the advantageous information of each modality and can be better used for subsequent emotion recognition and other related tasks.

[0084] Moreover, when encountering the situation of missing image modality, such as when the image display function is turned off during a video call, the system based on the improved Transformer encoder based on the meta-learning strategy can also enable a compensation mechanism, using existing text, voice and other modal information as well as previously learned related patterns and rules to fill the data gaps caused by the missing image modality, ensuring that the entire multimodal fusion process can still proceed stably and guaranteeing the accuracy and reliability of the final emotional fusion features.

[0085] For example, see Figure 2 As shown in 103, the feature alignment module first uses a nonlinear projection matrix to map the features of different modalities to a unified semantic space, and eliminates modal differences through adversarial training. The attention fusion module adopts a multi-layer multi-head cross-attention mechanism to establish fine-grained associations between modalities, and introduces a gating mechanism to control the flow of information. The dynamic weight allocation module is based on an improved transformer encoder structure and combines a meta-learning strategy to achieve adaptive evaluation and dynamic adjustment of feature importance. In addition, the system also introduces a modality missing compensation mechanism that can maintain stable performance when some modality data is missing.

[0086] As an optional embodiment, in 103, before the multimodal mapping feature matrix is ​​subjected to modal difference elimination processing through adversarial training processing and the first mapping feature matrix is ​​obtained, a corresponding gated unit model can also be established according to different modalities. Furthermore, the gated unit model is used to filter and delete invalid matrix elements corresponding to different modalities from the multimodal mapping feature matrix; wherein the invalid matrix elements satisfy the preset filtering conditions corresponding to the respective modalities.

[0087] Specifically, corresponding gated unit models are established for different modalities (such as text, speech, and images). Each gated unit model is designed based on the characteristics and data features of the corresponding modality. It can understand and process the information representation and potential semantic associations unique to the modality. For example, for the text modality, the gated unit model will take into account factors such as the grammatical structure and lexical semantics of the text; for the speech modality, it will combine acoustic features, rhythmic characteristics, etc.; for the image modality, it will refer to visual features, spatial layout and other aspects to build it, so that it can adapt to the data processing requirements of the corresponding modality.

[0088] Set corresponding preset filtering conditions for each modality, which are based on the analysis and experience of the modality data in the process of emotional feature expression and fusion. For example, in the text modality, the preset filtering conditions may involve feature vector elements corresponding to some low-frequency words that have no obvious effect on emotional judgment, or feature elements corresponding to some text fragments with chaotic grammatical structure, ambiguous semantics, and difficulty in conveying effective emotional information; in the speech modality, it may be acoustic feature elements with too low volume, too messy spectral features, and not in line with the normal law of speech emotional expression; for the image modality, it may be visual feature elements corresponding to the background area in the image that is irrelevant to the emotional expression of the character, or feature elements corresponding to the part where emotional information cannot be accurately extracted due to image quality problems (such as excessive blur, severe occlusion, etc.).

[0089] After the gating unit model corresponding to each modality and the preset screening conditions are determined, the multimodal mapping feature matrix is ​​input into the corresponding gating unit model. These gating unit models are like intelligent "sieves" that will check and judge the part of the multimodal mapping feature matrix belonging to the modality element by element according to the preset screening conditions set for each. For the matrix elements that meet the preset screening conditions, that is, the matrix elements that are judged to be invalid, they are screened out from the multimodal mapping feature matrix and deleted. In this way, after being processed by the gating unit model corresponding to each modality, those elements in the multimodal mapping feature matrix that are considered to be unhelpful or even likely to interfere with subsequent emotional fusion and analysis are removed, and a relatively purer feature matrix that is more conducive to the elimination of modal differences and subsequent fusion is obtained, that is, an intermediate matrix state in preparation for the subsequent generation of the first mapping feature matrix.

[0090] Through the above steps, in the process of multimodal data fusion, the feature information originally extracted from different modalities often contains some elements that have no practical value for sentiment analysis and fusion, or even interfere with it. By establishing a gated unit model for screening and deletion operations, these invalid matrix elements can be accurately removed to prevent them from being mixed into valid information in the subsequent elimination of modal differences and further fusion processes, causing information confusion and difficulties in model learning, so that the entire multimodal fusion process can focus more on valuable sentiment feature information and improve the efficiency and quality of fusion.

[0091] After each modality is screened, what remains are the feature elements that are more representative and more closely related to emotional expression, which makes the features of each modality more refined and accurate. For the text modality, what remains are text features that can more clearly convey emotional semantics; what remains for the speech modality are effective acoustic features that can better reflect emotional rhythm and acoustic characteristics; what remains for the image modality are key visual features that are closely related to the emotional expression of the character. In this way, when the modal differences are subsequently eliminated and fused, each modality can better cooperate and work together based on high-quality features, which helps to more accurately mine the emotional information contained in the multimodal data and improve the reliability and effectiveness of the final emotional fusion features.

[0092] Since invalid elements are removed in advance before the modal difference elimination process, it provides a better data foundation for subsequent modal difference elimination operations such as adversarial training. During the process of making different modal features closer at the semantic level through adversarial training, the model can focus more on the difference learning and adjustment of truly meaningful features, reducing the additional computational burden caused by the existence of invalid information and the possible wrong learning situations. Ultimately, it optimizes the effect of multimodal fusion as a whole, enabling the obtained first mapping feature matrix and the subsequent generated sentiment fusion features to more accurately reflect the sentiment states contained in the multimodal data, providing stronger support for related applications such as emotion recognition.

[0093] In summary, the operation of establishing a gated unit model according to different modalities and screening and deleting invalid matrix elements, based on reasonable principles, can bring various positive effects in the process of multimodal sentiment feature fusion, contributing to improving the performance and accuracy of the entire sentiment analysis system.

[0094] In the above steps, determining the preset screening conditions for different modalities is a complex but crucial process that requires considering various factors. Specifically as follows:

[0095] For the text modality, first, the sentiment tendency and intensity of words can be determined using sentiment lexicons. For words with unclear sentiment tendencies (such as neutral words) or extremely low sentiment intensities, in certain specific scenarios, if these words do not play a key role in text sentiment expression, the corresponding feature elements can be considered as screening conditions. For example, some very common conjunctions, auxiliary words, etc., like "de", "di", "he", etc., generally do not provide substantial help for sentiment judgment in most cases. At the same time, analyze the semantic importance of words in a specific domain or topic. If the text is about technical documents, some general words unrelated to technology may be relatively unimportant; if it is an emotional literary work, those descriptive words that cannot reflect the emotions of characters (such as simple environmental description words in some cases) may be regarded as screening conditions for invalid elements.

[0096] Next, observe the grammar and syntactic structure of sentences. For sentence parts with chaotic structures, not conforming to normal grammar rules and unable to accurately understand their semantics, they can be set as screening conditions. For example, some text fragments with ambiguous semantics due to typos or grammar errors, their corresponding feature elements may interfere with sentiment analysis.

[0097] Consider the contribution of sentence components to emotional expression. In a sentence, the subject and predicate usually carry the main emotional information, and some lengthy modifiers or interjections can also be used as screening conditions if they have no substantial impact on emotional judgment. For example, in the sentence "The decoration that actually looks a bit redundant makes him happy", the modifier "actually looks a bit redundant" may not be critical to the emotional judgment of "happy" in some cases, so it can be considered as a screening object.

[0098] Furthermore, by counting the frequency of occurrence of words in the text, words that appear infrequently and are not closely related to emotional expression can be used as screening conditions. For example, in a sentiment analysis task, if an uncommon word appears only once and its sentiment tendency cannot be clearly identified in the sentiment dictionary and contextual semantics, then its corresponding feature element may be screened out.

[0099] Analyze the distribution of words in texts of different sentiment categories. If a word appears evenly in texts of various sentiment categories (such as positive, negative, and neutral) without obvious sentiment bias, you can also consider using it as a screening condition.

[0100] For speech modalities, for spectral features, the screening conditions are determined based on the human hearing perception range and the common spectral laws of speech emotion expression. For example, a reasonable range of Mel-frequency cepstral coefficients (MFCC) is set. If the MFCC values ​​of certain frequency bands exceed the normal speech range (such as abnormal frequencies caused by noise or equipment failure), these corresponding acoustic physical vector elements can be used as screening conditions.

[0101] For prosodic features, the normal range of pitch, speech rate, and volume is considered. For example, the feature elements corresponding to the voice clips with too low volume (may be environmental noise interference or weak voice signal) or too high volume (may be abnormal screaming and other non-emotional expressive sounds), pitch beyond the normal human voice expression range (such as too high or too low pitch may be a signal error), and too fast or too slow speech rate (not in line with the normal emotional expression rhythm, such as the number of words per second exceeds the normal range) can be used as screening conditions.

[0102] For example, study the typical changing patterns of speech features under different emotional states. For example, in the expression of anger, the speech usually has a higher pitch, a louder volume, and a faster speaking speed, while in sad emotions, the speaking speed may be slower and the pitch may be lower. For those feature elements that do not conform to common emotional speech patterns, you can consider using them as screening conditions. Observe the relationship between features such as pauses and breathing in speech signals and emotional expression. In normal emotional expression, pauses and breathing have certain rules, such as pauses in thinking, emphasis, etc. If there are abnormally frequent or long pauses (which may be signal interruptions or other interference), or breathing patterns that do not conform to the logic of emotional expression, the corresponding feature elements can be used as screening conditions.

[0103] Consider the background noise characteristics in the voice recording environment. By analyzing the environmental noise, identify those acoustic feature elements that are mainly caused by noise as screening conditions. For example, if there is a continuous machine roar in the background, its corresponding frequency range and acoustic pattern can be marked as invalid elements and removed during the screening process.

[0104] For parts of the speech signal that may be affected by environmental reflections, echoes, etc., analyze the degree of interference with emotional expression. If factors such as echoes cause distortion of speech features and affect emotional judgment, the feature elements corresponding to these distorted parts can be used as screening conditions.

[0105] For image modality, it is mainly used for visual features related to facial expressions, focusing on the features of key parts such as eyes and mouth. If the face in the image is partially blocked, the facial key point features or expression features corresponding to the blocked part can be used as screening conditions. For example, if most of the eyes and mouth are blocked by hair or hands, these blocked parts may not be helpful for expression emotion analysis, and the corresponding feature elements can be filtered out.

[0106] Analyze the correlation between body movements and posture features and emotional expression. The characteristic elements corresponding to those small body movements that are not related to emotional expression (such as unconscious shaking, etc.) or postures that are not emotionally oriented in the current scene (such as simple standing postures in some cases) can be used as screening conditions.

[0107] Consider the relationship between the color distribution characteristics and emotional atmosphere in the image. If a certain color accounts for too high or too low a proportion in the image and has little correlation with common emotional colors (for example, in an image mainly used for character emotion analysis, a large area of ​​pure black background has no obvious emotional implications), its corresponding color distribution feature element can be used as a screening condition.

[0108] For blurred image areas, especially those where the blur affects the recognition of facial expressions and body movements, the corresponding visual feature elements are used as screening conditions. For example, the features corresponding to the parts where facial expressions cannot be clearly recognized due to low pixel of the shooting device or motion blur.

[0109] Analyze the impact of occlusion, shadows and other factors in the image on the extraction of emotional features. If an object blocks most of the person's body and the emotional state of the occluded part cannot be inferred through other information, the visual feature elements corresponding to these occluded areas can be used as screening conditions.

[0110] Consider the resolution and detail of the image. If the image resolution is too low to accurately extract key facial key points, postures and other emotion-related visual features, then the feature elements corresponding to these parts that are difficult to provide effective information due to insufficient resolution can be used as screening conditions.

[0111] As an optional embodiment, before performing multimodal fusion on the emotion feature information to obtain the emotion fusion feature in 103, if the amount of image information in the emotion feature information does not meet a preset image mode threshold, image data compensation may be performed on the emotion feature information.

[0112] Specifically, when the amount of image information is insufficient (for example, the preset image mode threshold is not reached), especially when the image modality is missing (such as in extreme cases where a video call is closed), the compensation mechanism will rely on existing text, voice and other modal information to find potential associations related to image features. For example, in daily communication scenarios, the scenes described in the text, the actions of the characters, or the emotional states reflected in the voice, etc., often have a certain logical connection with the expressions and postures of the characters in the corresponding image. Through the study and analysis of a large amount of annotated multimodal data, the corresponding rules between these modalities can be summarized. For example, when the text mentions "laughing happily" and the voice tone presents a cheerful rhythm, the characters in the image may usually have facial expressions such as raised corners of the mouth and squinting eyes, as well as more stretched body movements. Based on such rules, text and voice information can be used to infer the content of image information that may be missing.

[0113] In the early training process, we have accumulated a wealth of prior knowledge and learned the distribution patterns of different modal features in various emotional states through the model. For example, we know that when expressing anger, not only will the voice have features such as rising pitch and faster speech speed, but the text may also contain some fierce words, and the characters in the image often have typical expressions and body movements such as frowning and clenching fists. When image information is missing, with the help of these existing prior knowledge and model learning results, based on the emotional features shown by the current text and voice, we can reasonably infer the approximate appearance of the corresponding image features, thereby filling the data gap caused by the lack of image modality.

[0114] In the early steps of multimodal fusion, the features of different modalities are mapped to a unified high-dimensional semantic space through operations such as nonlinear projection matrices, and feature mapping relationships between the modalities are established. When performing image data compensation, the mapping relationship in this semantic space is used to further assist in filling in the gaps in image data. For example, in this unified semantic space, there is a relatively stable geometric relationship or semantic association between text feature vectors, speech feature vectors, and image feature vectors. When part of the image feature vector is missing, the reasonable value range or approximate shape that the image feature vector should have can be reversely calculated based on the position of other modalities in the semantic space and the mapping relationship between them, so as to generate the corresponding compensated image data, ensure that the subsequent fusion process can have relatively complete image features involved, and maintain the stability of the fusion process.

[0115] In multimodal fusion, the information of each modality complements and works together. If too much image modality information is missing and not compensated, the balance between the modalities will be broken, making it difficult for subsequent multimodal fusion-based operations (such as using a multi-layer multi-head cross-attention mechanism, processing based on a meta-learning strategy-improved Transformer encoder, etc.) to proceed smoothly because of the lack of data support for the important dimension of image features. The image data compensation mechanism can fill the gap in time to ensure that relatively complete multimodal data is available in each link of the fusion process, maintain the stable operation of the entire fusion process, and prevent the fusion process from being interrupted or errors from occurring due to insufficient image modalities.

[0116] Image information itself contains rich emotional clues, such as facial expressions, body movements, etc., which are key factors in judging emotional states. When image data is insufficient, compensation is performed so that the image features that are finally involved in the fusion can restore the complete emotional picture as much as possible, and better integrate with text and voice features, and combine the advantages of each modality to mine emotional information, thereby improving the accuracy of the final emotional fusion features. For example, in an emotional analysis scenario, the happy emotional clues conveyed by the character through the expression may be missed due to image loss, but after compensation, this part of the emotional information can be integrated, so that the fused emotional features can more comprehensively and accurately reflect the real emotional state, thereby providing a more reliable data foundation for subsequent work such as emotion recognition.

[0117] In actual application scenarios, image modality loss often occurs. By having an image data compensation mechanism, the model can better cope with such incomplete modality data and improve the generalization ability of the model under different data conditions. Regardless of whether there is a problem of image modality loss, the model can try its best to complete multimodal fusion and output relatively accurate emotional fusion features through reasonable compensation. This makes the model more widely applicable to various complex and changeable actual application scenarios. For example, in the case of network instability resulting in video call image loss or partial image data damage, it can still effectively perform emotional analysis to ensure the normal operation and function realization of the system.

[0118] In summary, the above-mentioned image data compensation mechanism can bring many positive effects in the multimodal fusion process, and help improve the performance and reliability of the entire multimodal-based sentiment analysis system.

[0119] As an optional embodiment, in step 103, the emotion fusion feature is input into an emotion recognition model to perform emotion state analysis to obtain a predicted emotion state corresponding to the user, including:

[0120] The emotion fusion feature is used to classify emotions through a deep multi-layer perceptron to obtain the corresponding user emotion category; the emotion fusion feature is constructed into a corresponding emotion state transfer matrix based on a bidirectional long short-term memory network; the emotion state transfer matrix is ​​used to represent the dynamic evolution process between user emotions; the time attention mechanism is used to extract the user's emotion short-term fluctuation trend and the emotion long-term change trend from the emotion fusion feature; according to the user emotion category, the emotion state transfer matrix, the emotion short-term fluctuation trend and the emotion long-term change trend, the user's emotion change tendency in the future time period and the corresponding predicted emotion category are predicted.

[0121] Specifically, first, the emotion fusion features obtained through the previous multimodal fusion step are used as input data and sent to the deep multilayer perceptron (MLP). MLP is a powerful neural network structure that processes the input emotion fusion features through multiple fully connected layers. In this process, neurons in each layer cooperate with each other to perform nonlinear transformations and combinations on the features, trying to dig out the hidden pattern information that can distinguish different emotion categories. For example, the emotion fusion features may contain various emotion-related clues extracted and fused from multimodal data such as text, voice, and images. MLP will combine these clues to determine the corresponding emotion category, and finally output specific user emotion categories such as anger, satisfaction, and anxiety to achieve preliminary emotion classification.

[0122] When using MLP for emotion classification, in order to improve the performance and stability of the model, two optimization techniques, batch normalization and residual connection, are used. Batch normalization is to normalize the data input to each layer of the network so that the mean and variance of the data remain in a relatively stable range. Doing so helps to speed up the convergence of the model and avoid the problem of training instability caused by large differences in data distribution. At the same time, it can also reduce overfitting to a certain extent, so that the model can have a relatively stable and good performance when facing different emotional fusion feature inputs. Residual connection allows a layer in the network to directly pass input information to the subsequent layers, solving problems such as gradient vanishing or gradient explosion that may occur as the network depth increases, allowing the model to better learn complex emotion classification patterns, further enhancing the stability of the model, and ensuring that the output user emotion category results are more reliable.

[0123] Next, based on the emotion fusion feature, the bidirectional long short-term memory network (bidirectional LSTM) is used to construct the emotion state transition matrix. Bidirectional LSTM has a unique structural advantage, which can process both the forward and reverse information of sequence data. For the emotion fusion feature, which contains multimodal emotion information and may be associated in the time dimension (such as the emotion information in a continuous video or dialogue scene), the bidirectional LSTM can capture the emotion evolution rules contained therein from two directions. By analyzing and learning the emotion fusion features at different times, it will summarize the probability of emotion transitioning from one state to another, and then construct the emotion state transition matrix. This matrix can clearly represent the dynamic evolution process between user emotions. For example, it can reflect the probability of transitioning from a happy emotional state to a calm state, or the probability of transitioning from a sad emotional state to an angry emotional state, etc., which provides an important basis for in-depth understanding of the changes in emotions in continuous scenes.

[0124] Based on the emotion fusion features, the temporal attention mechanism is used to mine the emotion change trend. The temporal attention mechanism can dynamically focus on the emotion information at key time points according to the importance of the emotion fusion features at different moments in the entire sequence. In the short term, it can capture the fluctuation of emotions in a short time interval. For example, in a short conversation, emotions may change from curiosity to surprise in an instant. This instantaneous fluctuation can be keenly captured by the temporal attention mechanism to form a short-term fluctuation trend of emotions. From a long-term perspective, it can analyze the overall development trend of emotions over a long period of time. For example, in the process of watching a movie, how emotions gradually transition from relaxation at the beginning to tension, and then to relief at the end, etc., thereby extracting the long-term change trend of emotions and more comprehensively showing the performance of user emotions in the time dimension.

[0125] Finally, by combining the user's emotion category, emotion state transition matrix, emotion short-term fluctuation trend, emotion long-term change trend and other information obtained previously, the user's emotion change tendency in the future period and the corresponding predicted emotion category are predicted. For example, it is known that the current user's emotion category is satisfaction. Combined with the probability of satisfied emotion transferring to other emotions in the emotion state transition matrix, and referring to the emotion short-term fluctuation trend (whether there are signs of rapid changes in emotions in the near future) and emotion long-term change trend (whether the overall emotion is developing in a positive or negative direction), through certain algorithms and model calculations, it is possible to infer the user's possible emotion change tendency in the next period of time, such as the emotion may continue to remain in a satisfied state, or there is a certain probability of changing to a happy state, etc., and then determine the corresponding predicted emotion category, so as to provide more accurate feedback basis for the follow-up, such as recommending appropriate content to users and adjusting the interaction method based on the prediction results, so as to better adapt to the user's emotional state.

[0126] Through the above series of steps, we can make full use of the emotion fusion features and use a variety of advanced neural network technologies and mechanisms to comprehensively and deeply analyze the user's emotional state and make reasonable predictions about future emotional conditions. This has important value in many application scenarios involving user emotion analysis.

[0127] As an optional embodiment, the confidence assessment based on the predicted emotional state in 104 may be implemented as follows:

[0128] The Bayesian deep learning method is used to convert the predicted emotional state into an emotional state probability distribution variable for uncertainty assessment; in the inference stage of the emotion recognition model, different emotion recognition sub-networks are constructed through multiple forward propagations, and the different emotion recognition sub-networks are used to process the emotional state probability distribution variables to obtain corresponding multiple predicted emotional states; the confidence levels corresponding to the multiple predicted emotional states are evaluated based on Monte Carlo Dropout; the reliability of the emotion category is quantitatively evaluated based on the confidence levels, and the target predicted emotional state for the final output is determined based on the quantitative evaluation results.

[0129] In related technologies, traditional deep learning usually treats model parameters as fixed values ​​for training and prediction, but Bayesian deep learning incorporates Bayesian inference and treats model parameters as random variables with probability distribution. The advantage of this is that when faced with data uncertainty and the complexity of the model itself, it can more comprehensively describe the uncertainty of the model rather than giving a deterministic prediction result.

[0130] In this scenario, the Bayesian deep learning method is used to convert the predicted emotional state that has been obtained into an emotional state probability distribution variable for uncertainty assessment. For example, the original predicted emotion category of the user may be a certain "happy", but now through Bayesian deep learning, the prediction result of "happy" will be expressed in the form of a probability distribution. For example, there is a certain probability of "happy", and there is also a small probability of "calm" or "slightly excited" and other related emotion categories. This probability distribution can reflect the degree of uncertainty of the prediction result, because different emotion categories are assigned corresponding probability values.

[0131] In the inference phase of the emotion recognition model (that is, the phase where the model has been trained and is used for prediction), multiple forward propagations are used. During each forward propagation, the model will calculate and output a result based on the input data (here, the converted emotional state probability distribution variable). Performing this operation multiple times is equivalent to processing the same input from different angles or based on different "states" of the model.

[0132] During each forward propagation, the model can use some randomized mechanisms (such as random deactivation of neurons) to make each calculation process seem like a different sub-network is at work. These different emotion recognition sub-networks each have slightly different structures or parameter states (differences generated by randomization within the overall model framework). They all process the same emotional state probability distribution variable and ultimately obtain multiple corresponding predicted emotional states. These multiple predicted emotional states reflect the model's judgment of the input from different perspectives, and because there are differences between the sub-networks, these prediction results will also be different, further reflecting the uncertainty of the prediction.

[0133] Monte Carlo Dropout is a method based on Dropout technology (randomly stopping some neurons from working during training to prevent overfitting) and extended to the inference stage. During inference, Dropout is applied randomly multiple times, and different subnetworks are obtained each time for prediction, just like the process of building different emotion recognition subnetworks in the previous section. The confidence is evaluated by observing the distribution of multiple predicted emotional states obtained by these different emotion recognition subnetworks. If multiple prediction results are relatively concentrated, for example, most of them point to the emotion category of "happy", with only a few deviations, it means that the model has low uncertainty for the prediction result of "happy", and the corresponding confidence is high. On the contrary, if the prediction results are very scattered, and each emotion category has a large proportion, it means that the uncertainty is high and the confidence is low. In this way, the dispersion of the prediction results is quantified as a confidence index, which intuitively reflects the reliability of each predicted emotional state.

[0134] Next, based on the calculated confidence, the reliability of the emotion category is further quantitatively evaluated. For example, a threshold can be set, and emotion categories with confidence levels above this threshold are considered to have high reliability, while those below the threshold are considered to have low reliability. For different predicted emotional states, corresponding reliability quantitative scores are assigned according to their confidence levels, so that the reliability of different emotion categories as final prediction results can be clearly compared.

[0135] Finally, the final output target predicted emotional state is determined based on the results of the quantitative evaluation. Usually, the emotional category with the highest reliability quantitative evaluation score (that is, the highest confidence) is selected as the final prediction result. In this way, when outputting the predicted emotional state, not only the emotional category is given, but also the relevant information of its reliability is attached, making the entire prediction result more scientific, rigorous, and more valuable for reference.

[0136] It is understandable that by converting the predicted emotional state into a probability distribution variable and using multiple forward propagations and different emotion recognition sub-networks, the uncertainty in the prediction results can be fully explored and presented. Unlike traditional single deterministic predictions, which ignore the multiple possibilities that the data and model itself may bring, this method comprehensively presents the probability of different emotion categories from a probabilistic perspective, which is more in line with the ambiguity and uncertainty of emotional judgment in actual situations, allowing users to grasp the prediction results more accurately.

[0137] Based on the Monte Carlo Dropout confidence assessment and subsequent reliability quantification assessment, the reliability of each predicted emotional state is carefully judged. In this way, when the target predicted emotional state is finally determined, the most reliable result is selected, avoiding blind reliance on a single prediction that may be inaccurate, reducing the risk of misjudging user emotions due to factors such as model errors or data noise, and significantly improving the accuracy and reliability of the output predicted emotional state, providing a more solid data foundation for subsequent various applications based on sentiment analysis (such as personalized recommendations, user interaction optimization, etc.).

[0138] The entire confidence assessment process is transparent and quantifiable. Users can not only see the final predicted emotional state, but also understand the confidence level of this result and how it is determined through a series of rigorous assessment steps. Compared with the traditional "black box" emotion recognition model prediction, this method increases the interpretability of the model, allowing users to better understand the decision-making basis and reliability of the model, making it easier to be accepted and trusted in practical applications, especially in some scenarios with high requirements for decision-making accuracy (such as medical, psychological counseling and other fields involving emotional judgment applications).

[0139] In summary, the confidence assessment method in this optional embodiment is based on reasonable principles, and can bring positive effects in improving prediction accuracy, reliability, and enhancing model interpretability, which helps to improve the performance and practical value of the entire emotion recognition system.

[0140] As an optional embodiment, generating a user emotion recognition result based on the predicted emotional state and the corresponding evaluation result in 104 can be implemented as follows: generating a corresponding attention heat map based on the target predicted emotional state; and / or generating the visual auxiliary analysis information according to the predicted emotional state to improve the interpretability of the recognition result.

[0141] In the embodiment of the present application, the attention heat map is a visualization tool that aims to show the distribution of key areas or key information that the model pays attention to during the emotion recognition process. The importance of different parts for the final predicted emotional state of the target is reflected by changing the depth of the color. The darker the color, the more critical the part is in the emotion judgment, and vice versa. Its main function is to help users intuitively understand how the model makes emotional judgments based on the input multimodal data (such as text, voice, images, etc.), thereby improving the interpretability of the entire emotion recognition result.

[0142] When generating an attention heat map based on the target predicted emotional state, first of all, there are different ways to process data of different modalities. Taking the text modality as an example, when the model analyzes the text to derive the emotional category, it will assign different attention weights to the words in the text. Words that play a key role in judging the target predicted emotional state (such as adjectives with strong emotional colors, verbs that express key emotional tendencies, etc.) will be given higher weights, and when generating the heat map, the areas corresponding to these words will be displayed in darker colors. For example, when judging that the emotion corresponding to the sentence "Today is a really happy day" is "happy", the area where the word "happy" is located may appear as a darker color in the heat map, indicating that the model focuses on this keyword that reflects the emotion.

[0143] For speech modality, the model may focus on certain key parts of the acoustic features. For example, when judging positive emotions, the time periods corresponding to features such as higher pitch and faster speaking speed will be reflected in darker colors in the heat map, showing the importance of these acoustic features in emotional judgment.

[0144] In terms of image modality, key parts of facial expressions (such as eyes, mouth, etc.) and parts of body movements that are closely related to emotional expression will be displayed in the heat map with corresponding color depths according to their contribution to the target predicted emotional state. For example, the area where the corners of the mouth are raised when smiling, the area where the eyes are narrowed when laughing, etc. If these features are critical to judging the current "happy" emotion, then the color of the corresponding facial area on the heat map will be darker, intuitively showing the emotion-related parts that the model focuses on in the image information.

[0145] By integrating the corresponding attention information of each part under different modalities, a comprehensive, cross-modal attention heat map is finally generated, which shows the overall focus of the model when deriving the target predicted emotional state, allowing users to clearly see the role of each modality and its different internal elements in emotion recognition.

[0146] Visualization-assisted analysis information includes multiple aspects, the most common of which are voice sentiment analysis, text sentiment analysis, facial expression sentiment analysis, etc. It further analyzes and displays the predicted emotional state from different angles, also to improve the interpretability of recognition results, so that users can have a more comprehensive and in-depth understanding of the basis and specific circumstances of emotion recognition.

[0147] By professionally analyzing the collected voice data, we extract acoustic features such as pitch, speaking speed, and volume, and then associate these features with the corresponding emotion categories and present them visually. For example, you can use a line graph to show how the pitch of a voice changes over time, and use different colored segments to mark the predicted emotional states corresponding to different stages of pitch change (such as the rising pitch stage corresponds to excitement, represented by red segments; the stable pitch stage corresponds to calm emotions, represented by green segments, etc.), so that users can intuitively see the connection between the changes in the acoustic features of the voice and the emotions. You can also use a bar graph to compare the differences in average speaking speed, volume and other features under different emotional states, and clearly show the supporting role of voice features in emotional judgment.

[0148] For the input text information, natural language processing technology is used to analyze the words, sentence structure, and semantics in the text to explore the emotional tendencies. When presenting visually, a word cloud chart can be used to display the emotional keywords that appear frequently in the text. For example, in a text judged to be positive, positive words such as "happy", "beautiful", and "like" in the word cloud chart will be displayed in a larger font, intuitively reflecting the emotional color of the text. Pie charts can also be used to show the proportion of different emotional categories (such as positive, negative, and neutral) in the entire text, helping users to grasp the emotional status of the text at a glance and the contribution of each part to the overall predicted emotional state.

[0149] Based on the facial expressions and body movements in the image information, different expression states and their corresponding emotional meanings are identified and displayed visually. For example, a dynamic visualization chart of expression changes can be made to show the changes in facial expressions from calm to smiling, and then to laughing over time or in different situations. At the same time, the predicted emotional state corresponding to each expression state is marked to clearly show the relationship between expression changes and emotional evolution. Or the typical emotions corresponding to different expressions (frown, glaring, upturned corners of the mouth, etc.) can be displayed in the form of a picture set, which is convenient for users to compare and understand the supporting role of expression features in the image in predicting emotional states.

[0150] By generating such rich and diverse visual auxiliary analysis information and combining it with attention heat maps, the entire emotion recognition process and basis can be presented to users from multiple dimensions in an intuitive and easy-to-understand manner, making the user's emotion recognition results more transparent and explainable, and helping users to better use these emotion recognition results for subsequent decision-making, service optimization and other related work.

[0151] In summary, by generating attention heat maps based on the target predicted emotional state and generating visual auxiliary analysis information based on the predicted emotional state, the interpretability of emotion recognition results can be significantly improved, so that the emotion recognition system not only outputs a simple emotion category, but also provides a comprehensive, intuitive and easy-to-understand emotion analysis display, which enhances the practicality and credibility of the system.

[0152] As an optional embodiment, in the data preparation stage, first, it is necessary to collect multimodal data related to emotion recognition, including text, voice, and image information. These data are the basic materials for generating attention heat maps, and they each contain emotional clues of different dimensions. For example, text data may be the user's chat records, comments, etc.; voice data is the recorded audio of the user speaking; image data can be video screenshots or photos containing the user's facial expressions and body movements, etc.

[0153] Perform feature extraction operations on data of different modalities. For text data, features such as word vectors, parts of speech, and syntactic structures are extracted. At the same time, natural language processing technology is used to analyze emotional keywords and semantic relationships in the text. These features help to determine the contribution of each part of the text to the target predicted emotional state. For example, word embedding technology is used to convert each word in the text into a vector representation, which makes it easier for computers to understand and process its semantic information.

[0154] In terms of speech data, audio processing algorithms are used to extract acoustic features such as Mel-frequency cepstral coefficients (MFCC), pitch, speaking rate, pauses, etc. These features reflect the rhythm, timbre and other characteristics of speech, are closely related to emotional expression, and are an important basis for judging emotions. They are also key factors in determining the attention of the speech part when subsequently generating heat maps.

[0155] For image data, computer vision technology is used to extract the coordinates of facial key points, facial expression features (such as the degree of eye opening, the upturn or drooping of the corners of the mouth, etc.), limb movement features (such as the extension or bending of the arms, body posture, etc.), and image color, texture and other visual features, in preparation for analyzing the importance of different areas in the image for emotional judgment.

[0156] According to the target predicted emotional state and the relevant algorithms and models of natural language processing, the importance of each element in the text to the emotional judgment is analyzed. For example, if the target predicted emotional state is "happy", then the words "joy", "pleasure", "laughter" and other words that directly express happy emotions in the text will be given a higher weight; at the same time, some descriptive sentences that are semantically related to happiness, such as "sunny days make people feel happy", the related words and phrases involved will also be assigned corresponding weights according to their contribution to the overall expression of happy emotions.

[0157] In addition, sentence structure, grammatical relations, etc. will also affect weight distribution. For example, in a sentence that emphasizes tone, "I'm really so happy," words such as "really" and "so" that play a role in strengthening the emotional tone will receive a relatively high weight because they strengthen the expression of the emotion "happy." By comprehensively considering these factors, the corresponding attention weight in emotional judgment is determined for each word, phrase, and other element in the text. The higher the weight value, the more critical it is to derive the target predicted emotional state.

[0158] According to the target predicted emotional state, the correlation between the acoustic features of the speech and the emotion is analyzed to determine the weight. Taking the emotion of "happiness" as an example, when a person is happy, the pitch of the voice may be relatively high, the speaking speed may be relatively fast, there may be fewer pauses, and the tone may be relatively upward. Then when analyzing the corresponding voice data, the time period with higher pitch, the part with faster speaking speed, and the voice segment with upward tone will be considered to be important in judging the emotion of "happiness", and thus be given a higher weight.

[0159] At the same time, the synergy between different acoustic features is also taken into account. For example, a part where the pitch rises and the speaking speed increases at the same time may be more representative of the expression of "happy" emotion than a single feature is prominent, and will be assigned a higher weight. By analyzing the time series of each acoustic feature in the speech data and their degree of fit with the target predicted emotional state, the attention weight of the speech part corresponding to each time segment in emotional judgment is determined.

[0160] In the image modality, the emotional state of the target is predicted, focusing on areas related to facial expressions and body movements. For the expression of "happy" emotions, facial features such as squinting eyes and upturned corners of the mouth are typical manifestations, so the areas where the eyes and mouth are located are crucial for judging happy emotions and will be given higher weights. Similarly, in terms of body movements, such as stretched body postures, which are often related to positive emotions, the corresponding body areas will also be assigned corresponding weights according to their importance in emotional expression.

[0161] Moreover, the combination of different features in the image and the overall visual effect will also affect the weight. For example, a smiling expression with a relaxed body posture will present a stronger atmosphere of happiness. Then the weight of the facial and body-related areas involved in judging the emotion will be reasonably improved by comprehensively considering these factors, so as to determine the attention weight of each part of the image in the emotion judgment.

[0162] During the heat map generation phase, you can choose a professional visualization library (such as Matplotlib and Seaborn in Python) to generate the heat map. Determine the coordinate system of the heat map, generally using the word order of the text, the time series of the speech, or the pixel position of the image as the coordinate axis, and set it accordingly according to the data characteristics of different modes, so as to accurately display the attention of each part.

[0163] Map the attention weights of different parts of each modality determined previously to the color space. Generally, the color depth is used to intuitively represent the weight size. For example, higher weight values ​​correspond to darker colors (such as red, orange and other warm colors are often used to represent areas with high attention), and lower weight values ​​correspond to lighter colors (such as light blue, light gray and other cold colors represent areas with low attention). Through this color mapping method, the weights can be presented in a visual form.

[0164] Further optionally, if it is necessary to display the overall attention across modalities, the attention heat maps corresponding to different modalities such as text, voice, and image can be integrated. For example, in the same visualization interface, the text area, voice timeline area, and image display area are divided separately, and then the heat maps of each modality are arranged and combined according to the corresponding positions and proportions to generate a comprehensive attention heat map, which comprehensively displays the importance distribution of different modalities and different elements within each modality when deriving the target predicted emotional state, so that users can see at a glance the focus of the model in the entire emotion recognition process.

[0165] Make detailed adjustments to the generated heat map, including setting the range of the color bar, marking the meaning of the coordinate axis, adding title descriptions, etc., to make it clearer, more intuitive and easier to understand. For example, mark the weight range corresponding to the color depth next to the color bar, so that users can accurately grasp the degree of attention represented by different colors. Indicate the corresponding text content, voice time point or specific location information of the image on the coordinate axis, so that users can check the attention situation by comparing it with the original data.

[0166] Through the above detailed steps, it is possible to generate an attention heat map based on the target predicted emotional state, clearly present the focus of the model in the emotion recognition process in a visual way, and improve the interpretability of the emotion recognition results.

[0167] As an optional embodiment, after generating the user emotion recognition result based on the predicted emotional state and the corresponding evaluation result in 104, the user's feedback information on the user emotion recognition result can also be used, and online learning is performed based on the feedback information to obtain model optimization parameters. Then, the model optimization parameters are used in the model adjustment process of the emotion recognition model to improve the emotion recognition accuracy of the emotion recognition model.

[0168] Specifically, after the user emotion recognition result is generated based on the predicted emotional state and the corresponding evaluation result and presented to the user, the user is encouraged to provide feedback on the recognition result. This feedback information can be in many forms, such as the user directly confirming whether the recognized emotion category is accurate (such as marking it as correct or wrong), or further providing more detailed correction suggestions, explaining where the actual emotion felt is different from the recognition result, etc. For example, the predicted emotional state output by the emotion recognition model is "happy", but the user feels that his actual emotion at the time is "calm", and feeds this difference back to the system, which constitutes an important basis for subsequent optimization of the model.

[0169] The collected user feedback information is taken into account as additional annotated data, and is linked to the multimodal data (text, voice, images, etc.) previously used to train the model and its corresponding true emotion labels (in the annotated training data set). Because there is a deviation between the output of the model (predicted emotional state) and the true emotion of the user feedback, these deviations can be analyzed to reversely infer possible problems with the current parameter settings of the model and find the direction in which the model parameters need to be adjusted. For example, if the user's "calm" emotion is misjudged as "happy" many times, it may mean that the model does not reasonably allocate weights to each modal feature when processing certain specific feature combinations (such as a flat voice tone but a slightly smiling facial expression), or that the parameters of some neural network layers in the model used for emotion classification and feature extraction do not accurately capture the key information to distinguish between the two emotions.

[0170] Use appropriate optimization algorithms (such as stochastic gradient descent and its variants) to calculate how the model parameters should be adjusted to reduce the model prediction error revealed by the feedback information. Specifically, guided by the loss function (a function that measures the difference between the model's prediction results and the actual results), the calculation of the loss function is adjusted through feedback information to more accurately reflect the current performance issues of the model. Then, using the optimization algorithm, based on the gradient information of the loss function about the model parameters, the model parameters are gradually updated, so that the model can develop in the direction of reducing the difference with the user's real emotions in subsequent predictions. This process is an online learning process based on feedback information. By continuously incorporating new feedback data for such learning and parameter adjustment, the continuous optimization of the model is achieved.

[0171] After obtaining the model optimization parameters through online learning, these new parameters are applied to the corresponding structure of the emotion recognition model to replace the original old parameters. For example, the weight parameters of each layer in the deep multi-layer perceptron (for emotion classification) and the relevant weights and bias parameters in the bidirectional long short-term memory network (for building the emotion state transfer matrix) are updated, so that the overall structure and function of the model are adjusted based on the new parameters, thereby changing the model's feature extraction, fusion and emotion judgment processing methods for multimodal data. It is expected that in subsequent emotion recognition tasks, the predicted emotional state that conforms to the user's real emotions can be output more accurately, so as to achieve the purpose of improving the accuracy of emotion recognition.

[0172] It is worth noting that by introducing user feedback information for online learning and model parameter adjustment, the model can correct its own errors and deficiencies based on the real feelings of users in actual applications. For example, due to limited training data or uneven data distribution, there may be many misjudgments of emotions in certain specific scenarios. However, as the model continuously absorbs user feedback and optimizes parameters, it can better adapt to various complex and changing real-world situations, more accurately identify users' real emotions, reduce the deviation between predicted emotional states and actual emotions, and significantly improve the accuracy of emotion recognition.

[0173] Different users may express and feel emotions very differently, and the actual application scenarios are also diverse. Using user feedback for online learning allows the model to be exposed to a wider range of emotion samples that are closer to real-world usage scenarios, and to continuously learn and adapt to these new situations, thereby broadening the scope of application of the model and enhancing its generalization capabilities in different user groups and different application scenarios. Regardless of the style of text expression, voice intonation characteristics, and facial expressions and body movements in images, the model can more reasonably perform emotion recognition based on optimized parameters and improve overall performance.

[0174] Accurate emotion recognition is crucial for many application scenarios involving human-computer interaction, user services, etc. When the accuracy of the emotion recognition model is improved, the system can provide more appropriate responses and services based on the recognition results that are more in line with the user's real emotions. For example, in the intelligent customer service scenario, if the user's dissatisfaction can be accurately judged, effective appeasement measures can be taken in time and more targeted solutions can be provided, thereby improving the user's satisfaction with the entire service, enhancing the good interactive experience between the user and the system, and making the system more practical in actual applications.

[0175] Model training in related technologies is often completed offline based on fixed data sets, and once deployed, it is difficult to adjust in time according to new situations in actual use. However, in the embodiment of the present application, the online learning mechanism based on user feedback information enables the emotion recognition model to continuously and dynamically optimize its own parameters during actual operation, realizes continuous iterative updates of the model, can always keep up with changes in user needs and actual application scenarios, maintain good performance status, extend the effective use cycle of the model, and reduce the cost of frequent redevelopment and training of new models.

[0176] In summary, using user feedback on emotion recognition results for online learning and adjusting model parameters based on reasonable principles can bring positive and significant results in improving emotion recognition accuracy, enhancing model adaptability, and improving user experience, which will help the emotion recognition system better serve various practical application scenarios.

[0177] In the technical solution of the present application, first, the multimodal data to be processed is obtained; the multimodal data includes at least: text information, audio information, and image information input by the user; the image information includes at least: facial expression images and / or body movement images. Then, the multimodal data is input into the emotion recognition model for emotional state analysis to obtain the corresponding predicted emotional state of the user. Finally, a confidence assessment is performed based on the predicted emotional state, and a user emotion recognition result is generated based on the predicted emotional state and the corresponding evaluation result. The user emotion recognition result includes at least: an attention heat map and / or visual auxiliary analysis information for displaying the user's emotional state from multiple dimensions; the auxiliary analysis information includes at least: speech emotion analysis, text emotion analysis, and expression emotion analysis. In the technical solution of the present application, the emotion recognition model is used to capture richer emotional details from multimodal data, realize cross-modal emotional intelligent recognition, and further improve the accuracy of emotion recognition in combination with confidence assessment, improve the efficiency of emotion recognition, and expand the application scope of emotion recognition.

[0178] In an optional embodiment, an example of user emotion recognition and service strategy optimization based on multimodal data is provided in an intelligent customer service scenario.

[0179] In the intelligent customer service scenario, users interact with the customer service system through text chat, voice conversation, video call, etc. The system aims to accurately identify the user's emotional state to achieve dynamic optimization of service strategies and improve customer experience and satisfaction.

[0180] In the input and preprocessing stage of multimodal data, for text input, after receiving the text sent by the user in the chat window, regular expressions and stop words are used to remove noise characters and meaningless words, and pre-trained large language models (such as Qwen or GLM) are used to complete text segmentation and semantic vectorization, and extract syntactic, semantic, emotional polarity and other features. For voice input, audio signals are obtained from call recordings, and noise reduction algorithms (such as Spectral Subtraction) are used to eliminate environmental noise. The voice endpoint detection technology is used to separate the effective voice segments, and then the acoustic features of the audio (such as pitch, resonance peak, rhythm features) are extracted through the Wav2Vec model. For image input, the user's facial image is obtained from the video call, OpenCV is used to solve the problem of uneven lighting and adjust the image brightness, and Dlib is used to detect facial key points and correct posture. After the image size is standardized, it is input into Swin Transformer to extract visual features.

[0181] Entering the feature extraction and representation learning stage, the text feature extraction uses the GLM model to generate multi-level emotional feature vectors covering word level, sentence level and context level, and uses a multi-task learning strategy to capture subtle emotional changes in the text, such as modal particles, metaphorical expressions, etc. The speech feature extraction uses the Wav2Vec model to extract spectral features (such as MFCC, spectral centroid) and rhythmic features (such as pitch curve, speaking speed), combined with the HuBERT model to identify emotional clues such as anger and joy implied in the voiceprint. Image feature extraction is based on Vision Transformer to extract multi-scale visual features, identify expression changes (such as smiling, frowning) by analyzing facial key points, and analyze emotional trends in combination with posture features.

[0182] Next is the multimodal feature fusion stage. First, the projection matrix is ​​used to map text, speech, and image features to a unified high-dimensional semantic space, and adversarial training is used to reduce inter-modal differences. Then, a multi-layer multi-head cross-attention mechanism is used to establish inter-modal associations and learn the importance of each modal feature. At the same time, a gating unit is introduced to dynamically control the flow of information to prevent invalid information interference. Based on the meta-learning strategy, the importance of each modal input is evaluated through the improved Transformer and the weight is dynamically adjusted. When the image modality is missing (such as when the video call is closed), the compensation mechanism is used to fill the data gap to ensure fusion stability.

[0183] In the emotion recognition and state tracking stage, a deep multi-layer perceptron (MLP) is used to classify the fused features and output the user emotion categories (such as anger, satisfaction, and anxiety). The model stability is enhanced with the help of batch normalization and residual connection optimization training. A bidirectional LSTM-based emotional state transfer matrix is ​​constructed to capture the dynamic evolution of emotions. The temporal attention mechanism is used to model the short-term fluctuations and long-term trends of user emotions. Combined with the sequential emotional changes, the user's possible emotional tendencies in the future are predicted, providing accurate feedback basis for service strategy adjustments.

[0184] Finally, in the result output and optimization stage, Bayesian deep learning is used to calculate the uncertainty of the classification results, Monte Carlo Dropout is used to evaluate the confidence, and the evidence theory framework is used to quantify the reliability of each emotion category. The emotional clues of multimodal features are intuitively displayed through attention heat maps, and auxiliary information such as voice pitch waveforms, text sentiment analysis, and expression feature images are provided to enhance the interpretability of the results. At the same time, the feedback given by users in the interaction is used for online learning, and the model parameters are continuously optimized to improve the accuracy of emotion recognition.

[0185] Through the embodiments of the present application, it is possible to accurately identify the emotional changes of users in intelligent customer service scenarios, for example, accurately capture the emotional transition process of users from anxiety to satisfaction, and dynamically adjust the service strategy accordingly, such as switching from eager explanation to gentle comfort or proactively providing solutions, thereby significantly improving the user experience. Moreover, the multimodal fusion design of this method enables it to have a strong ability to capture complex emotional expressions, and it can still maintain high performance even in the absence of data, effectively meeting the application needs of intelligent customer service in multiple scenarios, and providing strong support for the efficient operation and high-quality service of the intelligent customer service system.

[0186] In another embodiment of the present application, an emotion recognition device based on a large model is also provided. Figure 2 The device comprises the following units:

[0187] An acquisition unit is configured to acquire multimodal data to be processed; the multimodal data at least includes: text information, audio information, and image information input by a user; the image information at least includes: facial expression images and / or body movement images;

[0188] An analysis unit configured to input the multimodal data into an emotion recognition model to perform emotion state analysis and obtain a predicted emotion state corresponding to the user;

[0189] The output unit is configured to perform a confidence assessment based on the predicted emotional state, and generate a user emotion recognition result based on the predicted emotional state and the corresponding assessment result; the user emotion recognition result includes at least: an attention heat map for displaying the user's emotional state from multiple dimensions, and / or visual auxiliary analysis information; the auxiliary analysis information includes at least: voice emotion analysis, text emotion analysis, and facial expression emotion analysis.

[0190] Further optionally, the analysis unit inputs the multimodal data into an emotion recognition model to perform emotion state analysis to obtain a predicted emotion state corresponding to the user, and is configured to: extract corresponding emotion feature information under multimodality from the multimodal data; the emotion feature information includes at least one of the following: text emotion feature, acoustic emotion feature, visual emotion feature; perform multimodal fusion on the emotion feature information to obtain emotion fusion feature; input the emotion fusion feature into the emotion recognition model to perform emotion state analysis to obtain a predicted emotion state corresponding to the user.

[0191] Further optionally, the analysis unit extracts corresponding emotion feature information under multimodality from the multimodal data, and is configured as: extracting a multi-level text emotion feature vector from the text information through a generalized linear model; the text emotion feature vector includes at least: a word emotion feature vector at the word level, a sentence emotion feature vector at the sentence level, and a paragraph emotion feature vector at the context chapter level; performing multi-task learning on the text emotion feature vector to obtain micro-emotion change features implied in the text emotion feature vector; the micro-emotion change features include at least: modal particle features, implicit expression features; using the text emotion feature vector and the micro-emotion change features as the text emotion features.

[0192] Further optionally, the analysis unit extracts corresponding emotion feature information under multimodality from the multimodal data, and is configured to: convert the audio information into an acoustic physical vector through a Wav2Vec model; the acoustic physical vector includes at least: spectral features and / or prosodic features; the spectral features include at least: Mel-frequency cepstral coefficients MFCC, spectral centroid feature information; the prosodic features include at least: pitch curve features, speech rate change features, pause features, breathing features; the acoustic physical vector and the audio information are input into a HuBERT model to identify the emotional clues implied in the user's voiceprint to obtain the acoustic emotion features.

[0193] Further optionally, the analysis unit extracts corresponding emotion feature information under multimodality from the multimodal data, and is configured to: extract initial visual features under multi-scale from the image information through a Vision Transformer model; the initial visual features include at least: facial key point features, facial contour features, limb contour features, posture features, motion change features, and color distribution features; and perform emotion state recognition and change trend analysis on the initial visual features to obtain the visual emotion features.

[0194] Further optionally, before extracting the corresponding emotional feature information under multimodality from the multimodal data, the analysis unit is also configured to: split the multimodal data according to data type to obtain text preprocessing data, audio preprocessing data, and image preprocessing data; perform preprocessing operations on the text preprocessing data to obtain preprocessed text information; wherein, the preprocessing operations according to the text data type include: word segmentation, denoising, and standardization; perform preprocessing operations on the audio preprocessing data to obtain preprocessed audio information; wherein, the preprocessing operations according to the audio data type include: noise reduction, segmentation, and normalization; perform preprocessing operations on the image preprocessing data to obtain preprocessed image information; wherein, the preprocessing operations according to the image data type include: denoising, resizing, and color standardization.

[0195] Further optionally, the analysis unit performs multimodal fusion on the emotional feature information to obtain the emotional fusion feature, and is configured as follows: through a nonlinear projection matrix, the emotional feature information is mapped into a high-dimensional semantic space according to different modalities to obtain a multimodal mapping feature matrix; through adversarial training, the multimodal mapping feature matrix is ​​subjected to modal difference elimination processing to obtain a first mapping feature matrix; a multi-layer multi-head cross-attention mechanism is adopted to establish fine-grained correlations between different modalities in the first mapping feature matrix to obtain a second mapping feature matrix; based on the Transformer encoder improved by the meta-learning strategy, the second mapping feature matrix is ​​adaptively evaluated according to the importance of different modalities, and dynamic weight allocation is performed according to the evaluation results to obtain the emotional fusion feature.

[0196] Further optionally, the analysis unit, through adversarial training processing, performs modal difference elimination processing on the multimodal mapping feature matrix to obtain a first mapping feature matrix, is also configured to: establish corresponding gating unit models according to different modalities; use the gating unit model to filter and delete invalid matrix elements corresponding to different modalities from the multimodal mapping feature matrix; wherein the invalid matrix elements meet the preset filtering conditions corresponding to the respective modalities.

[0197] Further optionally, the analysis unit, before performing multimodal fusion on the emotion feature information to obtain the emotion fusion feature, is also configured to: if the amount of image information in the emotion feature information does not meet a preset image mode threshold, perform image data compensation on the emotion feature information.

[0198] Further optionally, the analysis unit inputs the emotion fusion feature into the emotion recognition model to perform emotion state analysis to obtain the user's corresponding predicted emotion state, and is configured to: perform emotion classification on the emotion fusion feature through a deep multi-layer perceptron to obtain the corresponding user emotion category; construct the emotion fusion feature into a corresponding emotion state transfer matrix based on a bidirectional long short-term memory network; the emotion state transfer matrix is ​​used to represent the dynamic evolution process between user emotions; a temporal attention mechanism is used to extract the user's emotion short-term fluctuation trend and emotion long-term change trend from the emotion fusion feature; and predict the user's emotion change tendency in the future time period and the corresponding predicted emotion category according to the user emotion category, the emotion state transfer matrix, the emotion short-term fluctuation trend and the emotion long-term change trend.

[0199] Further optionally, the output unit performs confidence assessment based on the predicted emotional state and is configured to use a Bayesian deep learning method to convert the predicted emotional state into an emotional state probability distribution variable for uncertainty assessment; in the inference stage of the emotion recognition model, different emotion recognition sub-networks are constructed through multiple forward propagations, and the emotional state probability distribution variables are processed using the different emotion recognition sub-networks to obtain corresponding multiple predicted emotional states; the confidence levels corresponding to the multiple predicted emotional states are evaluated based on Monte Carlo Dropout; a reliability quantitative assessment of the emotion category is performed based on the confidence levels, and the target predicted emotional state for the final output is determined based on the quantitative assessment results.

[0200] Further optionally, the output unit generates a user emotion recognition result based on the predicted emotional state and the corresponding evaluation result, and is configured to: generate a corresponding attention heat map based on the target predicted emotional state; and / or generate the visual auxiliary analysis information according to the predicted emotional state to improve the interpretability of the recognition result.

[0201] Further optionally, after the output unit generates the user emotion recognition result based on the predicted emotional state and the corresponding evaluation result, it is also configured to: adopt the user's feedback information on the user emotion recognition result, and perform online learning based on the feedback information to obtain model optimization parameters; use the model optimization parameters for the model adjustment process of the emotion recognition model to improve the emotion recognition accuracy of the emotion recognition model.

[0202] The system can implement various steps in the above method embodiment, which will not be expanded here.

[0203] See also Figure 3 , Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present application. Figure 3 As shown, an embodiment of the present application provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, an emotion recognition method based on a large model is implemented.

[0204] See also Figure 4 , Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present application. Figure 4 As shown, this embodiment provides a computer-readable storage medium 400, on which a computer program 411 is stored, and when the computer program 411 is executed by a processor, an emotion recognition method based on a large model is implemented. It should be noted that in the above embodiments, the descriptions of each embodiment have different focuses, and for the parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and variations.

Claims

1. A large model-based emotion recognition method, characterized in that: The method comprises: Acquire multimodal data to be processed; the multimodal data at least includes: text information, audio information, and image information input by a user; the image information at least includes: facial expression images and / or body movement images; Inputting the multimodal data into an emotion recognition model to perform emotion state analysis to obtain a predicted emotion state corresponding to the user; A confidence assessment is performed based on the predicted emotional state, and a user emotion recognition result is generated based on the predicted emotional state and the corresponding assessment result; the user emotion recognition result includes at least: an attention heat map for displaying the user's emotional state from multiple dimensions, and / or visual auxiliary analysis information; the auxiliary analysis information includes at least: voice emotion analysis, text emotion analysis, and facial expression emotion analysis.

2. The emotion recognition method based on a large model according to claim 1, characterized in that: The step of inputting the multimodal data into an emotion recognition model to perform emotion state analysis to obtain a predicted emotion state corresponding to the user includes: Extracting corresponding emotion feature information under multimodality from the multimodal data; the emotion feature information includes at least one of the following: text emotion feature, acoustic emotion feature, and visual emotion feature; Performing multimodal fusion on the emotion feature information to obtain emotion fusion features; The emotion fusion feature is input into the emotion recognition model to perform emotion state analysis to obtain the predicted emotion state of the user.

3. The emotion recognition method based on a large model according to claim 2, characterized in that: The extracting corresponding emotion feature information under multimodality from the multimodal data includes: Extracting a multi-level text sentiment feature vector from the text information through a generalized linear model; the text sentiment feature vector at least includes: a word-level sentiment feature vector, a sentence-level sentiment feature vector, and a context-paragraph-level sentiment feature vector; Performing multi-task learning on the text emotion feature vector to obtain the micro emotion change features implicit in the text emotion feature vector; the micro emotion change features at least include: modal particle features and implicit expression features; The text sentiment feature vector and the micro-sentiment change feature are used as the text sentiment feature.

4. The emotion recognition method based on a large model according to claim 2, characterized in that: The extracting corresponding emotion feature information under multimodality from the multimodal data includes: The audio information is converted into an acoustic physical vector through a Wav2Vec model; the acoustic physical vector includes at least: a spectrum feature and / or a rhythm feature; the spectrum feature includes at least: a Mel frequency cepstral coefficient MFCC and a spectrum centroid feature information; the rhythm feature includes at least: a pitch curve feature, a speech rate change feature, a pause feature, and a breathing feature; The acoustic physical vector and the audio information are input into the HuBERT model to identify the emotional clues implicit in the user's voiceprint and obtain the acoustic emotional features.

5. The emotion recognition method based on a large model according to claim 2, characterized in that: The extracting corresponding emotion feature information under multimodality from the multimodal data includes: Extracting initial visual features at multiple scales from the image information through a Vision Transformer model; the initial visual features at least include: facial key point features, facial contour features, limb contour features, posture features, motion change features, and color distribution features; The initial visual features are subjected to emotional state recognition and change trend analysis to obtain the visual emotional features.

6. The emotion recognition method based on a large model according to claim 2, characterized in that: Before extracting the corresponding emotion feature information under multimodality from the multimodal data, the method further includes: Splitting the multimodal data according to data types to obtain text preprocessing data, audio preprocessing data, and image preprocessing data; The text preprocessing data is subjected to a preprocessing operation to obtain preprocessed text information; wherein the preprocessing operation according to the text data type includes: word segmentation, denoising, and standardization; The audio preprocessing data is preprocessed to obtain preprocessed audio information; wherein the preprocessing operation according to the audio data type includes: noise reduction, segmentation, and normalization; The image preprocessing data is subjected to a preprocessing operation to obtain preprocessed image information; wherein the preprocessing operation according to the image data type includes: denoising, resizing, and color standardization.

7. The emotion recognition method based on a large model according to claim 2, characterized in that: The multi-modal fusion of the emotion feature information to obtain the emotion fusion feature includes: By using a nonlinear projection matrix, the emotional feature information is mapped into a high-dimensional semantic space according to different modalities to obtain a multimodal mapping feature matrix; Performing modal difference elimination processing on the multimodal mapping feature matrix through adversarial training processing to obtain a first mapping feature matrix; A multi-layer multi-head cross attention mechanism is used to establish fine-grained correlations between different modalities in the first mapping feature matrix to obtain a second mapping feature matrix; Based on the Transformer encoder improved by the meta-learning strategy, the second mapping feature matrix is ​​adaptively evaluated according to the importance of different modalities, and dynamic weight allocation is performed according to the evaluation results to obtain the emotion fusion feature.

8. The emotion recognition method based on a large model according to claim 7, characterized in that: Before performing modal difference elimination processing on the multimodal mapping feature matrix through adversarial training processing to obtain a first mapping feature matrix, the method further includes: Establish corresponding gating unit models according to different modes; The gate control unit model is used to screen and delete invalid matrix elements corresponding to different modes from the multi-modal mapping feature matrix; wherein the invalid matrix elements satisfy the preset screening conditions corresponding to the respective modes; Before performing multimodal fusion on the emotion feature information to obtain the emotion fusion feature, the method further includes: If the amount of image information in the emotion characteristic information does not meet a preset image mode threshold, image data compensation is performed on the emotion characteristic information.

9. The emotion recognition method based on a large model according to claim 2, characterized in that: The step of inputting the emotion fusion feature into the emotion recognition model to perform emotion state analysis to obtain the predicted emotion state of the user includes: The emotion fusion feature is used to classify emotions through a deep multi-layer perceptron to obtain corresponding user emotion categories; The emotion fusion feature is constructed into a corresponding emotion state transfer matrix based on a bidirectional long short-term memory network; the emotion state transfer matrix is ​​used to represent the dynamic evolution process between user emotions; A temporal attention mechanism is used to extract the user's short-term emotion fluctuation trend and long-term emotion change trend from the emotion fusion feature; According to the user emotion category, the emotion state transfer matrix, the emotion short-term fluctuation trend and the emotion long-term change trend, the emotion change tendency of the user in the future time period and the corresponding predicted emotion category are predicted.

10. An emotion recognition device based on a large model, characterized in that: The device comprises the following units, wherein: An acquisition unit is configured to acquire multimodal data to be processed; the multimodal data at least includes: text information, audio information, and image information input by a user; the image information at least includes: facial expression images and / or body movement images; An analysis unit configured to input the multimodal data into an emotion recognition model to perform emotion state analysis and obtain a predicted emotion state corresponding to the user; The output unit is configured to perform a confidence assessment based on the predicted emotional state, and generate a user emotion recognition result based on the predicted emotional state and the corresponding assessment result; the user emotion recognition result includes at least: an attention heat map for displaying the user's emotional state from multiple dimensions, and / or visual auxiliary analysis information; the auxiliary analysis information includes at least: voice emotion analysis, text emotion analysis, and facial expression emotion analysis.

Images