Robot emotion recognition method, system and storage medium based on AIGC

Abstract

The present application relates to the field of data processing technology, and discloses a robot emotion recognition method, system and storage medium based on AIGC. The method comprises: collecting multimodal emotion data through a target robot, performing cross-modal expansion processing on the multimodal emotion data, and obtaining an extended emotion sample set; performing feature extraction on the extended emotion sample set to obtain an emotion feature vector; performing situation-related data mapping on the emotion feature vector to obtain context-related emotion representation data; performing personalized feature analysis on the context-related emotion representation data to obtain mapped user emotion data; performing multimodal data fusion and classification on the mapped user emotion data to obtain emotion classification data; performing multi-dimensional analysis on the emotion classification data through a self-supervised learning algorithm to obtain emotion expression parameters and interaction strategy recommendations. The present application improves the efficiency and accuracy of robot emotion recognition based on AIGC.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a robot emotion recognition method, system and storage medium based on AIGC. Background Art

[0002] Existing robot emotion recognition technologies usually rely on single or limited modality data, such as facial expression recognition or speech emotion analysis. These methods can achieve a certain recognition accuracy in a controlled environment, but often perform poorly in complex practical application scenarios. In addition, traditional emotion recognition algorithms usually use predefined feature extraction and classification methods, lack of consideration of individual differences and situational factors, resulting in a lack of flexibility and personalization in the recognition results.

[0003] However, these existing technologies have some significant shortcomings. First, single-modality emotion recognition cannot fully capture the complexity and diversity of human emotions. Second, predefined features and classification methods are difficult to adapt to the differences in emotional expression of different individuals and emotional changes in different situations. Furthermore, existing technologies often lack the interpretability of recognition results, making it difficult for users to understand and trust the robot's emotional judgment. Finally, existing emotion recognition systems are usually static, lack the ability to continuously learn and adapt, and cannot continue to improve with interaction with users. Summary of the invention

[0004] The present application provides an AIGC-based robot emotion recognition method, system, and storage medium for improving the efficiency and accuracy of AIGC-based robot emotion recognition.

[0005] In a first aspect, the present application provides a robot emotion recognition method based on AIGC, and the robot emotion recognition method based on AIGC includes: collecting multimodal emotion data through a target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and performing cross-modal expansion processing on the multimodal emotion data to obtain an extended emotion sample set; performing feature extraction on the extended emotion sample set to obtain an emotion feature vector; performing context-related data mapping on the emotion feature vector to obtain context-related emotion representation data; performing personalized feature analysis on the context-related emotion representation data to obtain mapped user emotion data; performing multimodal data fusion and classification on the mapped user emotion data to obtain emotion classification data; performing multi-dimensional analysis on the emotion classification data through a self-supervised learning algorithm to obtain emotion expression parameters and interaction strategy recommendations.

[0006] In combination with the first aspect, in a first implementation of the first aspect of the present application, the multimodal emotion data is collected by a target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and the multimodal emotion data is cross-modally expanded to obtain an extended emotion sample set, including: collecting multimodal emotion data by a target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and performing emotion semantic segmentation on the visual data to obtain an emotion region feature map; performing dynamic expression sequence extraction on the emotion region feature map to obtain time series emotion feature data; performing spectral domain feature decomposition on the audio data to obtain acoustic features, and performing rhythmic pattern analysis on the acoustic features to obtain an emotion rhythm feature vector; performing emotion vocabulary recognition on the text data to obtain an emotion word embedding matrix, and performing contextual semantic enhancement on the emotion word embedding matrix to obtain an expanded emotion text representation; The temporal emotion feature data, the emotion rhythm feature vector and the extended emotion text representation are cross-modally aligned to obtain a target emotion feature space; data augmentation is performed on the target emotion feature space through a generative adversarial network to obtain an extended emotion sample set, and the extended emotion sample set is subjected to emotion intensity gradient processing to obtain continuously changing emotion samples; marginal emotion case synthesis is performed on the continuously changing emotion samples to obtain the extended emotion sample set.

[0007] In combination with the first aspect, in a second implementation method of the first aspect of the present application, the feature extraction of the extended emotion sample set to obtain an emotion feature vector includes: performing multimodal emotion feature mapping on the extended emotion sample set to obtain an initial emotion high-dimensional feature space, and extracting emotion semantic information from the initial emotion high-dimensional feature space to obtain an emotion semantic enhanced feature matrix; generating emotion virtual features for the emotion semantic enhanced feature matrix to obtain an emotion supplement feature set, and performing emotion feature fusion on the emotion supplement feature set to obtain a fused emotion feature representation; performing emotion self-supervised reconstruction on the fused emotion feature representation to obtain emotion reconstruction feature data, and performing data comparison analysis on the emotion reconstruction feature data and the extended emotion sample set to obtain a key emotion feature set; performing cross-modal emotion feature alignment on the key emotion feature set to obtain emotion consistency feature mapping data; performing dynamic emotion dimension adjustment on the emotion consistency feature mapping data to obtain an adaptive emotion feature vector; performing emotion information compression on the adaptive emotion feature vector to obtain an efficient emotion feature encoding; and performing emotion feature enhancement on the efficient emotion feature encoding through a contrast learning algorithm to obtain the emotion feature vector.

[0008] In combination with the first aspect, in a third implementation method of the first aspect of the present application, the emotion feature vector is mapped to context-related data to obtain context-related emotion representation data, including: semantic decomposition of the emotion feature vector to obtain multidimensional emotion semantic data, and context association analysis of the multidimensional emotion semantic data to obtain a preliminary context-emotion correspondence relationship; knowledge graph mapping of the preliminary context-emotion correspondence relationship to obtain a structured context emotion network; dynamic expansion of the structured context emotion network to obtain a context knowledge base, and emotion reaction spectrum generation for the context knowledge base to obtain a diversified emotion response pattern; time series prediction of the diversified emotion response pattern to obtain an emotion change trend graph, and scene derivation of the emotion change trend graph to obtain a multi-path emotion development matrix; similarity calculation of the multi-path emotion development matrix to obtain an emotion-context association pattern set, and comparative sample construction of the emotion-context association pattern set to obtain fine-grained context-emotion mapping data; the fine-grained context-emotion mapping data and the emotion feature vector are fused through a graph attention network algorithm to obtain context-related emotion representation data.

[0009] In combination with the first aspect, in a fourth implementation method of the first aspect of the present application, the personalized feature analysis of the context-related emotion representation data to obtain mapped user emotion data includes: performing user feature extraction on the context-related emotion representation data to obtain an initial user emotion feature set, and performing sample synthesis on the initial user emotion feature set to obtain extended user emotion expression data; performing expression-voice-text joint analysis on the extended user emotion expression data to obtain a multimodal personalized feature matrix, and performing style conversion on the multimodal personalized feature matrix to obtain user-specific emotion expression samples; performing contextual relevance analysis on the user-specific emotion expression samples to obtain a personalized emotion response pattern, and performing sequence prediction on the personalized emotion response pattern to obtain a user emotion response trend graph; performing semantic mapping on the user emotion response trend graph to obtain a personalized emotion vocabulary, and performing frequency-context analysis on the personalized emotion vocabulary to obtain a user emotion expression habit library; performing difference comparison on the user emotion expression habit library to obtain a personalized emotion feature deviation matrix; integrating the personalized emotion feature deviation matrix and the context-related emotion representation data through a transfer learning algorithm to obtain the mapped user emotion data.

[0010] In combination with the first aspect, in a fifth implementation of the first aspect of the present application, the mapped user emotion data is subjected to multimodal data fusion and classification to obtain emotion classification data, including: performing modal integrity detection on the mapped user emotion data to obtain a modal missing label matrix, and performing data completion processing on the mapped user emotion data according to the modal missing label matrix to obtain a target multimodal emotion data set; performing cross-modal consistency evaluation on the target multimodal emotion data set to obtain a modal consistency score vector, and performing threshold screening on the modal consistency score vector to obtain a high-confidence multimodal data subset; performing dynamic weight assignment on the high-confidence multimodal data subset to obtain a modal fusion weight matrix; performing weighted fusion on the modal fusion weight matrix and the high-confidence multimodal data subset to obtain a preliminary fusion feature vector, and performing context constraint processing on the preliminary fusion feature vector to obtain a context-sensitive fusion feature; performing counterfactual analysis on the context-sensitive fusion feature to obtain a modal importance assessment report, and performing decision rule generation on the modal importance assessment report to obtain a dynamic decision tree structure; The dynamic decision tree structure is mapped into a multi-dimensional sentiment space through an integrated learning algorithm to obtain sentiment classification data.

[0011] In combination with the first aspect, in a sixth implementation method of the first aspect of the present application, the emotion classification data is analyzed in multiple dimensions by a self-supervised learning algorithm to obtain emotion expression parameters and interaction strategy recommendations, including: generating multi-granular interpretations of the emotion classification data to obtain a hierarchical emotion interpretation structure, and semantically decomposing the hierarchical emotion interpretation structure to obtain an emotion cause element set; performing weight calculations on the emotion cause element set to obtain an emotion contribution matrix, and performing threshold analysis on the emotion contribution matrix to obtain key emotion triggering factors; constructing counterexamples for the key emotion triggering factors to obtain an emotion boundary case set, and performing similarity calculations on the emotion boundary case set to obtain an emotion difference space map; performing trajectory prediction on the emotion difference space map to obtain an emotion development script library, and performing context mapping on the emotion development script library to obtain an emotion-environment association network; generating interactive questions for the emotion-environment association network to obtain deep interpretation question and answer pairs; performing feature extraction and clustering on the deep interpretation question and answer pairs by a self-supervised learning algorithm to obtain emotion expression parameters and interaction strategy recommendations.

[0012] In a second aspect, the present application provides a robot emotion recognition system based on AIGC, and the robot emotion recognition system based on AIGC includes:

[0013] A collection module, used to collect multimodal emotion data through a target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and perform cross-modal expansion processing on the multimodal emotion data to obtain an extended emotion sample set;

[0014] An extraction module, used for performing feature extraction on the extended emotion sample set to obtain an emotion feature vector;

[0015] A mapping module, used for performing context-related data mapping on the emotion feature vector to obtain context-related emotion representation data;

[0016] An analysis module, used to perform personalized feature analysis on the context-related emotion representation data to obtain mapped user emotion data;

[0017] A fusion module, used to perform multimodal data fusion and classification on the mapped user emotion data to obtain emotion classification data;

[0018] The analysis module is used to perform multi-dimensional analysis on the sentiment classification data through a self-supervised learning algorithm to obtain sentiment expression parameters and interaction strategy suggestions.

[0019] The third aspect of the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the above-mentioned AIGC-based robot emotion recognition method.

[0020] In the technical solution provided by the present application, by collecting and processing multimodal emotional data, including visual, audio and text data, the information source of emotion recognition is expanded, making the recognition result more comprehensive and accurate, and the cross-modal expansion processing further enriches the sample set and enhances the generalization ability of the model. Secondly, the feature extraction technology adopted by this method, especially the extraction of emotional semantic information and the generation of emotional virtual features, significantly improves the expressiveness and richness of emotional features. Through emotional self-supervised reconstruction and cross-modal feature alignment, it is ensured that the extracted features can accurately capture key emotional information. Furthermore, this method introduces context-related data mapping, which enables emotion recognition to take into account the specific context environment, greatly improving the accuracy and applicability of recognition. By constructing a contextual knowledge base and generating diversified emotional response patterns, it is possible to better understand and predict emotional expressions in different situations. In addition, the personalized feature analysis link of this method allows the system to adapt to the unique emotional expressions of different users, and through user-specific emotional expression samples and personalized emotional response patterns, highly personalized emotion recognition is achieved. In the multimodal data fusion and classification stage, this method adopts modal integrity detection and cross-modal consistency evaluation, which effectively handles the data missing or inconsistency problems that may occur in practical applications. The introduction of dynamic weight allocation and context-sensitive fusion features further improves the flexibility and robustness of the fusion process. Finally, this method uses a self-supervised learning algorithm to perform multi-dimensional analysis, which can not only obtain accurate emotion classification results, but also generate detailed emotion expression parameters and interaction strategy recommendations. This multi-level explanation generation and counterexample construction mechanism greatly enhances the interpretability of the system, allowing users to better understand and trust the robot's emotional judgment. At the same time, the construction of the emotion development script library and the emotion-environment association network provides the ability to predict and adapt to complex emotional changes, improving the efficiency and accuracy of robot emotion recognition based on AIGC. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0022] Figure 1 This is a schematic diagram of an embodiment of a robot emotion recognition method based on AIGC in an embodiment of the present application;

[0023] Figure 2 This is a schematic diagram of an embodiment of a robot emotion recognition system based on AIGC in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application provide a robot emotion recognition method, system and storage medium based on AIGC. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of the robot emotion recognition method based on AIGC includes:

[0026] Step S101: collecting multimodal emotion data through a target robot, wherein the multimodal emotion data includes: visual data, audio data, and text data, and performing cross-modal expansion processing on the multimodal emotion data to obtain an expanded emotion sample set;

[0027] It is understandable that the execution subject of the present application may be a robot emotion recognition system based on AIGC, or a terminal or a server, which is not limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0028] Specifically, the robot collects visual, audio and text data through built-in high-resolution cameras, high-sensitivity microphone arrays and natural language processing modules. Visual data includes facial expression image sequences, body postures and gestures; audio data includes voice signals, intonation curves and speech speed change information; text data includes dialogue content, emotional vocabulary and semantic structure.

[0029] During the data collection process, the robot will record timestamps at the same time to ensure the synchronization of data in different modalities. For example, when a user says "I am very happy today", the robot will simultaneously capture the user's facial expressions, voice features and text content. Visual data is collected at a rate of 30 frames per second, audio data is recorded at a sampling rate of 44.1kHz, and text data is transcribed in real time.

[0030] After collecting the raw data, the robot performs cross-modal expansion processing on the multimodal emotional data to enrich the emotional sample set. First, the data of each modality is preprocessed: the facial key point detection algorithm is applied to the visual data to extract the coordinate sequence of 68 facial landmark points; the audio data is subjected to spectrum analysis to extract acoustic features such as fundamental frequency and resonance peaks; the text data is subjected to word segmentation, part-of-speech tagging, and dependency syntax analysis.

[0031] The robot uses cross-modal generation algorithms for data expansion. Specifically, it uses conditional generative adversarial networks (cGANs) to generate synthetic data of other modalities based on data of one modality. For example, given the text "I am frustrated", cGAN can generate corresponding facial expression images and voice samples. This process not only increases the number of samples, but also improves the diversity and representativeness of the data.

[0032] To further enhance the data quality, the robot applies a style transfer algorithm to improve the generated data, ensuring that the generated data has a more natural and realistic style while maintaining the emotional semantics. For example, the generated facial expression images are style-transferred to make them closer to the texture and lighting characteristics of real faces. In addition, this scheme also designs an emotion intensity and subtle change generator. This generator is based on interpolation technology to create continuously changing samples between basic emotions. For example, a series of transitional emotion samples are generated between "sadness" and "anger" to capture the gradual process of emotions. This step is crucial to improve the model's sensitivity to subtle emotional changes. Finally, the robot uses adversarial generative network technology to create difficult-to-distinguish edge emotion cases. These cases are located at the boundaries of different emotion categories and are extremely important for training robust emotion recognition systems. For example, subtle expressions between "surprise" and "fear" are generated, and these samples can help the model learn more refined emotion distinction capabilities.

[0033] Through this series of cross-modal expansion processing, the original multimodal emotion dataset has been significantly expanded. Assume that the original dataset contains 1,000 samples, each of which includes a facial expression image, a 3-second voice, and a sentence of text. After cross-modal generation, the number of samples may increase to 5,000. Style transfer and emotion intensity variation generation further expand the number of samples to 15,000. Finally, after adding edge emotion cases, the final extended emotion sample set may contain more than 20,000 high-quality and diverse emotion samples. For example, suppose the robot initially collects a sample of a user saying "This is really great". The original data includes a smiling face image (resolution of 256x256 pixels), a 2-second voice file (showing an upward tone and a fast speaking speed), and the corresponding text. After cross-modal expansion processing, the robot generates 10 variants, including images of different degrees of smiles, voice files with slightly different intonations and speeds, and synonymous expressions such as "It's really great" and "It makes people happy". Through style transfer, these generated images present a more natural light and shadow effect while maintaining the smile. The emotion intensity change generator creates a series of transitional expressions from smile to laugh. Finally, the edge emotion case generator produces some smiles with a little surprise or doubt. In this way, from one original sample, the robot finally obtained about 50 rich and diverse emotion samples, which greatly enhanced the training effect of the subsequent emotion recognition model.

[0034] Step S102: extracting features from the extended emotion sample set to obtain an emotion feature vector;

[0035] Specifically, multimodal emotion feature mapping is first performed on the extended emotion sample set to map visual, audio, and text data into a unified high-dimensional feature space. Specifically, for visual data, a pre-trained convolutional neural network (such as ResNet-50) is used to extract deep features of facial expressions and body postures; for audio data, Mel-frequency cepstral coefficients (MFCC) and an acoustic feature extractor based on a recurrent neural network are used to obtain acoustic features; for text data, pre-trained language models such as BERT are applied to extract semantic features. These features are concatenated into an initial high-dimensional feature vector to form an initial emotion high-dimensional feature space. Emotional semantic information is extracted from the initial emotion high-dimensional feature space to enhance the emotional expression ability of the features. This step uses an attention mechanism to focus on feature dimensions that are closely related to emotional expression. For example, for facial expressions, the attention mechanism pays more attention to features in the eye and mouth regions; for speech, it pays more attention to changes in pitch and speech rate. This process generates an emotion semantics enhanced feature matrix, in which each feature dimension is accompanied by a weight representing the emotional relevance.

[0036] Emotional virtual features are generated for the emotional semantic enhancement feature matrix. This step uses a generative adversarial network (GAN) to create some "hypothetical" features to supplement the existing feature set. For example, for a sample that shows mild sadness, GAN may generate a series of virtual features that show different degrees of sadness. These virtual features together with the original features constitute the emotional supplementary feature set, which greatly enriches the diversity of features. Then, the emotional supplementary feature set is subjected to emotional feature fusion, a process that uses a multi-head self-attention mechanism to allow different parts of the feature set to interact and influence each other. The fusion process considers the correlation and complementarity between different modal features to generate a fused emotional feature representation. This representation can more comprehensively capture the emotional information in multimodal data. In order to ensure that the extracted features can accurately reconstruct the original data, the fused emotional feature representation is subjected to emotional self-supervised reconstruction. This process uses an autoencoder to try to regenerate the original multimodal data from the extracted features. By minimizing the reconstruction error, it is ensured that the features retain the key emotional information of the original data. The reconstruction process produces emotional reconstructed feature data.

[0037] The sentiment reconstruction feature data is compared with the original extended sentiment sample set. This step uses contrastive learning methods to identify the features that best distinguish different sentiment states. The key sentiment feature set is screened by calculating the distance between the features of positive sample pairs (same sentiment) and negative sample pairs (different sentiment). The key sentiment feature set is cross-modal sentiment feature alignment. This process uses techniques such as canonical correlation analysis (CCA) to find the maximum correlation between features of different modalities and ensure that features of different modalities are semantically consistent. This step produces sentiment consistency feature mapping data.

[0038] In order to adapt to different emotion recognition tasks, the emotion consistency feature map data is dynamically adjusted for emotion dimension. This process uses dimensionality reduction techniques such as principal component analysis (PCA) to dynamically adjust the feature dimension according to the needs of specific tasks and generate adaptive emotion feature vectors.

[0039] Finally, the emotional information of the adaptive emotional feature vector is compressed, and efficient emotional feature coding is generated using technologies such as autoencoders. The emotional features of the efficient emotional feature coding are enhanced through a contrastive learning algorithm, and finally the emotional feature vector is obtained.

[0040] For example, suppose there is an extended emotion sample containing a video clip of a person smiling and saying "Today is a good day". First, extract a 256x256 pixel facial image from the video, and use ResNet-50 to extract 2048-dimensional visual features; extract a 3-second speech clip from the audio, and calculate 13-dimensional MFCC features; and use BERT to extract 768-dimensional semantic features from the text. These features are concatenated and fused into an initial high-dimensional feature vector of 2829 dimensions.

[0041] After extracting emotional semantic information, the attention mechanism may give higher weights to visual features of the eye and mouth areas, tonal features of the voice, and words such as "good day" in the text, forming a weighted feature matrix. The emotional virtual feature generation step may create some virtual features that express different degrees of pleasure, such as slightly excited or calm variants. In the emotional feature fusion stage, the multi-head self-attention mechanism may strengthen the connection between the smile features in the vision, the pleasant tone in the voice, and the positive words in the text, generating a more comprehensive emotional representation. The self-supervised reconstruction process attempts to reconstruct the original image, voice, and text from this representation to ensure that no key emotional information is lost.

[0042] Through contrastive learning, it is found that the degree of smile, the upward amplitude of voice tone, and the phrase "good day" are the key features that distinguish this sample from other emotional samples. Cross-modal feature alignment ensures that smiles, pleasant voice tones, and positive text are semantically consistent. Finally, after dimensionality adjustment and information compression, the original 2829-dimensional features can be compressed into a 128-dimensional efficient emotional feature vector. This vector not only contains the core emotional information of the original sample, but also enhances its ability to express and distinguish different emotional states through steps such as virtual feature generation and contrastive learning.

[0043] Step S103, mapping the emotion feature vector to context-related data to obtain context-related emotion representation data;

[0044] Specifically, the sentiment feature vector is semantically decomposed, and the non-negative matrix factorization (NMF) algorithm is used to decompose the high-dimensional feature vector into multiple semantic components, each of which represents a specific sentiment dimension or expression, thereby obtaining multi-dimensional sentiment semantic data. Subsequently, these multi-dimensional sentiment semantic data are context-related analyzed, and the conditional random field (CRF) model is used to capture the dependency between sentiment expression and environmental factors, generating preliminary context-emotion correspondence.

[0045] The preliminary context-emotion correspondences are transformed into a structured context-emotion network through knowledge graph mapping technology. This step uses the predefined emotion ontology and context knowledge base to organize the discrete correspondences into a coherent semantic network. Specifically, knowledge graph embedding methods such as TransE are used to map emotion concepts, context factors and the relationships between them into a unified vector space to form a structured context-emotion network.

[0046] In order to adapt to the dynamically changing emotional expression environment, the structured situational emotion network is dynamically expanded. This process uses an incremental learning algorithm to continuously update and expand the network structure based on the newly observed emotion-situation correspondence to obtain a more comprehensive and up-to-date situational knowledge base. At the same time, the emotional response spectrum is generated for this situational knowledge base, and the variational autoencoder (VAE) technology is used to generate a series of possible emotional responses based on the known emotion-situation correspondence, forming a diversified emotional response pattern.

[0047] The long short-term memory network (LSTM) model is used to predict the time series of diversified emotional response patterns, taking into account the time evolution characteristics of emotional states, predicting the possible trend of emotional changes, and generating an emotional change trend graph. Based on this trend graph, the scene is further derived, and the Monte Carlo tree search (MCTS) algorithm is used to simulate multiple possible paths of emotional development in different situations to obtain a multi-path emotional development matrix. The multi-path emotional development matrix is ​​similarity-calculated, and the representative emotion-situation evolution patterns are identified using cosine similarity and other measurement methods to form an emotion-situation association pattern set. In order to further refine these patterns, the emotion-situation association pattern set is constructed by contrasting samples, and a series of emotion-situation sample pairs with slight changes are generated using the adversarial generative network (GAN) technology, thereby obtaining fine-grained situation-emotion mapping data.

[0048] The fine-grained context-emotion mapping data and the original emotion feature vector are fused through the Graph Attention Network (GAT) algorithm. GAT can adaptively assign different weights to different nodes, thereby effectively integrating graph structure information and node features, and finally obtaining context-related emotion representation data.

[0049] For example, suppose there is an emotional feature vector that represents a person smiling at work and saying "this project is really challenging". This 256-dimensional vector is first decomposed into 10 emotional semantic components through NMF, including positive emotions, stress, excitement, etc. The CRF model analyzes the association between these components and situational factors such as "workplace" and "project challenges" to obtain preliminary situation-emotion correspondences. These relationships are mapped to a knowledge graph, where concepts such as "smile", "challenging project", and "work environment" are represented as nodes in the graph, and the relationships between them (such as "shown", "existing") are represented as edges. Through the TransE algorithm, each node and edge is embedded in a 100-dimensional vector space.

[0050] During the dynamic expansion process, assuming that 10 new samples in similar scenarios are newly observed, the incremental learning algorithm integrates this new information into the knowledge graph. VAE then generates 100 possible emotional reactions based on this expanded knowledge base, including different degrees of combinations such as "excitement", "pressure", and "confidence". The LSTM model then predicts the changing trends of these emotional states in the next hour and generates a 100x60 matrix representing the intensity changes of the 100 emotional states at 60 time points. The MCTS algorithm simulates 1,000 possible situational changes (such as project progress, colleague feedback, etc.) and generates a 1,000x100x60 multi-path emotional development matrix.

[0051] By calculating the cosine similarity between these 1000 paths, 10 typical emotion-context association patterns are identified. GAN generates 1000 slightly changed samples based on these 10 patterns to form fine-grained context-emotion mapping data. Finally, the GAT algorithm fuses these fine-grained mapping data with the original 256-dimensional emotion feature vector. GAT contains 3 attention layers, each with 8 attention heads. The first layer focuses on emotional features, the second layer focuses on contextual factors, and the third layer integrates the two. Finally, a 512-dimensional vector is output, which is the context-related emotion representation data. This vector not only contains the original emotional information, but also encodes rich contextual information and potential emotional development trends.

[0052] Step S104: performing personalized feature analysis on the context-related emotion representation data to obtain mapped user emotion data;

[0053] Specifically, user features are extracted from context-related emotion representation data. The Siamese network structure in the deep neural network is used to compare the user's historical emotion data with the current emotion representation, extract the user's unique emotion expression pattern, and obtain the initial user emotion feature set. This feature set contains the characteristics of the user's emotion expression in different situations, such as changes in tone and subtle differences in facial expressions. Samples are synthesized for the initial user emotion feature set, and the variational autoencoder (VAE) technology is used to generate more user-specific emotion samples. By learning the potential distribution of user emotion features, VAE can generate new samples that are consistent with the user's style but slightly changed, thereby obtaining extended user emotion expression data.

[0054] Perform expression-speech-text joint analysis on the extended user emotion expression data. This process uses multimodal fusion technology, specifically using tensor decomposition methods to map data of different modalities into a common feature space. By calculating the mutual information between modalities, the user's unique patterns in expression, speech and text are identified, thereby obtaining a multimodal personalized feature matrix. In order to further capture the user's emotional expression style, the multimodal personalized feature matrix is ​​style-transformed. Here, a cycle-consistent adversarial network (CycleGAN) is used to map the user's emotional expression features with the general emotional expression pattern to generate user-specific emotional expression samples. These samples retain the user's personalized characteristics while having sufficient variability, which helps the generalization ability of subsequent models.

[0055] Perform situational relevance analysis on user-specific emotion expression samples. Use the conditional random field (CRF) model to associate the user's emotion expression with different situational factors (such as social occasions, work environment, etc.) to obtain a personalized emotion response pattern. This pattern reflects the user's tendency to express emotion in different situations. In order to predict the user's emotion change trend, perform sequence prediction on the personalized emotion response pattern. Use the long short-term memory network (LSTM) model to consider the time series information of the user's historical emotion state, predict possible emotion changes in the future, and generate a user emotion response trend chart. This trend chart shows the dynamic process of the user's emotion state changing over time and situation.

[0056] Semantically map the user's emotional response trend graph, and use word embedding technologies such as Word2Vec to convert the emotional state into corresponding language descriptions to form a personalized emotional vocabulary. This vocabulary contains the user's commonly used emotional expression vocabulary and its frequency of use. Perform frequency-context analysis on the personalized emotional vocabulary. Use the TF-IDF (term frequency-inverse document frequency) algorithm to calculate the importance weight of each emotional word in different contexts to obtain the user's emotional expression habit library. This habit library reflects the emotional expression methods that users tend to use in specific contexts. In order to highlight the user's unique emotional expression characteristics, the user's emotional expression habit library is compared. Using cosine similarity calculation, the user's emotional expression pattern is compared with the general pattern, the most personalized features are identified, and a personalized emotional feature deviation matrix is ​​generated.

[0057] The personalized emotion feature deviation matrix and context-related emotion representation data are integrated through the transfer learning algorithm. Specifically, the domain adversarial neural network (DANN) is used to adapt the general emotion feature domain to the user-specific emotion feature domain to obtain the final mapped user emotion data.

[0058] For example: Suppose there is a user whose 1,000 emotional expression data in the past 30 days have been collected by the system. First, the Siamese network compares these historical data with the current 512-dimensional context-related emotional representation to extract a 256-dimensional initial user emotional feature vector. VAE then generates 5,000 new user-specific emotional samples based on this vector, each of which is a 256-dimensional vector. The multimodal fusion process maps the expression, voice, and text features of these 5,000 samples to a 100-dimensional common feature space, forming a 5000x300 multimodal personalized feature matrix. CycleGAN then performs style conversion on each sample in this matrix with the general emotional pattern to generate 10,000 user-specific emotional expression samples.

[0059] The CRF model then analyzes the performance of these 10,000 samples in 10 different situations and generates a 10,000x10 personalized emotional response pattern matrix. Based on this matrix, LSTM predicts the user's emotional changes in these 10 situations in the next 24 hours and generates a 10x24 user emotional response trend chart. Word2Vec maps this trend chart to a vocabulary of 500 emotional words, each represented by a 50-dimensional vector. The TF-IDF algorithm calculates the weights of these 500 words in 10 situations to form a 500x10 user emotional expression habit matrix. By calculating the cosine similarity between this matrix and the general expression pattern, the 100 most personalized features are identified to form a 100x10 personalized emotional feature deviation matrix. Finally, DANN integrates this deviation matrix with the original 512-dimensional context-related emotional representation and outputs a 1024-dimensional mapped user emotional data vector. This vector not only contains general emotional information, but also encodes the user's unique emotional expression characteristics.

[0060] Step S105: performing multimodal data fusion and classification on the mapped user emotion data to obtain emotion classification data;

[0061] Specifically, the mapped user emotion data is tested for modality integrity, and a missing data detection algorithm, such as density-based local outlier detection (LOF), is used to identify whether there is a modality missing in each sample, and a modality missing labeling matrix is ​​generated. This matrix is ​​a two-dimensional array, in which each row represents a sample, each column represents a modality, a value of 1 indicates that the modality data exists, and 0 indicates that it is missing. According to the modality missing labeling matrix, the mapped user emotion data is processed for data completion. Here, the multimodal generative adversarial network (MM-GAN) technology is used to generate synthetic data of missing modalities based on existing modal data. MM-GAN contains multiple generators and discriminators, each corresponding to a modality, and cross-modal data generation is achieved through adversarial training. This step ensures that all samples have complete multimodal information and obtain the target multimodal emotion dataset.

[0062] The target multimodal sentiment dataset is evaluated for cross-modal consistency. The canonical correlation analysis (CCA) method is used to calculate the correlation between different modal features and generate a modal consistency score vector. CCA measures the degree of consistency between different modalities by finding the projection direction that maximizes the correlation between different modal features. Each element in the score vector represents the modal consistency score of a sample. Then, the modal consistency score vector is threshold screened, and an adaptive threshold selection algorithm, such as Otsu's method, is used to determine the optimal segmentation threshold and screen out a high-confidence multimodal data subset. This subset contains samples with high inter-modal consistency, providing a reliable basis for subsequent fusion.

[0063] Dynamically assign weights to high-confidence multimodal data subsets, and use attention mechanisms such as Multi-Head Attention to dynamically calculate the weight of each modality based on the importance of different modalities in the current emotion recognition task to form a modality fusion weight matrix. This matrix reflects the contribution of different modalities in different samples and emotion categories. Subsequently, the modality fusion weight matrix and the high-confidence multimodal data subset are weighted fused. Using tensor fusion methods such as Tucker decomposition, the features of different modalities and their weights are integrated into a unified multimodal representation to obtain a preliminary fused feature vector. Tucker decomposition can capture high-order correlations between different modalities and generate a richer and more compact fusion representation.

[0064] The preliminary fused feature vector is processed with context constraints. The conditional random field (CRF) model is used to combine the fused features with the previously obtained context information, and the context dependency of emotional expression is considered to generate context-sensitive fused features. The CRF model can model long-range dependencies in sequence data, making the emotion recognition results more consistent with the actual context. In order to evaluate the impact of different modalities on emotion judgment, counterfactual analysis is performed on the context-sensitive fused features. Causal reasoning methods such as counterfactual regression are used to simulate the emotion judgment results when different modalities are missing or changed, and a modality importance assessment report is generated. This report tests the importance of each modality in the recognition of different emotion categories. Based on the modality importance assessment report, decision rules are generated. A decision tree algorithm such as C4.5 is used to construct a decision structure that can adapt to different emotion categories and modality combinations according to the importance and feature performance of different modalities, and a dynamic decision tree structure is obtained. This decision tree structure can flexibly select the optimal decision path based on the input multimodal features.

[0065] Finally, the dynamic decision tree structure is mapped to a multi-dimensional sentiment space through an ensemble learning algorithm. Specifically, the Random Forest method is used to generate multiple decision trees, and their outputs are mapped to a multi-dimensional sentiment space to obtain the final sentiment classification data. Random Forest improves the generalization and robustness of the model through bagging algorithms and random feature selection. For example: suppose there is a user sentiment mapping dataset containing 1,000 samples, each of which contains features of three modalities: vision, audio, and text, with dimensions of 256, 128, and 64, respectively. First, the modality missing is detected through the LOF algorithm, and 50 samples are found to have modality missing, generating a 1000x3 modality missing labeling matrix. MM-GAN then completes the data for these 50 samples to ensure that all samples have complete data of the three modalities.

[0066] CCA is used to calculate the correlation between the three modalities and obtain a 1000-dimensional modality consistency score vector. The threshold is determined to be 0.7 through the image grayscale binarization algorithm, and 800 high-consistency samples are screened out. The multi-head self-attention mechanism then calculates the weights of the three modalities in these 800 samples and generates an 800x3 modality fusion weight matrix. Tucker decomposition compresses the features of the three modalities (256+128+64=448 dimensions) into a 200-dimensional preliminary fusion feature vector. The CRF model considers 10 different contextual factors and expands these 200-dimensional features into a 210-dimensional context-sensitive fusion feature. Counterfactual regression analysis simulates the absence of each modality and generates a 21-dimensional vector containing the importance scores of the three modalities in seven basic emotion categories.

[0067] Based on this importance assessment, the C4.5 algorithm constructs a decision tree with a maximum depth of 10. Finally, the random forest algorithm generates 100 such decision trees and maps their outputs to a 7-dimensional sentiment space (corresponding to seven basic emotions), obtaining the final 1000x7 sentiment classification data matrix. Each row in this matrix represents the probability distribution of a sample on the seven basic emotions. For example, [0.1, 0.2, 0.5, 0.05, 0.05, 0.05, 0.05] may mean that the probability of a sample being classified as "happy" is 50%, "surprised" is 20%, "neutral" is 10%, and other emotions are 5% each.

[0068] Step S106: Perform multi-dimensional analysis on the sentiment classification data through a self-supervised learning algorithm to obtain sentiment expression parameters and interaction strategy suggestions.

[0069] Specifically, the sentiment classification data is interpreted at multiple granularities, and the sentiment classification results are decomposed into interpretations at different abstract levels using a hierarchical self-attention mechanism. This mechanism contains multiple layers of self-attention networks, each of which focuses on sentiment features of different granularities, thereby generating a hierarchical sentiment interpretation structure. For example, for a sample classified as "sad", the highest layer may give an interpretation of "negative emotion", the middle layer gives a label of "sadness", and the lowest layer may point out specific manifestations such as "furrowed brows and low voice". The hierarchical sentiment interpretation structure is semantically decomposed, and the non-negative matrix factorization (NMF) algorithm is used to decompose the complex sentiment interpretation into basic sentiment causal elements. NMF decomposes the sentiment interpretation matrix into the product of two non-negative matrices, one of which represents the basic sentiment elements and the other represents the combined weights of these elements. This step obtains a set of sentiment causal elements, each of which represents a basic sentiment inducement or manifestation. Next, the set of sentiment causal elements is weighted, and the attention mechanism is used to calculate the importance of each element in different sentiment categories and contexts, generating a sentiment contribution matrix. Specifically, the use of the Multi-Head Attention mechanism allows the model to focus on multiple different combinations of emotional elements at the same time, thereby capturing complex emotional compositions.

[0070] Then, threshold analysis is performed on the emotion contribution matrix, and the key emotion triggers are determined using an adaptive threshold algorithm (such as the Otsu method). This step screens out the core elements that play a decisive role in emotion judgment and provides a focus for subsequent emotion interpretation and interaction strategy formulation.

[0071] In order to deeply understand the boundary conditions of emotions, counterexamples are constructed for key emotional triggers. Generative Adversarial Network (GAN) technology is used to generate a series of emotion boundary case sets that are close to the original emotions but slightly different. The generator of GAN is responsible for creating seemingly reasonable emotion variants, while the discriminator is responsible for distinguishing whether these variants are close enough to the original emotion category but have obvious differences. Subsequently, the similarity of the emotion boundary case set is calculated, and the emotion difference space map is constructed using metrics such as cosine similarity. This space map shows the relative relationship and transition area between different emotion categories, which helps to understand the continuity and fuzzy boundaries of emotions. Trajectory prediction is performed on the emotion difference space map, and the long short-term memory network (LSTM) model is used to predict the possible future emotion change path based on the historical emotion state sequence, and generate an emotion development script library. The long-term dependency capture ability of LSTM enables it to simulate complex emotion evolution processes.

[0072] In order to link the development of emotion with the specific environment, the emotion development script library is contextually mapped. The conditional random field (CRF) model is used to associate the predicted emotion sequence with possible environmental factors to generate an emotion-environment association network. CRF can effectively model the contextual dependency in sequence data, making the emotion prediction more in line with the actual situation.

[0073] Based on the emotion-environment association network, interactive questions are generated. A sequence-to-sequence (Seq2Seq) model with attention mechanism is used to generate a series of targeted interactive questions based on the predicted emotional state and environmental factors to form deep explanation question-answer pairs. These question-answer pairs are designed to verify and deepen the understanding of the user's emotional state through interaction.

[0074] Finally, the deep explanation question-answer pairs are feature extracted and clustered through a self-supervised learning algorithm. Specifically, contrastive learning methods such as SimCLR (Simple Framework for Contrastive Learning of Visual Representations) are used to learn the implicit representations of question-answer pairs. Then, the K-means++ algorithm is used to cluster these representations to obtain the final sentiment expression parameters and interaction strategy recommendations.

[0075] For example, suppose there is a sentiment classification dataset with 1,000 samples, each of which has a 7-dimensional sentiment classification vector (corresponding to 7 basic emotions). First, the hierarchical self-attention mechanism interprets each sample into 3 levels: the top layer is 2-dimensional (positive / negative), the middle layer is 7-dimensional (corresponding to 7 emotions), and the bottom layer is 20-dimensional (specific expression). This forms a 1000x29 hierarchical sentiment interpretation matrix.

[0076] NMF decomposes this 1000x29 matrix into two matrices, 1000x50 and 50x29, where the 50x29 matrix represents 50 basic emotion-causing factors. The multi-head attention mechanism (8 heads) then calculates the importance of these 50 factors in 7 emotions and 10 common situations, generating a 50x70 emotion contribution matrix. Through the Otsu method, 15 key emotion triggers are screened out from the 50 factors. GAN then generates 10 boundary cases for each original sample based on these 15 factors, resulting in a 10000x15 emotion boundary case set. The cosine similarity between these 10,000 cases is calculated, a 10000x10000 similarity matrix is ​​constructed, and a two-dimensional emotion difference space map is generated through dimensionality reduction (such as t-SNE).

[0077] The LSTM model (containing 2 layers, 128 units per layer) predicts the emotional changes in the next 5 time steps based on the emotional state of the past 10 time steps, generating a 1000x5x7 emotional development script library. The CRF model then associates these scripts with 20 possible environmental factors to form a 1000x5x27 emotion-environment association network.

[0078] The Seq2Seq model (2 layers of LSTM and 256 units for both encoder and decoder) generates 5 interactive questions for each sample based on the emotion-environment association network, forming a dataset of 5,000 question-answer pairs. Finally, SimCLR obtains a 128-dimensional representation vector for each question-answer pair through contrastive learning, and the K-means++ algorithm clusters these vectors into 20 categories, each of which corresponds to a combination of emotional expression parameters and corresponding interaction strategy suggestions. For example, for a sample classified as "mild sadness", the final output parameters may include: facial expression parameters (brow droop angle 15°, mouth corner droop 5mm), voice parameters (base frequency reduced by 10Hz, speech speed slowed down by 20%), and corresponding interaction strategy suggestions such as "gently ask about recent difficulties, express understanding and support". These parameters and suggestions enable the robot to more accurately identify and respond to the user's emotional state, and achieve more natural and empathetic human-computer interaction.

[0079] In the embodiment of the present application, by collecting and processing multimodal emotional data, including visual, audio and text data, the information source of emotion recognition is expanded, making the recognition result more comprehensive and accurate, and the cross-modal expansion processing further enriches the sample set and enhances the generalization ability of the model. Secondly, the feature extraction technology adopted by the present method, especially the extraction of emotional semantic information and the generation of emotional virtual features, significantly improves the expressiveness and richness of emotional features. Through emotional self-supervised reconstruction and cross-modal feature alignment, it is ensured that the extracted features can accurately capture key emotional information. Furthermore, the present method introduces context-related data mapping, which enables emotion recognition to take into account the specific context environment, greatly improving the accuracy and applicability of recognition. By constructing a contextual knowledge base and generating diversified emotional response patterns, it is possible to better understand and predict emotional expressions in different situations. In addition, the personalized feature analysis link of the present method allows the system to adapt to the unique emotional expressions of different users, and through user-specific emotional expression samples and personalized emotional response patterns, highly personalized emotion recognition is achieved. In the multimodal data fusion and classification stage, the present method adopts modal integrity detection and cross-modal consistency evaluation, which effectively handles the data missing or inconsistency problems that may occur in practical applications. The introduction of dynamic weight allocation and context-sensitive fusion features further improves the flexibility and robustness of the fusion process. Finally, this method uses a self-supervised learning algorithm to perform multi-dimensional analysis, which can not only obtain accurate emotion classification results, but also generate detailed emotion expression parameters and interaction strategy recommendations. This multi-level explanation generation and counterexample construction mechanism greatly enhances the interpretability of the system, allowing users to better understand and trust the robot's emotional judgment. At the same time, the construction of the emotion development script library and the emotion-environment association network provides the ability to predict and adapt to complex emotional changes, improving the efficiency and accuracy of robot emotion recognition based on AIGC.

[0080] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0081] (1) Collect multimodal emotional data through the target robot, where the multimodal emotional data includes visual data, audio data, and text data, and perform emotional semantic segmentation on the visual data to obtain an emotional region feature map;

[0082] (2) Extract dynamic expression sequences from the emotional region feature map to obtain temporal emotional feature data;

[0083] (3) Perform spectral domain feature decomposition on the audio data to obtain acoustic features, and then perform rhythmic pattern analysis on the acoustic features to obtain the emotional rhythmic feature vector;

[0084] (4) Perform sentiment word recognition on text data to obtain a sentiment word embedding matrix, and perform contextual semantic enhancement on the sentiment word embedding matrix to obtain an expanded sentiment text representation;

[0085] (5) Perform cross-modal alignment on the temporal emotional feature data, emotional prosody feature vectors, and extended emotional text representation to obtain the target emotional feature space;

[0086] (6) Generate adversarial networks to augment the target emotional feature space to obtain an extended emotional sample set, and perform emotional intensity gradient processing on the extended emotional sample set to obtain continuously changing emotional samples;

[0087] (7) Perform marginal emotion case synthesis on continuously changing emotion samples to obtain an extended emotion sample set.

[0088] Specifically, the robot uses a high-definition camera to collect visual data, including facial expressions and body postures; a high-sensitivity microphone array to collect audio data; and a natural language processing module to obtain text data. For visual data, a fully convolutional network (FCN) is used for emotional semantic segmentation to segment the image into different emotional regions, such as eyes, mouth, etc., to obtain an emotional region feature map. This feature map is a three-dimensional tensor, in which two dimensions correspond to the spatial position of the image, and the third dimension represents different emotional region categories. Dynamic expression sequences are extracted from the emotional region feature map, and a three-dimensional convolutional neural network (3D-CNN) is used to process continuous feature map sequences to capture the temporal dynamic changes of expressions and obtain temporal emotional feature data. This data is a four-dimensional tensor, in which the first three dimensions correspond to time, width, and height, respectively, and the last dimension represents the feature channel.

[0089] For audio data, we first perform spectral domain feature decomposition and use short-time Fourier transform (STFT) to convert the time domain signal into a time-frequency spectrum. Then, we perform rhythmic pattern analysis on the obtained acoustic features and use long short-term memory network (LSTM) to extract rhythmic features such as pitch, intensity, and length to obtain the emotional rhythmic feature vector. This vector contains the emotional information contained in the audio signal, such as the fluctuation of intonation and the change of speech speed.

[0090] Text data processing first performs sentiment word recognition and uses a pre-trained word embedding model (such as Word2Vec) to convert the text into a sentiment word embedding matrix. Subsequently, this matrix is ​​contextually enhanced and a bidirectional long short-term memory network (Bi-LSTM) is used to capture the long-distance dependencies between words to obtain an expanded sentiment text representation. This representation not only contains the sentiment information of a single word, but also takes into account the context of the entire sentence.

[0091] In order to unify the features of different modalities into a common feature space, the temporal emotional feature data, emotional rhythm feature vectors and extended emotional text representations are cross-modally aligned. This step uses the Cross-modal Transformer with an attention mechanism to learn the associations between features of different modalities through self-attention and cross-attention layers to obtain the target emotional feature space. This space is a high-dimensional vector space, and each dimension represents a certain cross-modal emotional feature. In order to increase the diversity and robustness of the data, the target emotional feature space is augmented by a generative adversarial network (GAN). The generator of GAN is responsible for creating new emotional feature samples, while the discriminator is responsible for distinguishing between real samples and generated samples. The quality of generated samples is continuously improved through adversarial training to obtain an extended emotional sample set.

[0092] The extended emotion sample set is processed into an emotion intensity gradient, and interpolation technology is used to create a smooth transition between different emotion intensities to obtain continuously changing emotion samples. This step ensures that the emotion samples cover the full range of emotion intensities from weak to strong.

[0093] Finally, the continuously changing emotion samples are synthesized into edge emotion cases. A Gaussian mixture model (GMM) is used to generate new samples in the boundary areas of different emotion categories. These samples represent fuzzy emotion states that are difficult to clearly classify, thus obtaining the final extended emotion sample set.

[0094] For example: Suppose the robot is interacting with a user and collects 5 seconds of multimodal data. The visual data contains a 150-frame 1080p resolution image sequence; the audio data is a 44.1kHz sampling rate speech signal; the text data is a transcribed sentence "This result makes me a little disappointed, but it is not completely hopeless." First, FCN performs sentiment semantic segmentation on each frame of the image to obtain 150 224x224x10 feature maps, where 10 represents 10 key emotional areas. 3D-CNN then processes these 150 frames and outputs a 16x28x28x64 temporal emotional feature data that captures the changes in expression within 5 seconds.

[0095] For audio, STFT converts 5 seconds of speech into a 257x216 time-frequency spectrum (assuming the window size is 512 and the step size is 128). LSTM analyzes this spectrum and extracts a 128-dimensional emotional prosody feature vector that contains information such as intonation and speech rate.

[0096] The text data is first converted into a 15x300 word embedding matrix through Word2Vec (assuming that the sentence contains 15 words, each word is represented by a 300-dimensional vector). Bi-LSTM processes this matrix and outputs a 512-dimensional expanded sentiment text representation that encodes the sentiment semantics of the entire sentence.

[0097] The cross-modal transformer aligns these three features into a 1024-dimensional target emotion feature space. GAN then generates 1000 new samples based on this feature space, expanding the original dataset. The emotion intensity gradient processing subdivides these 1000 samples into 10 intensity levels on each emotion dimension, obtaining 10,000 continuously changing emotion samples.

[0098] Finally, GMM synthesizes 5,000 edge emotion cases in the boundary areas of different emotion categories, and finally obtains an extended emotion sample set containing 15,000 samples. This sample set not only has a greatly increased number, but also covers a variety of emotional states from clear to vague.

[0099] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0100] (1) Perform multimodal emotional feature mapping on the extended emotional sample set to obtain the initial emotional high-dimensional feature space, and extract emotional semantic information from the initial emotional high-dimensional feature space to obtain the emotional semantic enhanced feature matrix;

[0101] (2) Generate emotional virtual features for the emotional semantic enhancement feature matrix to obtain an emotional supplementary feature set, and perform emotional feature fusion on the emotional supplementary feature set to obtain a fused emotional feature representation;

[0102] (3) Perform emotional self-supervision reconstruction on the fused emotional feature representation to obtain emotional reconstruction feature data, and perform data comparison and analysis on the emotional reconstruction feature data and the extended emotional sample set to obtain the key emotional feature set;

[0103] (4) Perform cross-modal emotional feature alignment on the key emotional feature set to obtain emotional consistency feature mapping data;

[0104] (5) Dynamically adjust the emotional dimension of the emotional consistency feature mapping data to obtain an adaptive emotional feature vector;

[0105] (6) Compress the emotional information of the adaptive emotional feature vector to obtain efficient emotional feature encoding;

[0106] (7) The emotional feature is enhanced by using a contrastive learning algorithm to encode the efficient emotional feature and obtain the emotional feature vector.

[0107] Specifically, a deep neural network is used to extract features from each modality data: ResNet-50 is used to extract 2048-dimensional features from visual data, 1D-CNN is used to extract 256-dimensional features from audio data, and BERT model is used to extract 768-dimensional features from text data. These features are concatenated to form an initial 3072-dimensional high-dimensional emotional feature space, and each sample is mapped to a point in this space. The initial emotional high-dimensional feature space is used to extract emotional semantic information, and a multi-head self-attention network (Multi-Head Self-Attention) with an attention mechanism is used. The network contains 8 attention heads, each of which outputs 384-dimensional features, and finally obtains a 3072x3072 emotional semantic enhancement feature matrix. This matrix not only retains the original feature information, but also captures the semantic associations between different feature dimensions. Emotional virtual features are generated for the emotional semantic enhancement feature matrix. This step uses a variational autoencoder (VAE), whose encoder compresses the 3072-dimensional input into a 512-dimensional latent vector, and the decoder generates new 3072-dimensional features from this latent space. By sampling and interpolating in the latent space, VAE generates 10,000 new virtual features to form a sentiment-supplemented feature set.

[0108] The graph attention network (GAT) is used to perform sentiment feature fusion on the sentiment supplementary feature set. GAT regards each feature as a node in the graph and fuses related feature information through a message passing mechanism. Specifically, a three-layer GAT is used, each layer contains 64 attention heads, and finally outputs a 1024-dimensional fused sentiment feature representation.

[0109] In order to ensure the effectiveness of the features, the fused emotional feature representation is reconstructed by emotional self-supervision. A deep autoencoder is used, whose encoder and decoder each contain 3 fully connected layers, and the hidden layer sizes are 512, 256, and 512 respectively. The autoencoder learns the intrinsic structure of the features by minimizing the reconstruction error and outputs 1024-dimensional emotional reconstruction feature data.

[0110] A comparative analysis of the sentiment reconstruction feature data and the extended sentiment sample set was performed. The t-SNE algorithm was used to reduce the dimensions of the two sets of data to 2D space for visualization. The k nearest neighbors (k=10) of each sample in the original space and the reconstructed space were calculated, and samples with high neighbor consistency were retained to form a key sentiment feature set. Cross-modal sentiment feature alignment was performed on the key sentiment feature set, using canonical correlation analysis (CCA). CCA looks for the maximum correlation between features of different modalities, and projects the visual, audio, and text features into a common 512-dimensional space to obtain sentiment consistency feature mapping data.

[0111] Next, the emotional consistency feature map data is dynamically adjusted for emotional dimension. The principal component analysis (PCA) method is used to dynamically select the number of principal components based on the cumulative explained variance ratio. For example, if the threshold is set to 95%, PCA will select the minimum number of principal components that can explain 95% of the variance and obtain an adaptive emotional feature vector.

[0112] In order to further improve efficiency, the adaptive emotional feature vector is compressed. Using the nonlinear dimensionality reduction technology t-SNE, the features are compressed to 128 dimensions to obtain efficient emotional feature encoding. This encoding not only retains the key information of high-dimensional features, but also greatly reduces the computational complexity.

[0113] Finally, the contrastive learning algorithm is used to enhance the emotional features of efficient emotional feature coding. The SimCLR framework is used to generate different views of the same sample using data enhancement techniques (such as random cropping and color jitter). By maximizing the mutual information between different views of the same sample and minimizing the mutual information between different samples, a more discriminative feature representation is learned. This step outputs the final 256-dimensional emotional feature vector.

[0114] For example: Suppose there is an extended emotion sample set containing 10,000 multimodal emotion samples. Each sample contains a 224x224 face image, a 3-second audio, and a sentence of text. First, ResNet-50 extracts 2048-dimensional features from the image, 1D-CNN extracts 256-dimensional features from the audio, and BERT extracts 768-dimensional features from the text. These features are concatenated into 3072-dimensional vectors to form an initial high-dimensional feature matrix of 10000x3072. The multi-head self-attention network processes this matrix and outputs a 10000x3072 emotion semantic enhanced feature matrix. VAE then generates 10,000 new virtual features based on this matrix to obtain a 20000x3072 emotion supplement feature set. GAT fuses these features and outputs a 20000x1024 fused emotion feature representation.

[0115] The autoencoder reconstructs the fused features to obtain 20000x1024 sentiment reconstruction feature data. Through t-SNE visualization and k-nearest neighbor analysis, 15,000 key samples are screened to form a 15,000x1024 key sentiment feature set. CCA aligns this feature set to a 512-dimensional space to obtain 15,000x512 sentiment consistency feature mapping data.

[0116] PCA dynamically selects principal components. Assuming that 200 principal components are selected, a 15000x200 adaptive sentiment feature vector is obtained. t-SNE further compresses it to 128 dimensions, resulting in a 15000x128 efficient sentiment feature encoding. Finally, the SimCLR framework enhances the features to 256 dimensions through contrastive learning, resulting in a final 15000x256 sentiment feature vector. This vector not only has a greatly reduced dimension, but also contains rich cross-modal sentiment information, providing high-quality feature representation for subsequent sentiment recognition tasks.

[0117] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0118] (1) Perform semantic decomposition on the emotional feature vector to obtain multi-dimensional emotional semantic data, and perform context association analysis on the multi-dimensional emotional semantic data to obtain preliminary context-emotion correspondence;

[0119] (2) Map the preliminary context-emotion correspondence into a knowledge graph to obtain a structured context-emotion network;

[0120] (3) Dynamically expand the structured situational emotion network to obtain a situational knowledge base, and generate an emotion response spectrum for the situational knowledge base to obtain a diversified emotion response model;

[0121] (4) Perform time-series prediction on the diversified emotional response patterns to obtain an emotional change trend graph, and then perform scenario derivation on the emotional change trend graph to obtain a multi-path emotional development matrix;

[0122] (5) Calculate the similarity of the multi-path emotion development matrix to obtain the emotion-context association pattern set, and construct comparative samples of the emotion-context association pattern set to obtain fine-grained context-emotion mapping data;

[0123] (6) The fine-grained situation-emotion mapping data and the emotion feature vector are fused through the graph attention network algorithm to obtain context-related emotion representation data.

[0124] Specifically, the 256-dimensional emotional feature vector is semantically decomposed, and the feature vector is decomposed into basic emotional semantic components using the non-negative matrix factorization (NMF) algorithm. NMF decomposes the feature matrix V into the product of two non-negative matrices W and H, where W represents the basic emotional semantics and H represents the weights of these semantics in the original features. Assuming that 20 basic emotional semantics are selected, a 256x20 W matrix and a 20xN H matrix are obtained, where N is the number of samples. These 20-dimensional vectors constitute multi-dimensional emotional semantic data. The multi-dimensional emotional semantic data is context-related analyzed using the conditional random field (CRF) model. CRF considers the interdependence between emotional state and environmental factors, and the input is 20-dimensional emotional semantics and predefined context features (such as location, time, social occasion, etc.), and the output is the emotion-context correspondence. For example, for the context of "workplace", CRF may output an emotional distribution such as {"focus": 0.6, "pressure": 0.3, "excitement": 0.1}. The preliminary context-emotion correspondence is mapped to the knowledge graph using the TransE algorithm. TransE represents sentiment concepts, contextual factors, and their relationships as low-dimensional vectors, and learns these representations by minimizing the energy function of triples (head entity, relationship, tail entity). Assuming that 100-dimensional vectors are used to represent entities and relationships, for each triple (h, r, t), the optimization goal is to make h + r ≈ t. This step transforms discrete correspondences into structured contextual sentiment networks.

[0125] The structured contextual emotion network is dynamically expanded using incremental learning algorithms such as Online TransE. When new emotion-context relationships are observed, Online TransE updates the vector representations of related entities and relationships through stochastic gradient descent, thereby continuously expanding the contextual knowledge base. Based on the expanded contextual knowledge base, a variational autoencoder (VAE) is used to generate the emotional response spectrum. The encoder of the VAE maps the 100-dimensional context vector to a low-dimensional latent space (such as 32 dimensions), and the decoder generates diverse emotional responses from this latent space. By sampling in the latent space, the VAE can generate diversified emotional response patterns associated with a given context.

[0126] The long short-term memory network (LSTM) is used to perform time series prediction on the diversified emotional response patterns. LSTM receives a series of emotional states as input and predicts future emotional changes. For example, the emotional states of the past 10 time steps are used to predict the changes in the next 5 time steps to obtain an emotional change trend graph. Then, the emotional change trend graph is scenario-derived using the Monte Carlo Tree Search (MCTS) algorithm. MCTS simulates multiple possible situational changes. Each simulation selects an action based on the current state, updates the state, and evaluates the results. Through a large number of simulations, multiple possible emotional development paths are generated to form a multi-path emotional development matrix.

[0127] The dynamic time warping (DTW) algorithm is used to calculate the similarity of the multi-path emotion development matrix. DTW calculates the similarity between time series of different lengths, identifies representative emotion-context evolution patterns, and forms an emotion-context association pattern set.

[0128] In order to refine these patterns, a contrast sample construction is performed on the emotion-context association pattern set using a generative adversarial network (GAN). The generator of GAN creates samples that are similar to but slightly different from the original pattern, while the discriminator distinguishes between real samples and generated samples. Through adversarial training, a series of fine-grained context-emotion mapping data are generated. Finally, the fine-grained context-emotion mapping data and the original emotion feature vector are fused through the graph attention network (GAT) algorithm. GAT regards each sample as a node in the graph and fuses the information of related nodes through a message passing mechanism. Specifically, a multi-layer GAT is used, each layer contains multiple attention heads, and the importance weights between different nodes are dynamically calculated, and finally the emotion representation that integrates the context information is output.

[0129] For example: suppose there is a set of sentiment feature vectors containing 10,000 samples, each of which is 256-dimensional. First, NMF decomposes this 256x10000 matrix into a 256x20 W matrix and a 20x10000 H matrix to obtain a 20-dimensional basic sentiment semantic representation. CRF then analyzes the relationship between these 20-dimensional semantics and 10 predefined contexts, and outputs a 20x10 sentiment-context probability matrix. TransE converts this matrix into a knowledge graph containing 30 nodes (20 sentiment nodes and 10 context nodes) and 200 relationship edges, each of which is represented by a 100-dimensional vector.

[0130] Through Online TransE, the system observed 1,000 new emotion-context relationships during operation, expanding the knowledge graph to 50 nodes and 500 relationship edges. Based on this expanded knowledge base, VAE generates 5,000 diversified emotion response patterns, each of which is a 20-dimensional vector representing the distribution of emotions in a specific context. LSTM receives these 5,000 patterns as input, predicts changes in the next 5 time steps, and generates a 5,000x5x20 emotion change trend graph. Based on this trend graph, MCTS then simulates 1,000 possible situation change sequences, each containing 10 time steps, to obtain a 1,000x10x20 multi-path emotion development matrix.

[0131] DTW calculates the similarity between these 1,000 paths and clusters them to obtain 100 representative emotion-context association patterns. Based on these 100 patterns, GAN generates 10,000 fine-grained context-emotion mapping samples, each of which is a 120-dimensional vector (20-dimensional emotion + 100-dimensional context). Finally, GAT fuses these 10,000 fine-grained samples with the original 10,000 256-dimensional emotion feature vectors. GAT contains 3 layers, each with 8 attention heads, and ultimately outputs 10,000 512-dimensional vectors, which are context-related emotion representation data. These representations not only contain the original emotional information, but also incorporate rich context semantics.

[0132] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0133] (1) Perform user feature extraction on context-related emotion representation data to obtain an initial user emotion feature set, and perform sample synthesis on the initial user emotion feature set to obtain extended user emotion expression data;

[0134] (2) Perform expression-speech-text joint analysis on the extended user emotion expression data to obtain a multimodal personalized feature matrix, and perform style conversion on the multimodal personalized feature matrix to obtain user-specific emotion expression samples;

[0135] (3) Perform contextual relevance analysis on user-specific emotion expression samples to obtain personalized emotion response patterns, and perform sequence prediction on personalized emotion response patterns to obtain user emotion response trend graphs;

[0136] (4) Perform semantic mapping on the user's emotional response trend graph to obtain a personalized emotional vocabulary, and perform frequency-context analysis on the personalized emotional vocabulary to obtain a user emotional expression habit library;

[0137] (5) Compare the differences of the user's emotional expression habit library to obtain a personalized emotional feature deviation matrix;

[0138] (6) The personalized emotion feature deviation matrix and context-related emotion representation data are integrated through the transfer learning algorithm to obtain the mapped user emotion data.

[0139] Specifically, the Siamese network is used to compare and learn the user's historical data and current emotion representation to extract the user's unique emotion expression pattern. The Siamese network contains two identical sub-networks, each of which consists of 3 fully connected layers, with a 512-dimensional emotion representation as input and a 256-dimensional user feature vector as output. By minimizing the distance between different samples of the same user and maximizing the distance between samples of different users, the network learns a representation that can distinguish the personalized features of users, thereby obtaining the initial user emotion feature set. Samples of the initial user emotion feature set are synthesized, and a variational autoencoder (VAE) is used to generate more user-specific emotion samples. The encoder of the VAE compresses the 256-dimensional user features into a 64-dimensional latent vector, and the decoder generates new 256-dimensional features from this latent space. By sampling and interpolating in the latent space, the VAE generates a large number of new user-specific emotion samples to form extended user emotion expression data.

[0140] Perform expression-speech-text joint analysis on the extended user emotion expression data. A multimodal attention mechanism is used to project the features of different modalities into a common semantic space. Specifically, three independent attention modules are used to process expression, speech and text features respectively, and then the information of these three modalities is fused through a multi-head attention mechanism (8 heads) to obtain a 512-dimensional multimodal personalized feature matrix. In order to capture the user's unique emotion expression style, style transfer is performed on the multimodal personalized feature matrix. A cycle-consistent adversarial network (CycleGAN) is used to map the user's emotion expression features to the general emotion expression pattern. CycleGAN contains two generators and two discriminators. It learns the user's specific emotion expression style through adversarial training and generates a series of user-specific emotion expression samples.

[0141] Perform situational relevance analysis on user-specific emotion expression samples. Use the conditional random field (CRF) model to associate the user's emotion expression with different situational factors (such as social occasions, work environment, etc.). The input of CRF is user-specific emotion expression samples and predefined situational features, and the output is the emotion distribution probability in different situations, so as to obtain a personalized emotion response pattern. In order to predict the user's emotion change trend, the personalized emotion response pattern is sequence predicted. The long short-term memory network (LSTM) is used to consider the time series information of the user's historical emotion state. LSTM contains two layers, each with 128 hidden units. The input is the emotion state of the past 10 time steps, and the emotion change in the next 5 time steps is predicted to generate a user emotion response trend chart.

[0142] Semantically map the user's emotional response trend graph. Use word embedding technologies such as Word2Vec to convert the predicted emotional state into corresponding language descriptions. Specifically, train a Word2Vec model to map each emotional state to a 300-dimensional word vector space to form a personalized emotional vocabulary.

[0143] Perform frequency-context analysis on the personalized sentiment vocabulary. Use the TF-IDF (term frequency-inverse document frequency) algorithm to calculate the importance weight of each sentiment word in different contexts. TF-IDF takes into account the frequency of occurrence of words in specific contexts and the prevalence in all contexts, so as to identify the keywords that best represent the user's emotional expression in a specific context and obtain the user's emotional expression habit library. In order to highlight the user's unique emotional expression characteristics, the user's emotional expression habit library is compared for differences. The cosine similarity calculation is used to compare the user's emotional expression pattern with the general pattern. Calculate the cosine similarity between each sentiment word vector and the general sentiment word vector, select the words with lower similarity as the user's unique expression, and generate a personalized sentiment feature deviation matrix.

[0144] Finally, the personalized emotion feature bias matrix and context-dependent emotion representation data are integrated through transfer learning algorithm. A domain adversarial neural network (DANN) is used to adapt the general emotion feature domain to the user-specific emotion feature domain. DANN consists of a feature extractor, a sentiment classifier, and a domain discriminator. The feature extractor learns domain-invariant feature representations, the sentiment classifier is trained on the source domain (generic emotion), and the domain discriminator ensures that the learned features are indistinguishable between the source domain and the target domain (user-specific emotion), thereby achieving knowledge transfer and obtaining the final mapped user emotion data.

[0145] For example, suppose there is a user whose 1,000 emotion expression data in the past 30 days are collected by the system. First, the Siamese network compares these historical data with the current 512-dimensional context-related emotion representation and extracts a 256-dimensional initial user emotion feature vector. VAE then generates 5,000 new user-specific emotion samples based on this vector, each of which is a 256-dimensional vector.

[0146] The multimodal attention mechanism maps the expression, speech, and text features of these 5,000 samples into a common 512-dimensional feature space, forming a 5000x512 multimodal personalized feature matrix. CycleGAN then performs style conversion on each sample in this matrix with the general emotion pattern to generate 10,000 user-specific emotion expression samples.

[0147] The CRF model analyzes the performance of these 10,000 samples in 10 different situations and generates a 10,000x10 personalized emotional response pattern matrix. Based on this matrix, the LSTM predicts the user's emotional changes in these 10 situations in the next 24 hours and generates a 10x24 user emotional response trend chart.

[0148] Word2Vec maps this trend graph to a vocabulary of 500 sentiment words, each represented by a 300-dimensional vector. The TF-IDF algorithm calculates the weights of these 500 words in 10 contexts to form a 500x10 matrix of user sentiment expression habits. By calculating the cosine similarity between this matrix and the general expression pattern, the 100 most personalized features are identified to form a 100x300 personalized sentiment feature deviation matrix.

[0149] Finally, DANN integrates this bias matrix with the original 512-dimensional context-dependent sentiment representation. DANN's feature extractor contains 3 fully connected layers, mapping the 812-dimensional input (512+300) to 256 dimensions. The sentiment classifier and domain discriminator each contain 2 fully connected layers. After 1000 rounds of training, DANN outputs a 1024-dimensional vector of mapped user sentiment data. This vector not only contains general sentiment information, but also encodes the user's unique sentiment expression characteristics.

[0150] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0151] (1) Perform modal integrity detection on the mapped user emotion data to obtain the modal missing label matrix, and perform data completion processing on the mapped user emotion data based on the modal missing label matrix to obtain the target multimodal emotion dataset;

[0152] (2) Perform cross-modal consistency evaluation on the target multimodal sentiment dataset to obtain a modal consistency score vector, and perform threshold screening on the modal consistency score vector to obtain a high-confidence multimodal data subset;

[0153] (3) Dynamically assign weights to high-confidence multimodal data subsets to obtain the modal fusion weight matrix;

[0154] (4) Perform weighted fusion on the modal fusion weight matrix and the high-confidence multimodal data subset to obtain a preliminary fusion feature vector, and perform context-constraint processing on the preliminary fusion feature vector to obtain a context-sensitive fusion feature;

[0155] (5) Perform counterfactual analysis on the context-sensitive fusion features to obtain a modal importance assessment report, and generate decision rules for the modal importance assessment report to obtain a dynamic decision tree structure;

[0156] (6) The dynamic decision tree structure is mapped into a multi-dimensional sentiment space through an integrated learning algorithm to obtain sentiment classification data.

[0157] Specifically, a density-based local outlier detection (LOF) algorithm is used to evaluate the integrity of each modality of each sample. LOF calculates the local density of each data point and compares it with the density of neighboring points to identify abnormal or missing modalities. Specifically, for each 1024-dimensional mapping user emotion data vector, it is divided into three sub-vectors: vision, audio, and text, and the LOF score is calculated respectively. If the LOF score of a modality is significantly higher than the threshold (such as set to 2.0), it is marked as missing. This process generates an N×3 modality missing labeling matrix, where N is the number of samples, 3 represents the three modalities, and the matrix elements are 0 or 1, 1 indicates that the modality is missing. According to the modality missing labeling matrix, the mapping user emotion data is processed for data completion. Here, the multimodal generative adversarial network (MMGAN) technology is used to generate synthetic data of missing modalities based on existing modal data. MMGAN contains three generators and one discriminator, each generator is responsible for the generation of one modality. The input of the generator is the features and random noise of the non-missing modality, and the output is the synthetic features of the missing modality. The discriminator learns to distinguish between real data and generated data. Through adversarial training, MMGAN gradually improves the quality of generated data and finally obtains a complete target multimodal emotion dataset.

[0158] The cross-modal consistency of the target multimodal sentiment dataset is evaluated. The canonical correlation analysis (CCA) method is used to calculate the correlation between features of different modalities. CCA seeks the projection direction that maximizes the correlation of features of different modalities and quantifies the degree of consistency between different modalities. For each sample, the canonical correlation coefficients between the three pairs of modalities of vision-audio, vision-text, and audio-text are calculated, and the average value is taken as the modal consistency score of the sample, thereby obtaining an N-dimensional modal consistency score vector. Then, the modal consistency score vector is threshold screened. The Otsu's method is used to automatically determine the optimal segmentation threshold. The Otsu's method finds the optimal threshold by maximizing the inter-class variance and divides the samples into two categories: high consistency and low consistency. The samples above the threshold are retained to form a high-confidence multimodal data subset.

[0159] Dynamically assign weights to high-confidence multimodal data subsets using attention mechanisms such as Multi-Head Attention. Specifically, a multi-head attention layer with 8 attention heads is constructed, with the input being the features of the three modalities and the output being the weights of each modality. This process takes into account the importance of different modalities in the current emotion recognition task, dynamically calculates the weight of each modality, and forms a modality fusion weight matrix. Subsequently, the modality fusion weight matrix and the high-confidence multimodal data subset are weighted fused. Tensor fusion methods such as Tucker decomposition are used to integrate the features of different modalities and their weights into a unified multimodal representation. Tucker decomposition can capture the high-order correlations between different modalities, generate a richer and more compact fusion representation, and obtain a preliminary fused feature vector.

[0160] Next, the preliminary fused feature vector is subjected to contextual constraints. The fused features are combined with the previously obtained contextual information using the conditional random field (CRF) model to consider the contextual dependency of emotional expression. The input of CRF is the preliminary fused feature vector and the predefined contextual features, and the output is the emotional representation that takes contextual factors into account, namely, the context-sensitive fused features.

[0161] In order to evaluate the impact of different modalities on sentiment judgment, counterfactual analysis is performed on context-sensitive fusion features. The contribution of each modality to the final sentiment judgment is calculated using SHAP (SHapley Additive exPlanations) values. SHAP explains model predictions by calculating the marginal contribution of features and generates a modality importance assessment report.

[0162] Based on the modality importance assessment report, decision rules are generated. Using the C4.5 decision tree algorithm, according to the importance and feature performance of different modalities, a decision structure that can adapt to different emotion categories and modality combinations is constructed to obtain a dynamic decision tree structure.

[0163] Finally, the dynamic decision tree structure is mapped to a multi-dimensional sentiment space through an ensemble learning algorithm. Specifically, the Random Forest method is used to generate multiple decision trees and map their outputs to a multi-dimensional sentiment space. The Random Forest improves the generalization and robustness of the model through bagging and random feature selection, and finally obtains sentiment classification data.

[0164] For example: suppose there is a user sentiment mapping dataset containing 10,000 samples, each of which is a 1024-dimensional vector. First, the LOF algorithm detects modality completeness and generates a 10,000×3 modality missing labeling matrix. 500 samples are found to be modally missing, and MMGAN completes these samples. CCA then calculates the correlation between modalities and obtains a 10,000-dimensional consistency score vector. Otsu's method determines the threshold as 0.75 and screens out 8,000 highly consistent samples. The multi-head self-attention mechanism calculates the weights of the three modalities in these 8,000 samples and generates an 8,000×3 modality fusion weight matrix. Tucker decomposition compresses the weighted features into a 512-dimensional preliminary fusion feature vector. CRF considers 10 different contextual factors and expands these 512-dimensional features into 600-dimensional context-sensitive fusion features. SHAP analysis calculates the importance of the three modalities in the seven basic emotion categories and generates a 21-dimensional modality importance vector. The C4.5 algorithm builds a decision tree with a maximum depth of 10 based on this vector. Finally, the random forest algorithm generates 100 such decision trees and maps their outputs to a 7-dimensional emotional space (corresponding to 7 basic emotions), obtaining the final 8000×7 emotional classification data matrix. Each row in this matrix represents the probability distribution of a sample on the 7 basic emotions, such as [0.1, 0.6, 0.1, 0.05, 0.05, 0.05,0.05], which may mean that the probability of a sample being classified as "happy" is 60%, "neutral" is 10%, and the other emotions are 5-10% each.

[0165] In a specific embodiment, the process of executing step S106 may specifically include the following steps:

[0166] (1) Generate multi-granularity interpretations of sentiment classification data to obtain a hierarchical sentiment interpretation structure, and perform semantic decomposition on the hierarchical sentiment interpretation structure to obtain a set of sentiment cause elements;

[0167] (2) Calculate the weights of the emotion-causing factor set to obtain the emotion contribution matrix, and perform threshold analysis on the emotion contribution matrix to obtain the key emotion triggering factors;

[0168] (3) Construct counterexamples for key emotional triggers to obtain an emotional boundary case set, and calculate the similarity of the emotional boundary case set to obtain an emotional difference space map;

[0169] (4) Predict the trajectory of the emotion difference space map to obtain the emotion development script library, and then perform context mapping on the emotion development script library to obtain the emotion-environment association network;

[0170] (5) Generate interactive questions for the emotion-environment association network to obtain deeply explained question-answer pairs;

[0171] (6) Through a self-supervised learning algorithm, feature extraction and clustering of deep explanation question and answer pairs are performed to obtain emotional expression parameters and interaction strategy recommendations.

[0172] Specifically, a hierarchical self-attention mechanism is used to interpret the sentiment classification data at multiple levels. The mechanism consists of three layers of self-attention networks, each of which focuses on sentiment features of different granularity. Specifically, the first layer focuses on basic sentiment categories (such as happiness and sadness), the second layer focuses on sentiment intensity and subtle changes, and the third layer focuses on specific sentiment expressions (such as facial expressions and voice features). This process expands the 7-dimensional sentiment classification vector into a three-layer structure, for example: [basic sentiment (7 dimensions), sentiment intensity (5 dimensions), specific expression (20 dimensions)], forming a hierarchical sentiment interpretation structure. The hierarchical sentiment interpretation structure is semantically decomposed, and the non-negative matrix factorization (NMF) algorithm is used to decompose the complex sentiment interpretation into basic sentiment causal elements. NMF decomposes the 32-dimensional (7+5+20) sentiment interpretation vector into the product of two non-negative matrices: a 32×K basic matrix W and a K×N coefficient matrix H, where K is the preset number of sentiment causal elements (such as 50) and N is the number of samples. In this way, 50 basic emotion-causing factors are obtained, each of which is a 32-dimensional vector, representing a basic emotion inducement or expression.

[0173] Then, the set of emotion-causing factors is weighted, and the Multi-Head Attention mechanism is used to calculate the importance of each factor in different emotion categories and contexts. Specifically, 8 attention heads are used, each of which outputs 64-dimensional features, and finally a 50×512 emotion contribution matrix is ​​obtained. Each row in this matrix represents an emotion-causing factor, and each column represents the importance weight of the factor in different emotion categories and context combinations. Threshold analysis is performed on the emotion contribution matrix, and the OTSU algorithm is used to automatically determine the optimal threshold. The OTSU method finds the optimal threshold by maximizing the inter-class variance, and divides the 50 emotion-causing factors into two categories: high importance and low importance. The factors above the threshold are retained to obtain the key emotion triggers, and usually 15-20 of the most important factors are retained.

[0174] In order to deeply understand the boundary conditions of emotions, counterexamples are constructed for key emotional triggers. Generative Adversarial Network (GAN) technology is used to generate a series of emotional boundary case sets that are similar to the original emotions but slightly different. The generator of GAN receives key emotional triggers as input and generates a new 32-dimensional emotional vector, while the discriminator learns to distinguish between real emotional samples and generated samples. Through adversarial training, 10 boundary cases are generated for each key trigger, and an emotional boundary case set containing 150-200 samples is obtained.

[0175] The similarity of the emotion boundary case set is calculated, and the t-SNE (t-distributed StochasticNeighbor Embedding) algorithm is used to map the high-dimensional emotion vector to the 2D plane to construct the emotion difference space map. t-SNE maintains the relative distance relationship between samples in the high-dimensional space, and intuitively displays the relative position and transition area between different emotion states on the 2D plane. The trajectory of the emotion difference space map is predicted using the long short-term memory network (LSTM) model. LSTM contains two layers, each with 128 hidden units. The input is a 2D coordinate sequence of the past 10 time steps, and the coordinate changes in the next 5 time steps are predicted. This process simulates the evolution trajectory of the emotion state over time and generates an emotion development script library. Each script is a 15-step 2D coordinate sequence, representing a possible emotion development path.

[0176] In order to link emotion development with the specific environment, the emotion development script library is contextually mapped. The predicted emotion sequence is associated with possible environmental factors using the conditional random field (CRF) model. The input of CRF is a 15-step emotion coordinate sequence and 10 predefined environmental factors, and the output is the most likely environment-emotion combination at each time step, thus obtaining an emotion-environment association network.

[0177] Based on the emotion-environment association network, interactive questions are generated. A sequence-to-sequence (Seq2Seq) model with attention mechanism is used to generate a series of targeted interactive questions based on the predicted emotional state and environmental factors. The encoder and decoder of the Seq2Seq model each contain 2 layers of LSTM with 256 hidden units. The encoder input is the emotion-environment sequence, and the decoder generates the corresponding questions. This process produces deeply explained question-answer pairs, each of which contains an emotion-environment description and the corresponding interactive question.

[0178] Finally, the self-supervised learning algorithm is used to extract features and cluster the deep explanation question-answer pairs. The BERT (Bidirectional Encoder Representations from Transformers) model is used to encode the question-answer pairs to obtain a 768-dimensional feature vector. Then, the K-means++ algorithm is used to cluster these feature vectors, and the number of clusters is set to 20. Each cluster center represents a typical emotional expression pattern and the corresponding interaction strategy, thereby obtaining emotional expression parameters and interaction strategy recommendations.

[0179] For example: suppose there is a sentiment classification dataset containing 10,000 samples, each of which has a 7-dimensional sentiment classification vector. First, the hierarchical self-attention mechanism expands each sample into a 32-dimensional hierarchical explanation structure. NMFthen decomposes this 10,000×32 matrix into two matrices of 32×50 and 50×10,000, and obtains 50 basic sentiment causal factors. The multi-head attention mechanism calculates the importance of these 50 factors in 7 emotions and 10 common situations, and generates a 50×70 sentiment contribution matrix. The OTSU method analyzes this matrix, determines the threshold value as 0.6, and screens out 18 key sentiment triggers. Based on these 18 factors, GAN generates 180 boundary cases to form a set of sentiment boundary cases.

[0180] t-SNE maps these 180 32-dimensional vectors to a 2D plane to generate a spatial map of emotional differences. Based on this spatial map, the LSTM model predicts 5,000 possible emotional development trajectories, each containing 15 time steps. The CRF model associates these trajectories with 10 environmental factors to generate a 5000×15×10 emotion-environment association network.

[0181] The Seq2Seq model generates 25,000 question-answer pairs based on this network. BERT encodes these question-answer pairs and obtains 25,000 768-dimensional vectors. The K-means++ algorithm clusters these vectors into 20 groups, each representing an emotional expression pattern and interaction strategy. For example, one group may represent "mild anxiety under work pressure", and the corresponding interaction strategy recommendation is "ask about the specific source of stress and provide relaxation techniques." In this way, the system can select the most appropriate interaction method based on the recognized emotional state to achieve personalized and context-aware emotional interaction.

[0182] The above describes the robot emotion recognition method based on AIGC in the embodiment of the present application. The following describes the robot emotion recognition system based on AIGC in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of the robot emotion recognition system based on AIGC includes:

[0183] The acquisition module 201 is used to acquire multimodal emotion data through a target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and perform cross-modal expansion processing on the multimodal emotion data to obtain an extended emotion sample set;

[0184] An extraction module 202 is used to extract features from the extended emotion sample set to obtain an emotion feature vector;

[0185] A mapping module 203 is used to perform context-related data mapping on the emotion feature vector to obtain context-related emotion representation data;

[0186] An analysis module 204 is used to perform personalized feature analysis on the context-related emotion representation data to obtain mapped user emotion data;

[0187] A fusion module 205 is used to perform multimodal data fusion and classification on the mapped user emotion data to obtain emotion classification data;

[0188] The analysis module 206 is used to perform multi-dimensional analysis on the emotion classification data through a self-supervised learning algorithm to obtain emotion expression parameters and interaction strategy suggestions.

[0189] Through the collaboration of the above components, by collecting and processing multimodal emotional data, including visual, audio and text data, the information source of emotion recognition is expanded, making the recognition results more comprehensive and accurate. The cross-modal expansion processing further enriches the sample set and enhances the generalization ability of the model. Secondly, the feature extraction technology adopted by this method, especially the extraction of emotional semantic information and the generation of emotional virtual features, significantly improves the expressiveness and richness of emotional features. Through emotional self-supervised reconstruction and cross-modal feature alignment, it is ensured that the extracted features can accurately capture key emotional information. Furthermore, this method introduces context-related data mapping, which enables emotion recognition to take into account the specific context environment, greatly improving the accuracy and applicability of recognition. By building a contextual knowledge base and generating diversified emotional response patterns, it is possible to better understand and predict emotional expressions in different situations. In addition, the personalized feature analysis link of this method allows the system to adapt to the unique emotional expressions of different users, and through user-specific emotional expression samples and personalized emotional response patterns, highly personalized emotion recognition is achieved. In the multimodal data fusion and classification stage, this method adopts modal integrity detection and cross-modal consistency evaluation to effectively deal with data missing or inconsistency problems that may occur in practical applications. The introduction of dynamic weight allocation and context-sensitive fusion features further improves the flexibility and robustness of the fusion process. Finally, this method uses a self-supervised learning algorithm to perform multi-dimensional analysis, which can not only obtain accurate emotion classification results, but also generate detailed emotion expression parameters and interaction strategy recommendations. This multi-level explanation generation and counterexample construction mechanism greatly enhances the interpretability of the system, allowing users to better understand and trust the robot's emotional judgment. At the same time, the construction of the emotion development script library and the emotion-environment association network provides the ability to predict and adapt to complex emotional changes, improving the efficiency and accuracy of robot emotion recognition based on AIGC.

[0190] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the AIGC-based robot emotion recognition method.

[0191] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0192] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot emotion recognition method based on AIGC, It is characterized in that The AIGC-based robot emotion recognition method includes: Collecting multimodal emotion data through a target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and performing cross-modal expansion processing on the multimodal emotion data to obtain an expanded emotion sample set; Extracting features from the extended emotion sample set to obtain an emotion feature vector; Performing context-related data mapping on the emotion feature vector to obtain context-related emotion representation data; Performing personalized feature analysis on the context-related emotion representation data to obtain mapped user emotion data, including: performing user feature extraction on the context-related emotion representation data to obtain an initial user emotion feature set, and performing sample synthesis on the initial user emotion feature set to obtain extended user emotion expression data; performing expression-speech-text joint analysis on the extended user emotion expression data to obtain a multimodal personalized feature matrix, and performing style conversion on the multimodal personalized feature matrix to obtain user-specific emotion expression samples; performing context-relevance analysis on the user-specific emotion expression samples to obtain a personalized emotion response pattern, and performing sequence prediction on the personalized emotion response pattern to obtain a user emotion response trend graph; performing semantic mapping on the user emotion response trend graph to obtain a personalized emotion vocabulary, and performing frequency-context analysis on the personalized emotion vocabulary to obtain a user emotion expression habit library; performing difference comparison on the user emotion expression habit library to obtain a personalized emotion feature deviation matrix; integrating the personalized emotion feature deviation matrix and the context-related emotion representation data through a transfer learning algorithm to obtain the mapped user emotion data; Performing multimodal data fusion and classification on the mapped user emotion data to obtain emotion classification data; The sentiment classification data is analyzed in multiple dimensions through a self-supervised learning algorithm to obtain sentiment expression parameters and interaction strategy recommendations.

2. The robot emotion recognition method based on AIGC according to claim 1, It is characterized in that The multimodal emotion data is collected by the target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and the multimodal emotion data is cross-modally extended to obtain an extended emotion sample set, including: Collecting multimodal emotion data through a target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and performing emotion semantic segmentation on the visual data to obtain an emotion region feature map; Extracting dynamic expression sequences from the emotion region feature graph to obtain time-series emotion feature data; performing spectral domain feature decomposition on the audio data to obtain acoustic features, and performing rhythmic pattern analysis on the acoustic features to obtain an emotion rhythmic feature vector; Performing sentiment vocabulary recognition on the text data to obtain a sentiment word embedding matrix, and performing context semantic enhancement on the sentiment word embedding matrix to obtain an expanded sentiment text representation; Performing cross-modal alignment on the temporal emotional feature data, the emotional prosody feature vector and the expanded emotional text representation to obtain a target emotional feature space; Performing data augmentation on the target emotion feature space by generating an adversarial network to obtain an extended emotion sample set, and performing emotion intensity gradient processing on the extended emotion sample set to obtain continuously changing emotion samples; The continuously changing emotion samples are synthesized into edge emotion cases to obtain the extended emotion sample set.

3. The robot emotion recognition method based on AIGC according to claim 1, It is characterized in that The step of extracting features from the extended emotion sample set to obtain an emotion feature vector includes: Performing multimodal emotion feature mapping on the extended emotion sample set to obtain an initial emotion high-dimensional feature space, and performing emotion semantic information extraction on the initial emotion high-dimensional feature space to obtain an emotion semantic enhanced feature matrix; Generating emotional virtual features on the emotional semantic enhancement feature matrix to obtain an emotional supplementary feature set, and performing emotional feature fusion on the emotional supplementary feature set to obtain a fused emotional feature representation; Performing emotional self-supervision reconstruction on the fused emotional feature representation to obtain emotional reconstruction feature data, and performing data comparison and analysis on the emotional reconstruction feature data and the extended emotional sample set to obtain a key emotional feature set; Performing cross-modal emotional feature alignment on the key emotional feature set to obtain emotional consistency feature mapping data; Dynamically adjusting the emotion dimension of the emotion consistency feature mapping data to obtain an adaptive emotion feature vector; Compressing the emotional information of the adaptive emotional feature vector to obtain efficient emotional feature coding; The emotional feature vector is obtained by performing emotional feature enhancement on the efficient emotional feature encoding through a contrastive learning algorithm.

4. The robot emotion recognition method based on AIGC according to claim 1, It is characterized in that The step of mapping the emotion feature vector to context-related data to obtain context-related emotion representation data includes: Performing semantic decomposition on the emotion feature vector to obtain multi-dimensional emotion semantic data, and performing context association analysis on the multi-dimensional emotion semantic data to obtain a preliminary context-emotion correspondence relationship; Performing knowledge graph mapping on the preliminary situation-emotion correspondence to obtain a structured situation-emotion network; Dynamically expanding the structured situational emotion network to obtain a situational knowledge base, and generating an emotion response spectrum for the situational knowledge base to obtain a diversified emotion response pattern; Performing time series prediction on the diversified emotional response patterns to obtain an emotional change trend graph, and performing scenario derivation on the emotional change trend graph to obtain a multi-path emotional development matrix; Performing similarity calculation on the multi-path emotion development matrix to obtain an emotion-context association pattern set, and performing comparative sample construction on the emotion-context association pattern set to obtain fine-grained context-emotion mapping data; The fine-grained context-emotion mapping data and the emotion feature vector are fused through a graph attention network algorithm to obtain context-related emotion representation data.

5. The robot emotion recognition method based on AIGC according to claim 1, It is characterized in that The step of performing multimodal data fusion and classification on the mapped user emotion data to obtain emotion classification data includes: Performing modality integrity detection on the mapped user emotion data to obtain a modality missing labeling matrix, and performing data completion processing on the mapped user emotion data according to the modality missing labeling matrix to obtain a target multimodal emotion data set; Performing cross-modal consistency evaluation on the target multimodal emotion dataset to obtain a modal consistency score vector, and performing threshold screening on the modal consistency score vector to obtain a high-confidence multimodal data subset; Dynamically assigning weights to the high-confidence multimodal data subset to obtain a modality fusion weight matrix; Performing weighted fusion on the modal fusion weight matrix and the high-confidence multimodal data subset to obtain a preliminary fusion feature vector, and performing context constraint processing on the preliminary fusion feature vector to obtain a context-sensitive fusion feature; Performing counterfactual analysis on the context-sensitive fusion features to obtain a modality importance assessment report, and generating decision rules for the modality importance assessment report to obtain a dynamic decision tree structure; The dynamic decision tree structure is mapped into a multi-dimensional sentiment space through an integrated learning algorithm to obtain sentiment classification data.

6. The robot emotion recognition method based on AIGC according to claim 5, It is characterized in that The multi-dimensional analysis of the emotion classification data by the self-supervised learning algorithm to obtain emotion expression parameters and interaction strategy suggestions includes: Performing multi-granularity interpretation generation on the sentiment classification data to obtain a hierarchical sentiment interpretation structure, and performing semantic decomposition on the hierarchical sentiment interpretation structure to obtain a sentiment cause element set; Performing weight calculation on the emotion cause factor set to obtain an emotion contribution matrix, and performing threshold analysis on the emotion contribution matrix to obtain key emotion triggering factors; Constructing counterexamples for the key emotion triggering factors to obtain an emotion boundary case set, and performing similarity calculation on the emotion boundary case set to obtain an emotion difference space map; Performing trajectory prediction on the emotion difference space map to obtain an emotion development script library, and performing context mapping on the emotion development script library to obtain an emotion-environment association network; Generate interactive questions for the emotion-environment association network to obtain deep explanation question-answer pairs; The deep explanation question-answer pairs are subjected to feature extraction and clustering through a self-supervised learning algorithm to obtain sentiment expression parameters and interaction strategy recommendations.

7. A robot emotion recognition system based on AIGC, It is characterized in that Used to execute the AIGC-based robot emotion recognition method according to any one of claims 1 to 6, the AIGC-based robot emotion recognition system comprises: A collection module, used to collect multimodal emotion data through a target robot, wherein the multimodal emotion data includes: visual data, audio data and text data, and perform cross-modal expansion processing on the multimodal emotion data to obtain an extended emotion sample set; An extraction module, used for performing feature extraction on the extended emotion sample set to obtain an emotion feature vector; A mapping module, used for performing context-related data mapping on the emotion feature vector to obtain context-related emotion representation data; An analysis module is used to perform personalized feature analysis on the context-related emotion representation data to obtain mapped user emotion data, including: extracting user features from the context-related emotion representation data to obtain an initial user emotion feature set, and performing sample synthesis on the initial user emotion feature set to obtain extended user emotion expression data; performing expression-speech-text joint analysis on the extended user emotion expression data to obtain a multimodal personalized feature matrix, and performing style conversion on the multimodal personalized feature matrix to obtain user-specific emotion expression samples; performing context-relevance analysis on the user-specific emotion expression samples to obtain a personalized emotion response pattern, and performing sequence prediction on the personalized emotion response pattern to obtain a user emotion response trend graph; performing semantic mapping on the user emotion response trend graph to obtain a personalized emotion vocabulary, and performing frequency-context analysis on the personalized emotion vocabulary to obtain a user emotion expression habit library; performing difference comparison on the user emotion expression habit library to obtain a personalized emotion feature deviation matrix; integrating the personalized emotion feature deviation matrix and the context-related emotion representation data through a transfer learning algorithm to obtain the mapped user emotion data; A fusion module, used to perform multimodal data fusion and classification on the mapped user emotion data to obtain emotion classification data; The analysis module is used to perform multi-dimensional analysis on the sentiment classification data through a self-supervised learning algorithm to obtain sentiment expression parameters and interaction strategy suggestions.

8. A computer-readable storage medium having instructions stored thereon, It is characterized in that When the instructions are executed by the processor, the AIGC-based robot emotion recognition method as described in any one of claims 1-6 is implemented.

Images