A serial multimodal emotion recognition method for robots

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically to a serial multimodal emotion recognition method for robots, comprising: obtaining an audio data sequence and an image data sequence; inputting an audio instruction text and an audio data sequence into a trained audio multimodal model, and outputting an audio feedback text; aligning the audio feedback text and the image data sequence by timestamp, and then inputting the trained image multimodal model, and outputting an emotion recognition result and an image feedback text; in a model training process, the initial audio and image multimodal models are trained separately in a pre-training phase, the pre-trained audio multimodal model and the image multimodal model are trained based on a cross-modal bidirectional feedback mechanism in a collaborative training phase, and the corresponding weights of the positive feedback and reverse feedback losses are dynamically adjusted in a target training phase, and the collaboratively trained audio multimodal model and the image multimodal model are trained. The present invention improves the processing efficiency and recognition accuracy of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a serial multimodal emotion recognition method and system for robots. Background Art

[0002] With the rapid development of artificial intelligence (AI), emotion recognition has become a key research area in the field of multimodal AI. The goal of emotion recognition is to extract emotional information from multimodal inputs (such as audio, images, and text) and classify them. The main challenges of multimodal emotion recognition lie in extracting effective emotional information from multiple input modalities and effectively fusing multimodal data, particularly considering the complementary and temporal nature of the two information sources: audio and images.

[0003] Currently, multimodal emotion recognition methods often use parallel processing and joint training to process multimodal data. Parallel processing methods treat audio and images as independent inputs, using convolutional neural networks (CNNs) to process image data and recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) to process audio data, before fusing the features of the two data types. While this method can process multimodal data simultaneously, it fails to fully exploit the potential dynamic interactions between audio and images. In emotion analysis tasks, audio features such as tone, intonation, and speech rate significantly influence the emotion recognition of image data (such as facial expressions and emotional states). However, the guiding role of audio data on image data is not effectively explored and utilized, resulting in inaccurate multimodal data fusion and reduced overall emotion recognition accuracy. Joint training methods optimize the fusion of audio and image features through multiple models or multiple tasks. Existing joint training methods often share audio and image features in the same representation space through multi-task learning (MTL) or shared representation learning, and then jointly optimize across multiple modal tasks. Although this method can improve the integration capability of the model, it requires a large amount of labeled data and computing resources, and the optimization process is relatively complex and easily affected by data imbalance or bias. At the same time, the lack of in-depth exploration of temporal information and feedback mechanisms leads to low accuracy in emotion recognition.

[0004] In addition, existing technologies often use feature splicing and weighted fusion to fuse multimodal data, but lack deep modeling of the complex interactive relationships between multimodal data, resulting in inefficient and inaccurate transmission and sharing of emotional information. At the same time, existing technologies do not consider the temporal changes of emotional information in real application scenarios that need to process user emotional fluctuations, resulting in low efficiency in information feedback and fusion processing of audio and image data, and thus low accuracy in emotion recognition. Summary of the Invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the low efficiency of parallel processing and joint training in the existing technology, the failure to consider emotional temporal information and feedback mechanism, and the failure of the fusion method to consider the interactive relationship of multimodal data, which leads to low accuracy of emotion recognition.

[0006] To solve the above technical problems, the present invention provides a serial multimodal emotion recognition method for robots, comprising:

[0007] Obtain the audio data sequence and image data sequence of the current emotional activity;

[0008] Input the preset audio command text and audio data sequence into the trained audio multimodal model, and output audio feedback text of the current emotional activity; the audio feedback text includes the volume dynamic range, fundamental frequency contour information and tone classification results of the audio data sequence;

[0009] After aligning the timestamps of the audio feedback text and image data sequence, the trained image multimodal model is input to output the emotion recognition results of the current emotional activity and the image feedback text; the image feedback text includes voice response and action response information.

[0010] Preferably, an audio-text training set is constructed based on each audio data sequence and its corresponding audio emotion label in an open source dataset, combined with the audio instruction text; wherein the open source dataset contains audio data sequences and their corresponding image data sequences;

[0011] Using the audio-text training set, the initial audio multimodal model is trained by minimizing the first audio loss function to obtain a pre-trained audio multimodal model;

[0012] Inputting the audio-text training set into the pre-trained audio multimodal model, extracting audio data feature vectors, and outputting each first audio feedback text; after aligning the timestamps of each first audio feedback text with each image data sequence, combining the image emotion labels corresponding to each image data sequence, constructing the first image-text training set;

[0013] The first image-text training set is used to train the initial image multimodal model by minimizing the first image loss function to obtain a pre-trained image multimodal model.

[0014] Preferably, the pre-trained audio multimodal model and image multimodal model are trained based on a cross-modal bidirectional feedback mechanism, including:

[0015] Input the first audio feedback text and image data sequence with aligned timestamps into the pre-trained image multimodal model. Using a CLIP architecture dual-tower encoder, encode the first audio feedback text and image data at the same timestamp, respectively, and extract the first audio feedback text feature vector and the image data feature vector.

[0016] generating a first modified weight matrix based on the image data feature vector extracted by the pre-trained image multimodal model and the audio data feature vector extracted by the pre-trained audio multimodal model;

[0017] Correcting the audio emotion probability distribution output by the pre-trained audio multimodal model classification layer according to the first corrected weight matrix, the emotion probability distribution output by the pre-trained audio multimodal model classification layer, and the emotion probability distribution output by the pre-trained image multimodal model classification layer to obtain a first corrected audio emotion probability distribution;

[0018] Constructing a first reverse feedback loss function based on the first corrected audio emotion probability distribution and the image emotion probability distribution output by the pre-trained image multimodal model classification layer;

[0019] Based on the first inverse feedback loss function and its corresponding weight and the first audio loss function and its corresponding weight, a second audio loss function is constructed; using the audio-text training set, the pre-trained audio multimodal model is trained by minimizing the second audio loss function to obtain a collaboratively trained audio multimodal model;

[0020] Input the audio-text training set into the collaboratively trained audio multimodal model, extract the audio data feature vector, and output each second audio feedback text; after aligning the timestamps of each second audio feedback text with each image data sequence, the second image-text training set is constructed by combining the image emotion labels corresponding to each image data sequence;

[0021] Constructing the first positive feedback loss function based on the audio emotion probability distribution output by the classification layer of the co-trained audio multimodal model;

[0022] Based on the first forward feedback loss function and its corresponding weights and the first image loss function and its corresponding weights, a second image loss function is constructed; using the second image-text training set, the pre-trained image multimodal model is trained by minimizing the second image loss function to obtain a collaboratively trained image multimodal model.

[0023] Preferably, the expression of the first modified weight matrix is:

[0024] ;

[0025] in, represents the first revised weight matrix; Represents the audio data feature vector extracted by the pre-trained audio multimodal model; Represents the image data feature vector extracted by the pre-trained image multimodal model; Represents the Sigmoid activation function; represents the learnable weight matrix;

[0026] The expression of the first modified audio emotion probability distribution is:

[0027] ;

[0028] in, represents the audio emotion probability distribution after the first correction; Represents element-wise multiplication operation; Represents the audio emotion probability distribution output by the classification layer of the pre-trained audio multimodal model; Represents the image emotion probability distribution output by the classification layer of the pre-trained image multimodal model.

[0029] Preferably, the first inverse feedback loss function The expression is:

[0030] ;

[0031] in, represents KL divergence; represents the audio emotion probability distribution after the first correction; represents the image emotion probability distribution output by the classification layer of the pre-trained image multimodal model;

[0032] The first forward feedback loss function The expression is:

[0033] ;

[0034] in, represents KL divergence; represents the audio emotion probability distribution output by the classification layer of the co-trained audio multimodal model; Represents the image emotion probability distribution output by the classification layer when training the pre-trained image multimodal model.

[0035] Preferably, it also includes:

[0036] The second audio feedback text and image data sequence with aligned timestamps are input into the collaboratively trained image multimodal model. The CLIP architecture dual-tower encoder encodes the second audio feedback text and image data at the same timestamp, extracting the second audio feedback text feature vector and the image data feature vector.

[0037] generating a second modified weight matrix based on the image data feature vector extracted by the collaboratively trained image multimodal model and the audio data feature vector extracted by the collaboratively trained audio multimodal model;

[0038] Correcting the audio emotion probability distribution output by the collaboratively trained audio multimodal model classification layer according to the second corrected weight matrix, the emotion probability distribution output by the collaboratively trained audio multimodal model classification layer, and the emotion probability distribution output by the collaboratively trained image multimodal model classification layer to obtain a second corrected audio emotion probability distribution;

[0039] Based on the second corrected audio emotion probability distribution and the image emotion probability distribution output by the classification layer of the collaboratively trained image multimodal model, a second inverse feedback loss function is constructed; according to the modal confidence range, the corresponding weight of the second inverse feedback loss function is dynamically adjusted to construct a third audio loss function;

[0040] Using the audio-text training set, the audio multimodal model after collaborative training is trained by minimizing the third audio loss function to obtain an audio multimodal model that completes the target training;

[0041] Input the audio-text training set into the target trained audio multimodal model to output each third audio feedback text; after aligning the timestamps of each third audio feedback text with each image data sequence, the third image-text training set is constructed in combination with the image emotion labels corresponding to each image data sequence;

[0042] Based on the audio emotion probability distribution output by the classification layer of the target trained audio multimodal model, a second positive feedback loss function is constructed; according to the modal confidence range, the corresponding weight of the second positive feedback loss function is dynamically adjusted to construct a third image loss function; wherein the modal confidence range is 0.1-0.3;

[0043] The collaboratively trained image multimodal model is trained using the third image-text training set by minimizing the third image loss function to obtain an image multimodal model that completes the target training.

[0044] Preferably, in the pre-training stage, the proportion of audio instruction text replaced by empty characters in the audio-text training set and the proportion of audio feedback text replaced by empty characters in the first image-text training set are both set to the first data ratio; in the collaborative training stage, the proportion of audio instruction text replaced by empty characters in the audio-text training set and the proportion of audio feedback text replaced by empty characters in the second image-text training set are both set to the second data ratio; in the target training stage, the proportion of audio instruction text replaced by empty characters in the audio-text training set and the proportion of audio feedback text replaced by empty characters in the third image-text training set are both set to the third data ratio.

[0045] Preferably, the step of training the initial audio multimodal model by minimizing the first audio loss function using the audio-text training set to obtain the pre-trained audio multimodal model comprises:

[0046] Based on a deep neural network, a teacher audio multimodal model is constructed. The audio data sequence and audio instruction text are used as the input of the teacher audio multimodal model, and the audio emotion label corresponding to the audio data sequence is used as the output of the teacher audio multimodal model. The teacher audio multimodal model is trained using an audio-text training set to obtain a trained teacher audio multimodal model.

[0047] Based on the trained teacher audio multimodal model, a student audio multimodal model is constructed as the initial audio multimodal model; a loss function between the teacher audio multimodal model and the student audio multimodal model is constructed as the first audio loss function; using the audio-text training set, the student audio multimodal model is trained by minimizing the first audio loss function to obtain the trained student multimodal model as the pre-trained audio multimodal model.

[0048] The present invention also provides a serial multimodal emotion recognition system for robots, comprising:

[0049] A data acquisition module is used to obtain the audio data sequence and image data sequence of the current emotional activity;

[0050] The perception module is connected to the data acquisition module and the instruction text acquisition module, including:

[0051] An audio processing unit, configured to input a preset audio instruction text and audio data sequence into a trained audio multimodal model and output an audio feedback text of the current emotional activity; wherein the audio feedback text includes the volume dynamic range, fundamental frequency contour information, and tone classification results of the audio data sequence;

[0052] The image processing unit is in communication with the audio processing unit; it is used to align the timestamps of the audio feedback text and the image data sequence, input the trained image multimodal model, and output the emotion recognition results of the current emotional activity and the image feedback text; wherein the image feedback text includes voice response and action response information.

[0053] Preferably, it also includes:

[0054] The interaction module is connected to the perception module; it is used to convert the voice response information in the image feedback text into an audio stream, and obtain the stiffness data, motion angle data and motion time data of each joint of the robot based on the action response information in the image feedback text;

[0055] The voice playback module is located on the robot head and is connected to the interaction module. It is used to play the audio stream at a preset volume.

[0056] The control module is located on top of the robot and is in communication with the interaction module. It is used to control the robot to perform corresponding limb movements based on the stiffness data, motion angle data, and motion time data of each joint of the robot.

[0057] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0058] (1) The serial multimodal emotion recognition method for robots described in the present invention processes audio and image data in series, first analyzes the audio data using an audio multimodal model, and inputs the results as conditions into an image multimodal model, thereby fully exploring the potential connections and interactive relationships between audio and image data and effectively capturing the temporal changes in emotional information. The trend of emotional changes in audio analysis can dynamically affect the image analysis process, enabling the model to better understand the fluctuations of emotions in the temporal dimension, and further improving the accuracy, naturalness, and fluency of emotion recognition. At the same time, this information fusion method that combines static image features with dynamic audio information makes the emotional analysis results more comprehensive and accurate, and improves the communication between the robot and the user. The naturalness and fluency of emotional interaction; the serial processing method of first processing the audio and then using audio feedback to optimize image processing can optimize the use of computing resources, reduce system complexity, improve the overall processing efficiency of the system, and demonstrate strong adaptability and flexibility in multiple practical application scenarios; in addition, through audio instruction text, the audio multimodal model is guided to process audio data in different emotional scenarios, so that the model can better adapt to various emotional activity scenarios and improve the ability to recognize emotions in different scenarios; open source data sets and real-time collected data are used to train audio and image multimodal models, so that the model can learn a rich variety of emotional expression patterns, enhancing the model's adaptability and generalization capabilities to different scenarios and data.

[0059] (2) The present invention discloses a serial multimodal emotion recognition method for robots, which adopts a progressive training strategy to train the audio multimodal model and the image multimodal model. In the pre-training stage, the audio multimodal model and the image multimodal model are trained separately. By adjusting the proportion of empty characters replacing text in the audio-text training set and the first image-text training set, the basic representation capabilities of independent modalities such as audio and vision are quickly established, avoiding the premature introduction of multimodal noise to cause feature confusion. In the collaborative training stage, the audio multimodal model and the image multimodal model are trained collaboratively, and conflicts are dynamically corrected, so that the audio multimodal model and the image multimodal model treat each modal information equally. In addition, by reducing the proportion of empty characters replacing text in the audio-text training set and the second image-text training set, the proportion of multimodal data in the training set is increased, which promotes the audio multimodal model to be more sensitive to the image multimodal model. The audio and image modalities learn and collaborate with each other to capture the complex relationship between audio and image, strengthen the complementarity between multimodal modes, and avoid single dominance; in order to further improve the accuracy and efficiency of model training, in the target training phase, while the audio multimodal model and the image multimodal model are collaboratively trained, the loss weight is adjusted according to the modal confidence threshold, and by further reducing the proportion of empty character replacement text in the audio-text training set and the third image-text training set, the proportion of multimodal data is further improved, and more refined learning and optimization are carried out on complex multimodal information, thereby enhancing the model's ability to process multimodal data, suppressing the interference of highly uncertain modalities, improving adaptability to different emotional activity scenarios, and improving the accuracy and stability of emotion recognition; this progressive training design improves training efficiency and achieves low-error, highly robust multimodal collaborative reasoning.

[0060] (3) The present invention describes a serial multimodal emotion recognition method for robots. According to the reverse feedback strategy in the cross-modal bidirectional feedback mechanism, the image emotion probability distribution output by the classification layer of the image multimodal model is fed back to the audio multimodal model to generate a modified weight matrix to adjust the output of the classification layer of the audio multimodal model, which helps to enhance the information interaction and collaborative optimization between different modal models, so that the audio multimodal model can adjust its own output according to the information of the image multimodal mode, thereby improving the accuracy and robustness of the model in audio emotion classification; at the same time, the cross-modal bidirectional feedback mechanism enables a bidirectional information flow between the audio multimodal model and the image multimodal model, which helps to coordinate the optimization of the two models; in the forward and reverse directions, ... in the forward and reverse directions, the audio multimodal model can adjust its own output according to the information of the image multimodal mode, thereby improving the accuracy and robustness of the model in audio emotion classification; in the forward and reverse directions, the audio multimodal model can adjust its own output according to the information of the image multimodal mode, thereby improving the accuracy and robustness of the model in audio emotion classification; in the forward and reverse directions, the audio multimodal model can adjust its own output according to the information of the image multimodal mode, thereby improving the accuracy and robustness of the model in audio emotion classification; in the forward and reverse directions, the audio multimodal model can adjust its own output according to the information of the image multimodal mode, thereby improving the accuracy and robustness of the model in audio emotion classification; in the forward and reverse directions, the audio During the feedback process, the emotion probability distribution output by the classification layer of the audio multimodal model will affect the image multimodal model; during the reverse feedback process, the emotion probability distribution output by the classification layer of the image multimodal model can be fed back to the audio multimodal model, promoting the deep fusion of audio and image modal information, enabling the audio multimodal model to use image modal information to better understand the audio data; in addition, the reverse feedback loss function and the forward feedback loss function are constructed to narrow the difference in emotion probability distribution output by the classification layer of the audio multimodal model and the image multimodal model, so as to make the emotion category results of the two models consistent, further improve the stability and reliability of the model, and enable the multimodal model to give more accurate and consistent results in emotion classification and recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0062] Figure 1 This is a flow chart of a serial multimodal emotion recognition method for robots provided by the present invention;

[0063] Figure 2 This is a flowchart of the training process of the audio multimodal model and the image multimodal model;

[0064] Figure 3 This is a schematic diagram of a serial multimodal emotion recognition system for robots provided by the present invention. DETAILED DESCRIPTION

[0065] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0066] Reference Figure 1 As shown, Figure 1 The present invention provides a serial multimodal emotion recognition method for robots, which specifically includes:

[0067] S11: Acquire the audio data sequence and image data sequence of the current emotional activity;

[0068] S12: Inputting a preset audio instruction text and an audio data sequence into a trained audio multimodal model, and outputting an audio feedback text of the current emotional activity; wherein the audio feedback text includes the volume dynamic range, pitch contour information, and tone classification result of the audio data sequence; wherein the preset audio instruction text is a text including the volume, pitch contour, and tone information;

[0069] S13: After aligning the timestamps of the audio feedback text and the image data sequence, the trained image multimodal model is input to output the emotion recognition results of the current emotional activity and the image feedback text; wherein the image feedback text includes voice response and action response information.

[0070] Reference Figure 2 As shown, Figure 2 This is a flowchart of the training process for the audio multimodal model and the image multimodal model; specifically, it includes:

[0071] S21: Pre-train the initial audio multimodal model and image multimodal model, including:

[0072] S211: Constructing an audio-text training set based on each audio data sequence and its corresponding audio emotion label in an open source dataset, combined with the audio instruction text; wherein the open source dataset includes audio data sequences and their corresponding image data sequences;

[0073] Using the audio-text training set, the initial audio multimodal model is trained by minimizing the first audio loss function to obtain a pre-trained audio multimodal model, including:

[0074] Based on a deep neural network, a teacher audio multimodal model is constructed; the audio data sequence and audio instruction text are used as the input of the teacher audio multimodal model, and the audio emotion label corresponding to the audio data sequence is used as the output of the teacher audio multimodal model. The teacher audio multimodal model is trained using an audio-text training set to obtain a trained teacher audio multimodal model; in the teacher audio multimodal model constructed based on a deep neural network, Mel spectrum analysis is used to extract acoustic features such as the fundamental frequency contour and formant shift of the input audio data sequence. At the same time, the BERT encoder is used to extract features of the input audio instruction text to establish an audio-text cross-modal spatiotemporal correlation matrix;

[0075] Using the knowledge distillation framework, a student audio multimodal model is constructed as the initial audio multimodal model based on the trained teacher audio multimodal model. A loss function is constructed between the teacher audio multimodal model and the student audio multimodal model as the first audio loss function. Using the audio-text training set, the student audio multimodal model is trained by minimizing the first audio loss function to obtain the trained student multimodal model as the completed pre-trained audio multimodal model.

[0076] Among them, the first audio loss function includes: feature alignment strategy loss, emotion distribution transfer mechanism loss and attention transfer loss, and its expression is:

[0077] ;

[0078] in, represents the feature alignment strategy loss; Represents the weight corresponding to the feature alignment strategy loss; Indicates the loss of the emotion distribution transfer mechanism; Represents the weight corresponding to the loss of the sentiment distribution transfer mechanism; Indicates loss of attention shifting mechanism; represents the weight corresponding to the attention transfer loss; in a specific embodiment of the present invention, , , ;

[0079] The loss function of the feature alignment strategy is the mean square error loss function, which is expressed as:

[0080] ;

[0081] in, represents the first Feature output of the layer; Represents the first Feature output of the layer; represents the number of intermediate layers of the trained teacher audio multimodal model and the student audio multimodal model aligned; wherein, the mean square error loss function is used as the loss function of the feature alignment strategy. By optimizing the mean square error loss function, the difference between the intermediate layer features of the student audio multimodal model and the intermediate layer features of the teacher audio multimodal model can be narrowed, and the difference in intermediate layer features between the student model and the teacher model can be constrained; at the same time, the teacher model has been trained with a large amount of data, and its intermediate layer features often contain richer and more representative audio information. Optimizing the feature alignment strategy with the mean square error loss function can enable the student audio multimodal model to learn a better feature representation of the intermediate layer of the teacher model, so as to better extract audio features and more accurately capture key features, thereby improving the accuracy of the task;

[0082] The loss function of the emotion distribution transfer mechanism is the KL divergence loss function, which is expressed as:

[0083] ;

[0084] in, represents the logits output of the trained teacher audio multimodal model; represents the logits output of the student audio multimodal model; represents the softmax function; represents the temperature hyperparameter of the KL divergence loss; represents KL divergence; wherein, the KL divergence loss function is used as the loss function of the emotion distribution transfer mechanism. By optimizing the KL divergence loss function, the difference between the audio emotion probability distribution output by the classification layer of the student audio multimodal model and the audio emotion probability distribution output by the classification layer of the teacher audio multimodal model can be narrowed; at the same time, the audio emotion probability distribution output by the teacher model is based on its learning and understanding of a large amount of audio data. By optimizing the emotion distribution transfer mechanism through the KL divergence loss function, the student model can learn from the teacher model's experience in audio emotion classification, so as to better grasp the emotion categories corresponding to different audios, improve its performance in the audio emotion recognition task, and make its output emotion classification results more accurate;

[0085] The loss function of the attention transfer mechanism is the matrix norm loss function, which is expressed as:

[0086] ;

[0087] in, Indicates the number of attention heads; represents the first The attention matrix of the attention head; represents the first The attention matrix of the attention head; represents a dimension mapping function; wherein, the matrix norm loss function is used as the loss function of the attention mechanism. By optimizing the matrix norm loss function, the difference between the attention weight distribution of the student audio multimodal model on the audio data sequence and the attention weight distribution of the teacher audio multimodal model on the audio data sequence can be narrowed; at the same time, the attention mechanism is optimized with the help of the matrix norm loss function, so that the student audio multimodal model can learn the attention weight distribution of the teacher audio multimodal model on the audio data sequence, so that the student model can more effectively focus on the key parts of the audio, ignore irrelevant information, and enhance the adaptability of the student model to different audio data;

[0088] S212: Inputting the audio-text training set into the pre-trained audio multimodal model, extracting audio data feature vectors, and outputting first audio feedback texts; after timestamp alignment of each first audio feedback text with each image data sequence, combining the image emotion labels corresponding to each image data sequence, constructing a first image-text training set; wherein, timestamp alignment of the audio feedback text output by the audio multimodal model with the image data sequence ensures that, during subsequent processing, the audio feedback information and the corresponding image information at the same time point can be analyzed simultaneously, thereby providing consistency in the temporal dimension for accurate emotion recognition;

[0089] Using the first image-text training set, the initial image multimodal model is trained by minimizing the first image loss function to obtain a pre-trained image multimodal model;

[0090] The image loss function includes cross-modal contrast loss and cross-modal attention alignment loss, and its expression is:

[0091] ;

[0092] in, represents the cross-modal contrast loss; Represents the weight corresponding to the cross-modal contrast loss; represents the cross-modal attention alignment loss; represents the weight corresponding to the cross-modal attention alignment loss;

[0093] Among them, the expression of the cross-modal contrast loss is:

[0094] ;

[0095] in, represents the number of samples; Indicates the The image feature vector of samples; Indicates the The text feature vector of samples; Indicates the The text feature vector of samples; represents the temperature hyperparameter of the cross-modal contrastive loss; represents a natural constant. By optimizing the cross-modal contrastive loss function based on a cross-modal contrastive learning mechanism, the distance between positive samples (pairs of audio feedback text and image data sequences at the same timestamp) in the feature space can be shortened, while the distance between negative samples can be increased. This allows for the mining of semantic associations and enhances their effectiveness. This results in a semantic association space between text and images, enabling the mining of potential semantic connections between audio feedback text and image data, and improving the model's ability to understand multimodal data.

[0096] The expression of the cross-modal attention alignment loss is:

[0097] ;

[0098] in, Indicates the number of sub-features in each sample; represents the cross entropy loss function; Indicates the The image feature vector of the sample image sub-feature matrix; Indicates the The text feature vector of the sample text sub-feature matrices; among them, by optimizing the cross-modal attention alignment loss function constructed based on the differentiable cross-modal attention alignment mechanism, the image multimodal model can focus on the key areas of text and image, achieve more refined local associations, and improve the multimodal data fusion effect, accuracy and flexibility; at the same time, based on the cross-modal contrast learning mechanism and the differentiable cross-modal attention alignment mechanism, the semantic association space and local dynamic association space between text and image are learned, which helps the model capture more detailed semantic relationships and dynamic changes between different modal data, and improve the semantic understanding and association reasoning capabilities of multimodal data.

[0099] S22: Based on a cross-modal bidirectional feedback mechanism, the pre-trained audio multimodal model and image multimodal model are collaboratively trained, including:

[0100] S221: Input the timestamp-aligned first audio feedback text and image data sequence into the pre-trained image multimodal model, and respectively encode the first audio feedback text and image data at the same timestamp through the CLIP architecture dual-tower encoder, and extract the first audio feedback text feature vector and the image data feature vector. The visual branch channel of the CLIP architecture dual-tower encoder uses the ViT-B / 32 visual Transformer to block encode the JPG image (patch size 32×32). The text branch channel of the CLIP architecture dual-tower encoder uses the RoBERTa-base model to extract the deep semantics of the instruction text. The CLIP architecture dual-tower encoder extracts features of the image and text separately, effectively fusing the features of the audio feedback text and image modalities to obtain the feature space of the text and image. This cross-modal feature fusion can fully utilize the complementary information of data from different modalities and enrich the model's understanding and representation capabilities of the data.

[0101] Based on a cross-modal bidirectional feedback mechanism, an inverse correction pathway is constructed. The image emotion probability distribution output by the image multimodal model is fed back to the audio multimodal model through a gated linear unit to generate a correction weight matrix to adjust the output of the audio multimodal model classification layer. This helps to enhance information interaction and collaborative optimization between different modal models, enabling the audio multimodal model to adjust its own output based on the information of the image multimodal mode, thereby improving the accuracy and robustness of the model in audio emotion classification. Specifically:

[0102] Based on the image data feature vector extracted by the pre-trained image multimodal model and the audio data feature vector extracted by the pre-trained audio multimodal model, a first modified weight matrix is generated, which is expressed as follows:

[0103] ;

[0104] in, represents the first revised weight matrix; Represents the audio data feature vector extracted by the pre-trained audio multimodal model; Represents the image data feature vector extracted by the pre-trained image multimodal model; Represents the Sigmoid activation function; represents the learnable weight matrix;

[0105] According to the first corrected weight matrix, the emotion probability distribution output by the classification layer of the pre-trained audio multimodal model, and the emotion probability distribution output by the classification layer of the pre-trained image multimodal model, the audio emotion probability distribution output by the classification layer of the pre-trained audio multimodal model is corrected to obtain a first corrected audio emotion probability distribution, which is expressed as follows:

[0106] ;

[0107] in, represents the audio emotion probability distribution after the first correction; Represents element-wise multiplication operation; Represents the audio emotion probability distribution output by the classification layer of the pre-trained audio multimodal model; represents the image emotion probability distribution output by the classification layer of the pre-trained image multimodal model;

[0108] S222: Based on the first corrected audio emotion probability distribution and the image emotion probability distribution output by the pre-trained image multimodal model classification layer, construct a first reverse feedback loss function, which is expressed as:

[0109] ;

[0110] in, represents KL divergence; represents the audio emotion probability distribution after the first correction; represents the image emotion probability distribution output by the classification layer of the pre-trained image multimodal model;

[0111] Based on the first inverse feedback loss function and its corresponding weight and the first audio loss function and its corresponding weight, a second audio loss function is constructed, which is expressed as:

[0112] ;

[0113] in, Represents the updated weight corresponding to the feature alignment strategy loss; Represents the updated weight corresponding to the loss of the sentiment distribution transfer mechanism; Represents the updated weight corresponding to the attention transfer loss; represents the reverse feedback loss function; Represents the weight corresponding to the inverse feedback loss function; 、 and A weight representing a first audio loss function;

[0114] Using the audio-text training set, the pre-trained audio multimodal model is trained by minimizing the second audio loss function to obtain a collaboratively trained audio multimodal model.

[0115] Input the audio-text training set into the collaboratively trained audio multimodal model, extract the audio data feature vector, and output each second audio feedback text; after aligning the timestamps of each second audio feedback text with each image data sequence, the second image-text training set is constructed by combining the image emotion labels corresponding to each image data sequence;

[0116] Based on the audio emotion probability distribution output by the classification layer of the co-trained audio multimodal model, the first positive feedback loss function is constructed, which is expressed as:

[0117] ;

[0118] in, represents KL divergence; represents the audio emotion probability distribution output by the classification layer of the co-trained audio multimodal model; represents the probability distribution of image emotions output by the classification layer when training the pre-trained image multimodal model;

[0119] S223: Based on the first forward feedback loss function and its corresponding weight and the first image loss function and its corresponding weight, construct a second image loss function, which is expressed as:

[0120] ;

[0121] in, represents the updated weight corresponding to the cross-modal contrast loss; represents the updated weight corresponding to the cross-modal attention alignment loss; represents the positive feedback loss function; Represents the weight corresponding to the positive feedback loss function; and Both represent the weights corresponding to the first image loss function;

[0122] The pre-trained image multimodal model is trained using the second image-text training set by minimizing the second image loss function to obtain a collaboratively trained image multimodal model.

[0123] S23: Based on the modal confidence range, dynamic balance training is performed on the collaboratively trained audio multimodal model and image multimodal model, including:

[0124] S231: Inputting the second audio feedback text and the image data sequence after the timestamps are aligned into the collaboratively trained image multimodal model, encoding the second audio feedback text and the image data at the same timestamp using a CLIP architecture dual-tower encoder, and extracting the second audio feedback text feature vector and the image data feature vector;

[0125] generating a second modified weight matrix based on the image data feature vector extracted by the collaboratively trained image multimodal model and the audio data feature vector extracted by the collaboratively trained audio multimodal model;

[0126] Correcting the audio emotion probability distribution output by the collaboratively trained audio multimodal model classification layer according to the second corrected weight matrix, the emotion probability distribution output by the collaboratively trained audio multimodal model classification layer, and the emotion probability distribution output by the collaboratively trained image multimodal model classification layer to obtain a second corrected audio emotion probability distribution;

[0127] S232: Constructing a second reverse feedback loss function based on the second corrected audio emotion probability distribution and the image emotion probability distribution output by the classification layer of the collaboratively trained image multimodal model; dynamically adjusting the corresponding weights of the second reverse feedback loss function according to the modal confidence range to construct a third audio loss function;

[0128] Using the audio-text training set, the audio multimodal model after collaborative training is trained by minimizing the third audio loss function to obtain an audio multimodal model that completes the target training;

[0129] S233: Input the audio-text training set into the target trained audio multimodal model to output each third audio feedback text; after aligning the timestamps of each third audio feedback text with each image data sequence, and combining the image emotion labels corresponding to each image data sequence, construct a third image-text training set;

[0130] Based on the audio emotion probability distribution output by the classification layer of the target trained audio multimodal model, a second positive feedback loss function is constructed; according to the modal confidence range, the corresponding weight of the second positive feedback loss function is dynamically adjusted to construct a third image loss function; wherein the modal confidence range is 0.1-0.3;

[0131] The collaboratively trained image multimodal model is trained using the third image-text training set by minimizing the third image loss function to obtain an image multimodal model that completes the target training.

[0132] In summary, in the pre-training stage, the proportion of audio instruction text replaced by empty characters in the audio-text training set and the proportion of audio feedback text replaced by empty characters in the first image-text training set are both set to the first data ratio; in the collaborative training stage, the proportion of audio instruction text replaced by empty characters in the audio-text training set and the proportion of audio feedback text replaced by empty characters in the second image-text training set are both set to the second data ratio; in the target training stage, the proportion of audio instruction text replaced by empty characters in the audio-text training set and the proportion of audio feedback text replaced by empty characters in the third image-text training set are both set to the third data ratio; the first data ratio is 30%. In the pre-training process, using less multimodal data allows the model to first learn the audio and image unimodal data separately. Feature representation lays a solid foundation and avoids introducing too much complex multimodal information at the beginning, which makes it difficult for the model to learn and converge, just like letting students master basic knowledge first and then learn comprehensive knowledge; the second data ratio is 70%. During the two-way collaborative training process, using more multimodal data can promote mutual learning and collaboration between audio and image modalities, optimize the model through a two-way feedback mechanism, improve the ability to fuse multimodal data, capture the complex relationship between audio and image, and better perform emotion recognition; the third data ratio is 90%. During the dynamic balance training process, a higher proportion of multimodal data allows the model to make full use of previously learned knowledge, conduct more refined learning and optimization of complex multimodal information, adapt to different emotional expressions and scenarios, and improve the accuracy and stability of emotion recognition;

[0133] In a specific embodiment of the present invention, the number of pre-training iterations accounts for 10% of the entire model training process, which can enable the model to initially build a single-modal feature extraction capability within a reasonable time, preparing for multimodal collaboration. The number of bidirectional collaborative training iterations accounts for 60% of the entire model training process, which can ensure that the model fully learns through bidirectional feedback and collaborative optimization, reduce the audio-image modality conflict rate, and improve emotion recognition capabilities in complex scenarios, such as effectively compressing recognition errors in facial occlusion scenarios. The number of dynamic balance training iterations accounts for 30% of the entire model training process, which can further improve the model's ability to process complex multimodal information without causing the model to overfit due to excessive training time.

[0134] Ultimately, the final audio and image multimodal serial model obtained through training reduced the conflict rate by 58%, and compressed the emotion recognition error from 22% to 7% in facial occlusion scenarios through reverse feedback. In the inference stage, zero-latency reverse calibration was achieved through pre-compiled correction matrices, and the memory usage only increased by 2.1MB, while still maintaining a real-time processing capability of 14ms / frame in the robot embedded system.

[0135] The advantages of the present invention are:

[0136] (1) First, the audio data is processed to generate audio text results, and then the image data is processed to ensure the efficient integration of multimodal data; the results of audio analysis are fed back to the image processing module as conditional information, which enhances the accuracy of emotion recognition, especially in the case of emotional fluctuations; the final emotional response is fed back through the robot's voice and action generation to achieve natural emotional interaction; by serializing audio processing and image processing, the emotional feedback of the audio directly affects the image emotion analysis results, enhancing the interaction between multimodal data; the large model not only analyzes emotions, but also generates targeted responses and provides feedback through robot actions and voice, improving the naturalness and accuracy of human-computer interaction; through this serial, multimodal data processing method, the accuracy of emotion recognition can be significantly improved, and the robot can be provided with richer and more natural emotional interaction capabilities.

[0137] (2) In the audio multimodal model, a teacher audio multimodal model with a large parameter scale is first trained. The model performs spectral analysis on the audio waveform through a deep neural network architecture, and combines the semantic understanding of the instructional text to establish a cross-modal association between audio features and text descriptions; and after the teacher model reaches stable performance, a more compact student model is constructed to achieve knowledge transfer through feature alignment and distribution transfer, that is: using the knowledge distillation framework, based on the trained teacher audio multimodal model, and through feature alignment strategy and transfer learning mechanism, a student audio multimodal model is constructed; while retaining the 95% recognition accuracy of the teacher audio multimodal model, the student audio multimodal model reduces the number of parameters to 30%, reducing the amount of model calculation, and successfully reducing the analysis delay of key acoustic features such as fundamental frequency contour and resonance peak shift to within 15ms, so that the complex multimodal emotion analysis algorithm can run in real time on embedded devices such as robots, improving the overall processing efficiency of the model, thereby being more efficient in processing audio data, improving the robustness and accuracy of the model, improving the emotion recognition performance in various scenarios, and improving the emotion recognition ability in complex scenarios.

[0138] (3) The audio feedback text output by the audio multimodal model is a natural language description text, which includes the processed volume, pitch, tone, emotion change trend and other information (such as "two sudden pitch increases in the middle of the conversation, which corresponds to a 62% increase in the probability of anger"); the structured text output by the audio multimodal model is constructed by analyzing multi-dimensional acoustic parameters such as volume dynamic range (unit dB, sampling interval 100ms), fundamental frequency contour (F0, accuracy ±2Hz), tone classification label (such as "rapid / smooth / fluctuating") and emotion intensity change gradient (ΔE), to construct a directional control system for robot interaction, that is, the volume and emotion intensity gradient are encoded into conditional vectors through a learnable projection matrix, and the cross-modal attention query weight of the visual Transformer is dynamically adjusted to make the image model focus in high-volume scenes. The robot uses parameterized mapping of the fundamental frequency contour mean and standard deviation to control the prosodic parameters of speech synthesis, achieving adaptive adjustment of pitch offset and speech rate (for example, when the fundamental frequency is greater than 220Hz, the speech response delay is compressed to 800ms). The tone label and emotion gradient jointly drive the action intensity controller, mapping joint stiffness and swing frequency through a nonlinear function (the finger tremor frequency is increased to 2Hz under rapid tone), so that the limb movement amplitude when the emotion intensity gradient is greater than 0.3 is enhanced to 1.2 times the baseline value. This design enables the robot to maintain an 82% emotion recognition accuracy in noisy environments with facial occlusion, the speech emotion matching MOS score is improved to 4.6, and the movement fluency and empathy satisfaction are increased by 63% and 29% respectively, forming a closed-loop optimization of the multimodal response chain driven by acoustic features.

[0139] (4) In the image multimodal model, the image is divided into small blocks for feature extraction through the visual branch, the text branch parses the instruction semantics, and the dynamic attention mechanism is used to associate the key areas of the image (such as facial expressions) with the text description; an innovative two-way feedback mechanism is introduced, and the image analysis results are used to reversely correct the emotional judgment of the audio model, and the consistency of the audio-visual data is dynamically balanced through weight fusion. At the same time, a phased training strategy is adopted to gradually increase the proportion of multimodal data, transitioning from single-modal learning to deep collaboration; finally, the audio multimodal model achieves millisecond-level response (14ms / frame) on the robot embedded device, and the recognition error of the face occlusion scene is reduced to 7%, and the memory usage is only increased by 2.1MB, which significantly improves the robot's ability to analyze complex emotions (such as forced laughter, suppressed anger), providing reliable technical support for natural emotional interaction; and based on the analysis results, outputs language responses and body responses to the current emotional activity; constructs a reverse attention pathway, so that a dual-mode relationship is formed between the audio multimodal model and the image multimodal model. The reverse information flow contributes to the coordinated optimization of the two models; in the forward process, the information of the audio multimodal model affects the image multimodal model; and after the reverse attention pathway is turned on, the information of the image multimodal model can be fed back to the audio multimodal model, promoting the deep fusion of audio and image modal information, so that the audio multimodal model can use the information of the image modality to better understand the audio data; the feedback information of the reverse attention pathway can help the audio multimodal model adjust its feature representation by generating a modified weight matrix to adjust the output of the model classification layer, providing additional supervision information for the optimization of the audio multimodal model, so that the audio multimodal model not only considers its own characteristics and tasks when generating the audio emotion probability distribution, but also refers to the emotional information of the image modality, thereby achieving more accurate emotion classification, helping the model to discover more essential features, so that when faced with different types of audio data and various complex practical application scenarios, it can more accurately perform emotion classification and feature extraction, and show better generalization performance.

[0140] (5) Based on the gradual nature of multimodal learning and the need for conflict resolution, the proportion of multimodal data in the three stages of the progressive training strategy is set to 30%, 70%, and 90% respectively; in the early stage of training (the first 10% stage), the model quickly establishes the basic representation capabilities of independent modalities such as audio and vision through single-modal pre-training (multimodal data only accounts for 30%), avoiding the premature introduction of multimodal noise that leads to feature confusion. For example, in the emotion recognition task, this stage focuses on learning the time-frequency characteristics of the Mel spectrum from pure audio data, or extracting facial key points from pure visual data, laying the foundation for subsequent fusion; then enters the mid-term (the middle 60% stage), and the proportion of multimodal data increases to 70%. Through two-way collaborative training, the complementarity between modalities is strengthened, and conflicts are dynamically corrected. For example, when visual information fails due to facial occlusion, the model can rely on audio features to infer emotions, and vice versa. This stage forces the model to treat each modal information equally through balanced multimodal exposure to avoid single dominance; at the end of training (the last 30% stage), the proportion of multimodal data increases to 90%, simulating the complex input distribution of real scenes (such as robots receiving voice, image and environmental sensor signals at the same time), and introducing a dynamic balance optimization strategy to adjust the loss weight according to the real-time prediction confidence and suppress the interference of highly uncertain modalities. This progressive design has been proven to gradually reduce the modal conflict rate from 18% in the early stage to 6% in the final stage, while improving training efficiency by 30% (compared to the single-stage strategy), ultimately achieving low-error (7%) and highly robust multimodal collaborative reasoning.

[0141] Reference Figure 3 As shown, Figure 3 This is a schematic diagram of a serial multimodal emotion recognition system for robots provided by the present invention; specifically comprising:

[0142] The data acquisition module is used to obtain the audio data sequence and image data sequence of the current emotional activity, including:

[0143] A microphone located on the robot's head; used to record audio data of current emotional activity within a preset period of time and construct an audio data sequence of the current emotional activity; wherein, the audio file format is WAV or OGG through the four microphones with a sensitivity of 40dB on the robot's head;

[0144] A camera is located on the robot's forehead; it is used to continuously capture multiple images of the current emotional activity and construct an image data sequence of the current emotional activity; wherein the camera captures images with a maximum resolution of 1288*968;

[0145] The perception module is connected to the data acquisition module and the instruction text acquisition module, including:

[0146] An audio processing unit, configured to input a preset audio instruction text and audio data sequence into a trained audio multimodal model and output an audio feedback text of the current emotional activity; wherein the audio feedback text includes the volume dynamic range, fundamental frequency contour information, and tone classification results of the audio data sequence;

[0147] An image processing unit, in communication with the audio processing unit, is configured to align the timestamps of the audio feedback text and the image data sequence, input the trained image multimodal model, and output the emotion recognition result of the current emotional activity and the image feedback text; wherein the image feedback text includes voice response and action response information;

[0148] The interaction module is connected to the perception module; it is used to convert the voice response information in the image feedback text into an audio stream, and obtain the stiffness data, motion angle data and motion time data of each joint of the robot based on the action response information in the image feedback text;

[0149] The voice playback module is located on the left and right sides of the robot head and is in communication with the interaction module; it is used to play the audio stream at a preset volume; wherein, the voice playback module has a built-in speaker with a maximum output power of 2W;

[0150] The control module is located on top of the robot and is in communication with the interaction module. It is used to control the robot to perform corresponding limb movements based on the stiffness data, motion angle data, and motion time data of each joint of the robot.

[0151] Wherein, the robot is a Naoqi robot.

[0152] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A serial multimodal emotion recognition method for robots, characterized in that: include: Obtain the audio data sequence and image data sequence of the current emotional activity; Input the preset audio command text and audio data sequence into the trained audio multimodal model, and output audio feedback text of the current emotional activity; the audio feedback text includes the volume dynamic range, fundamental frequency contour information and tone classification results of the audio data sequence; After aligning the timestamps of the audio feedback text and image data sequence, the trained image multimodal model is input to output the emotion recognition results of the current emotional activity and the image feedback text; the image feedback text includes voice response and action response information; Based on the cross-modal bidirectional feedback mechanism, the pre-trained audio multimodal model and image multimodal model are trained, including: Input the first audio feedback text and image data sequence with aligned timestamps into the pre-trained image multimodal model. Using a CLIP architecture dual-tower encoder, encode the first audio feedback text and image data at the same timestamp, respectively, and extract the first audio feedback text feature vector and the image data feature vector. generating a first modified weight matrix based on the image data feature vector extracted by the pre-trained image multimodal model and the audio data feature vector extracted by the pre-trained audio multimodal model; Correcting the audio emotion probability distribution output by the pre-trained audio multimodal model classification layer according to the first corrected weight matrix, the emotion probability distribution output by the pre-trained audio multimodal model classification layer, and the emotion probability distribution output by the pre-trained image multimodal model classification layer to obtain a first corrected audio emotion probability distribution; Constructing a first reverse feedback loss function based on the first corrected audio emotion probability distribution and the image emotion probability distribution output by the pre-trained image multimodal model classification layer; Based on the first inverse feedback loss function and its corresponding weight and the first audio loss function and its corresponding weight, a second audio loss function is constructed; using the audio-text training set, the pre-trained audio multimodal model is trained by minimizing the second audio loss function to obtain a collaboratively trained audio multimodal model; Input the audio-text training set into the collaboratively trained audio multimodal model, extract the audio data feature vector, and output each second audio feedback text; after aligning the timestamps of each second audio feedback text with each image data sequence, the second image-text training set is constructed by combining the image emotion labels corresponding to each image data sequence; Constructing the first positive feedback loss function based on the audio emotion probability distribution output by the classification layer of the co-trained audio multimodal model; Based on the first forward feedback loss function and its corresponding weights and the first image loss function and its corresponding weights, a second image loss function is constructed; using the second image-text training set, the pre-trained image multimodal model is trained by minimizing the second image loss function to obtain a collaboratively trained image multimodal model.

2. A serial multimodal emotion recognition method for robots according to claim 1, characterized in that: Construct an audio-text training set based on each audio data sequence and its corresponding audio emotion label in an open source dataset, combined with the audio instruction text; wherein the open source dataset contains audio data sequences and their corresponding image data sequences; Using the audio-text training set, the initial audio multimodal model is trained by minimizing the first audio loss function to obtain a pre-trained audio multimodal model. Inputting the audio-text training set into the pre-trained audio multimodal model, extracting audio data feature vectors, and outputting each first audio feedback text; after aligning the timestamps of each first audio feedback text with each image data sequence, combining the image emotion labels corresponding to each image data sequence, constructing the first image-text training set; The first image-text training set is used to train the initial image multimodal model by minimizing the first image loss function to obtain a pre-trained image multimodal model.

3. The serial multimodal emotion recognition method for robots according to claim 1, characterized in that: The expression of the first modified weight matrix is: ； in, represents the first revised weight matrix; Represents the audio data feature vector extracted by the pre-trained audio multimodal model; Represents the image data feature vector extracted by the pre-trained image multimodal model; Represents the Sigmoid activation function; represents the learnable weight matrix; The expression of the first modified audio emotion probability distribution is: ； in, represents the probability distribution of audio emotions after the first correction; Represents element-wise multiplication operation; Represents the audio emotion probability distribution output by the classification layer of the pre-trained audio multimodal model; Represents the image emotion probability distribution output by the classification layer of the pre-trained image multimodal model.

4. The serial multimodal emotion recognition method for robots according to claim 1, characterized in that: The first inverse feedback loss function The expression is: ； in, represents KL divergence; represents the probability distribution of audio emotions after the first correction; Represents the image emotion probability distribution output by the classification layer of the pre-trained image multimodal model; The first forward feedback loss function The expression is: ； in, represents KL divergence; represents the audio emotion probability distribution output by the classification layer of the co-trained audio multimodal model; Represents the image emotion probability distribution output by the classification layer when training the pre-trained image multimodal model.

5. The serial multimodal emotion recognition method for robots according to claim 1, characterized in that: Also includes: The second audio feedback text and image data sequence with aligned timestamps are input into the collaboratively trained image multimodal model. The CLIP architecture dual-tower encoder encodes the second audio feedback text and image data at the same timestamp, extracting the second audio feedback text feature vector and the image data feature vector. generating a second modified weight matrix based on the image data feature vector extracted by the collaboratively trained image multimodal model and the audio data feature vector extracted by the collaboratively trained audio multimodal model; Correcting the audio emotion probability distribution output by the collaboratively trained audio multimodal model classification layer according to the second corrected weight matrix, the emotion probability distribution output by the collaboratively trained audio multimodal model classification layer, and the emotion probability distribution output by the collaboratively trained image multimodal model classification layer to obtain a second corrected audio emotion probability distribution; Based on the second corrected audio emotion probability distribution and the image emotion probability distribution output by the classification layer of the collaboratively trained image multimodal model, a second inverse feedback loss function is constructed; according to the modal confidence range, the corresponding weight of the second inverse feedback loss function is dynamically adjusted to construct a third audio loss function; Using the audio-text training set, the audio multimodal model after collaborative training is trained by minimizing the third audio loss function to obtain an audio multimodal model that completes the target training; Input the audio-text training set into the target trained audio multimodal model to output each third audio feedback text; after aligning the timestamps of each third audio feedback text with each image data sequence, the third image-text training set is constructed in combination with the image emotion labels corresponding to each image data sequence; Based on the audio emotion probability distribution output by the classification layer of the target trained audio multimodal model, a second positive feedback loss function is constructed; according to the modal confidence range, the corresponding weight of the second positive feedback loss function is dynamically adjusted to construct a third image loss function; wherein the modal confidence range is 0.1-0.3; The collaboratively trained image multimodal model is trained using the third image-text training set by minimizing the third image loss function to obtain an image multimodal model that completes the target training.

6. The serial multimodal emotion recognition method for robots according to claim 5, characterized in that: In the pre-training stage, the ratio of the audio instruction text replaced by the blank character in the audio-text training set and the ratio of the audio feedback text replaced by the blank character in the first image-text training set are both set to the first data ratio; In the collaborative training stage, the proportion of audio instruction text replaced by empty characters in the audio-text training set and the proportion of audio feedback text replaced by empty characters in the second image-text training set are both set to the second data ratio; in the target training stage, the proportion of audio instruction text replaced by empty characters in the audio-text training set and the proportion of audio feedback text replaced by empty characters in the third image-text training set are both set to the third data ratio.

7. The serial multimodal emotion recognition method for robots according to claim 2, characterized in that: The method of training the initial audio multimodal model by minimizing the first audio loss function using the audio-text training set to obtain the pre-trained audio multimodal model includes: Based on a deep neural network, a teacher audio multimodal model is constructed. The audio data sequence and audio instruction text are used as the input of the teacher audio multimodal model, and the audio emotion label corresponding to the audio data sequence is used as the output of the teacher audio multimodal model. The teacher audio multimodal model is trained using an audio-text training set to obtain a trained teacher audio multimodal model. Based on the trained teacher audio multimodal model, a student audio multimodal model is constructed as the initial audio multimodal model; a loss function between the teacher audio multimodal model and the student audio multimodal model is constructed as the first audio loss function; using the audio-text training set, the student audio multimodal model is trained by minimizing the first audio loss function to obtain the trained student multimodal model as the pre-trained audio multimodal model.

8. A serial multimodal emotion recognition system for robots, characterized in that: include: A data acquisition module is used to obtain the audio data sequence and image data sequence of the current emotional activity; The perception module is connected to the data acquisition module and the instruction text acquisition module, including: An audio processing unit, configured to input a preset audio instruction text and audio data sequence into a trained audio multimodal model and output an audio feedback text of the current emotional activity; wherein the audio feedback text includes the volume dynamic range, fundamental frequency contour information, and tone classification results of the audio data sequence; An image processing unit, in communication with the audio processing unit, is configured to align the timestamps of the audio feedback text and the image data sequence, input the trained image multimodal model, and output the emotion recognition result of the current emotional activity and the image feedback text; wherein the image feedback text includes voice response and action response information; Based on the cross-modal bidirectional feedback mechanism, the pre-trained audio multimodal model and image multimodal model are trained, including: Input the first audio feedback text and image data sequence with aligned timestamps into the pre-trained image multimodal model. Using a CLIP architecture dual-tower encoder, encode the first audio feedback text and image data at the same timestamp, respectively, and extract the first audio feedback text feature vector and the image data feature vector. generating a first modified weight matrix based on the image data feature vector extracted by the pre-trained image multimodal model and the audio data feature vector extracted by the pre-trained audio multimodal model; Correcting the audio emotion probability distribution output by the pre-trained audio multimodal model classification layer according to the first corrected weight matrix, the emotion probability distribution output by the pre-trained audio multimodal model classification layer, and the emotion probability distribution output by the pre-trained image multimodal model classification layer to obtain a first corrected audio emotion probability distribution; Constructing a first reverse feedback loss function based on the first corrected audio emotion probability distribution and the image emotion probability distribution output by the pre-trained image multimodal model classification layer; Based on the first inverse feedback loss function and its corresponding weight and the first audio loss function and its corresponding weight, a second audio loss function is constructed; using the audio-text training set, the pre-trained audio multimodal model is trained by minimizing the second audio loss function to obtain a collaboratively trained audio multimodal model; Input the audio-text training set into the collaboratively trained audio multimodal model, extract the audio data feature vector, and output each second audio feedback text; after aligning the timestamps of each second audio feedback text with each image data sequence, the second image-text training set is constructed by combining the image emotion labels corresponding to each image data sequence; Constructing the first positive feedback loss function based on the audio emotion probability distribution output by the classification layer of the co-trained audio multimodal model; Based on the first forward feedback loss function and its corresponding weights and the first image loss function and its corresponding weights, a second image loss function is constructed; using the second image-text training set, the pre-trained image multimodal model is trained by minimizing the second image loss function to obtain a collaboratively trained image multimodal model.

9. The serial multimodal emotion recognition system for robots according to claim 8, characterized in that: Also includes: An interaction module, communicating with the perception module; It is used to convert the voice response information in the image feedback text into an audio stream, and obtain the stiffness data, motion angle data, and motion time data of each joint of the robot based on the action response information in the image feedback text; The voice playback module is located on the robot head and is connected to the interaction module. It is used to play the audio stream at a preset volume. The control module is located on top of the robot and is in communication with the interaction module. It is used to control the robot to perform corresponding limb movements based on the stiffness data, motion angle data, and motion time data of each joint of the robot.

Images