A multimodal emotion recognition method and device based on gradient dynamic adjustment

Abstract

This invention discloses a multimodal emotion recognition method and apparatus based on gradient dynamic adjustment. The method includes acquiring multimodal emotion recognition data to be recognized; inputting preprocessed multimodal emotion recognition data into a trained multimodal emotion recognition model to obtain multimodal emotion recognition results; wherein, the training process of the multimodal emotion recognition model includes: acquiring multimodal emotion recognition data with real labels as training samples; constructing a multimodal emotion recognition model; inputting the preprocessed training samples into the constructed multimodal emotion recognition model for training; updating model parameters based on a gradient dynamic adjustment strategy until the model parameters converge or reach the maximum number of iterations; the gradient dynamic adjustment strategy provided by this invention dynamically adjusts the gradients of different modal parameters during the training process, making the parameter adjustment speed of the multimodal models similar, ensuring balanced optimization, and improving the accuracy of the multimodal emotion recognition model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal emotion recognition method and apparatus based on gradient dynamic adjustment. Background Technology

[0002] Traditional emotion recognition methods typically rely on information from a single modality, such as speech, text, or video. However, emotion recognition is a complex problem, and relying solely on information from a single modality sometimes fails to provide accurate emotion identification. For example, two sprinters might be crying: one crying with joy at winning a championship, and the other crying with sadness at being disqualified for multiple false starts. With the development of social media, people can express emotions through various forms such as video, audio, and text. Multimodal information can more comprehensively reflect human emotional states. Therefore, more and more researchers are integrating information from multiple modalities for emotion recognition analysis. The training process of multimodal models usually involves training different modalities simultaneously towards the same optimization goal. However, in existing multimodal model training, the training effects of different modalities often vary. The faster and better-trained modalities can suppress the relatively poorer modalities, thus preventing the multimodal model from fully realizing its potential. Therefore, how to train different modalities in a balanced way to improve the overall performance of the multimodal model has become an urgent technical problem to be solved. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal emotion recognition method and device based on gradient dynamic adjustment, which can train different modalities in a balanced way to improve the overall performance of the multimodal model, thereby improving the accuracy of multimodal emotion recognition.

[0004] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0005] In a first aspect, the present invention provides a multimodal emotion recognition method based on gradient dynamic adjustment, comprising:

[0006] Acquire the multimodal emotion recognition data to be identified, and preprocess the multimodal emotion recognition data;

[0007] The preprocessed multimodal emotion recognition data is input into the trained multimodal emotion recognition model to obtain the multimodal emotion recognition results;

[0008] The training process of the multimodal emotion recognition model includes:

[0009] Obtain multimodal emotion recognition data with real labels as training samples, and preprocess the training samples;

[0010] A multimodal emotion recognition model is constructed. The preprocessed training samples are input into the constructed multimodal emotion recognition model for training. The model parameters are updated based on a gradient dynamic adjustment strategy until the model parameters converge or the maximum number of iterations is reached.

[0011] Optionally, the multimodal emotion recognition data includes audio data and video data.

[0012] Optionally, the preprocessing of the multimodal emotion recognition data includes:

[0013] Frame images are extracted from video data at uniform intervals, and the audio data is converted into a two-dimensional spectral map using fast Fourier transform. The two-dimensional spectral map is then expanded to generate a three-dimensional spectral map.

[0014] The frame image and the three-dimensional spectral map are used as preprocessing results.

[0015] Optionally, the multimodal emotion recognition model includes a feature extraction module, a feature fusion module, and a linear classifier;

[0016] The feature extraction module employs a 3DResNet18 network, using the multi-input channel 3DResNet18 network as the feature extraction network for the preprocessed video data. v The 3DResNet18 network with a single input channel is used as the feature extraction network for the preprocessed audio data. a ;

[0017] The feature fusion module fuses the feature extraction network φ using a splicing method. v and the feature extraction network φ α Extracted feature vector φ v (θ v ,x v (i) and eigenvector φ a (θ a ,x a (i)) generates a fused feature vector [φ v (θ v ,x v (i));φ a (θ a ,x a (i))], where, θ v θ a For the feature extraction network φ v and feature extraction network φ a Network parameters, x v (i), x a (i) represents the preprocessed video and audio data of the i-th training sample;

[0018] The linear classifier performs classification processing on the fused feature vector to generate prediction results.

[0019]

[0020] In the formula, For the prediction result of the i-th training sample, W = [W a W v [W] is the weight matrix of the linear classifier. a W v The eigenvectors φ are respectively a (θ a ,x a (i) and eigenvector φ v (θ v ,x v (i) is the weight matrix, and b is the bias vector.

[0021] Optionally, the gradient-based dynamic adjustment strategy for updating model parameters includes:

[0022] The loss value L is calculated based on the loss value and using the cross-entropy loss function. t :

[0023]

[0024] In the formula, L t Let y(i) be the loss value for the t-th iteration. Let be the true label of the i-th training sample and the prediction result of the t-th iteration; For the prediction results Let y(i) be the probability of the predicted result. For the prediction results Let N be the probability of the k-th sentiment category; N and M are the number of training samples and the number of sentiment categories, respectively.

[0025] The gradient of the model parameters corresponding to each mode is calculated using the chain rule:

[0026]

[0027]

[0028]

[0029]

[0030] In the formula, For the feature extraction network φ in the t-th iteration α and feature extraction network φ vNetwork parameters, Let be the weight matrix of the linear classifier in the t-th iteration;

[0031] The gradient difference ratio of the model parameters for each mode is calculated using the ratio of the gradient 2-norm of the model parameters:

[0032]

[0033]

[0034]

[0035]

[0036] In the formula, ||·|2 is the 2-norm. for The corresponding gradient difference ratio; the gradient difference ratio reflects the difference in convergence speed between different modes;

[0037] The adjustment factor for the model parameters is calculated based on the gradient difference ratio:

[0038]

[0039] In the formula, Let p be the adjustment factor for the parameter p in the t-th iteration, where p∈{W} a ,θ a W v ,θ v}, where α is the adjustment coefficient;

[0040] Update the model parameters based on the adjustment factor:

[0041]

[0042] In the formula, p t+1 p t Let be the model parameters for the (t+1)th and tth iterations. r is the learning rate, β1 and β2 are the first and second exponential decay rates, respectively. t β1 t-j β1 is the power of t and tj, β2 is the power of the first exponential decay rate β1. t β2 t-j For the second exponential decay rate β2 raised to the powers of t and tj; The gradient of the model parameters, is the adjustment factor for parameter p in the j-th iteration.

[0043] Secondly, the present invention provides a multimodal emotion recognition device based on gradient dynamic adjustment, the multimodal emotion recognition device comprising:

[0044] The data processing module is used to acquire the multimodal emotion recognition data to be identified and to preprocess the multimodal emotion recognition data.

[0045] The emotion recognition module is used to input the preprocessed multimodal emotion recognition data into the trained multimodal emotion recognition model to obtain the multimodal emotion recognition result;

[0046] The sample processing module is used to acquire multimodal emotion recognition data with real labels as training samples and to preprocess the training samples.

[0047] The model training module is used to construct a multimodal emotion recognition model. The preprocessed training samples are input into the constructed multimodal emotion recognition model for training. The model parameters are updated based on a gradient dynamic adjustment strategy until the model parameters converge or the maximum number of iterations is reached.

[0048] Optionally, the multimodal emotion recognition data includes audio data and video data;

[0049] The multimodal emotion recognition model includes a feature extraction module, a feature fusion module, and a linear classifier;

[0050] The feature extraction module employs a 3DResNet18 network, using the multi-input channel 3DResNet18 network as the feature extraction network for the preprocessed video data. v The 3DResNet18 network with a single input channel is used as the feature extraction network for the preprocessed audio data. a ;

[0051] The feature fusion module fuses the feature extraction network φ using a splicing method. v and the feature extraction network φ α Extracted feature vector φ v (θ v ,x v (i) and eigenvector φ a (θ a ,x a (i)) generates a fused feature vector [φ v (θ v ,x v (i));φ a (θ a ,x a (i))], where, θv θ a For the feature extraction network φ v and feature extraction network φ a Network parameters, x v (i), x a (i) represents the preprocessed video and audio data of the i-th training sample;

[0052] The linear classifier performs classification processing on the fused feature vector to generate prediction results.

[0053]

[0054] In the formula, For the prediction result of the i-th training sample, W = [W a W v [W] is the weight matrix of the linear classifier. a W v The eigenvectors φ are respectively a (θ a ,x a (i) and eigenvector φ v (θ v ,x v (i) is the weight matrix, and b is the bias vector;

[0055] The gradient-based dynamic adjustment strategy for updating model parameters includes:

[0056] The loss value L is calculated using the cross-entropy loss function. t :

[0057]

[0058] In the formula, L t Let y(i) be the loss value for the t-th iteration. Let be the true label of the i-th training sample and the prediction result of the t-th iteration; For the prediction results Let y(i) be the probability of the predicted result. For the prediction results Let N be the probability of the k-th sentiment category; N and M are the number of training samples and the number of sentiment categories, respectively.

[0059] Based on the loss value, the gradients of the model parameters corresponding to each mode are calculated using the chain rule:

[0060]

[0061]

[0062]

[0063]

[0064] In the formula, For the feature extraction network φ in the t-th iteration α and feature extraction network φ v Network parameters, Let be the weight matrix of the linear classifier in the t-th iteration;

[0065] The gradient difference ratio of the model parameters for each mode is calculated using the ratio of the gradient 2-norm of the model parameters:

[0066]

[0067]

[0068]

[0069]

[0070] In the formula, ||·|2 is the 2-norm. for The corresponding gradient difference ratio; the gradient difference ratio reflects the difference in convergence speed between different modes;

[0071] The adjustment factor for the model parameters is calculated based on the gradient difference ratio:

[0072]

[0073] In the formula, Let p be the adjustment factor for the parameter p in the t-th iteration, where p∈{W} a ,θ a W v ,θ v}, where α is the adjustment coefficient;

[0074] Update the model parameters based on the adjustment factor:

[0075]

[0076] In the formula, p t+1 p t Let be the model parameters for the (t+1)th and tth iterations. r is the learning rate, β1 and β2 are the first and second exponential decay rates, respectively. t β1 t-j β1 is the power of t and tj, β2 is the power of the first exponential decay rate β1. t β2t-j For the second exponential decay rate β2 raised to the powers of t and tj; The gradient of the model parameters, is the adjustment factor for parameter p in the j-th iteration.

[0077] Thirdly, the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the multimodal emotion recognition method described above.

[0078] Fourthly, the present invention provides a computer device / equipment / system, comprising:

[0079] Memory, used to store computer programs / instructions;

[0080] A processor for executing the computer program / instructions to implement the steps of the multimodal emotion recognition method described above.

[0081] Fifthly, the present invention provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the multimodal emotion recognition method described above.

[0082] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0083] This invention provides a gradient-dynamic adjustment-based multimodal emotion recognition method. It proposes a gradient dynamic adjustment strategy that dynamically adjusts the gradients of the multimodal modes during training. This strategy directly measures the difference in convergence speed between different modes based on the ratio of the gradients of different modal parameters at the previous time step. Based on the difference in convergence speed, it dynamically adjusts the gradients of different modes at the next time step. Modes with faster loss descent will have their adjustment speed slowed down, making the parameter optimization speed of different modes similar and ensuring balanced optimization of parameters across different modes. The proposed gradient-dynamic adjustment-based multimodal emotion recognition model can effectively fuse video and audio data, fully utilizing information from both video and audio modalities and improving the accuracy of the multimodal emotion recognition model. Attached Figure Description

[0084] Figure 1 This is a flowchart of the multimodal emotion recognition method based on gradient dynamic adjustment provided in the embodiments of the present invention;

[0085] Figure 2 This is a structural diagram of the multimodal emotion recognition model provided in an embodiment of the present invention;

[0086] Figure 3 This is a flowchart of updating model parameters based on a gradient dynamic adjustment strategy provided in an embodiment of the present invention;

[0087] Figure 4 This is a training loss curve using the traditional Adam optimization strategy provided in an embodiment of the present invention;

[0088] Figure 5 This is a training loss curve using a gradient dynamic adjustment strategy provided in an embodiment of the present invention;

[0089] Figure 6 This is a test accuracy curve provided by an embodiment of the present invention using the traditional Adam optimization strategy;

[0090] Figure 7 This is a test accuracy curve using a gradient dynamic adjustment strategy provided in an embodiment of the present invention;

[0091] Figure 8 This is a test accuracy diagram of the gradient dynamic adjustment strategy and the traditional Adam optimization strategy provided in the embodiments of the present invention on different multimodal fusion methods. Detailed Implementation

[0092] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0093] Example 1:

[0094] like Figure 1 As shown, this embodiment of the invention provides a multimodal emotion recognition method based on gradient dynamic adjustment, including the following steps:

[0095] Step S1: Obtain multimodal emotion recognition data with real labels as training samples, and preprocess the training samples.

[0096] Specifically, in this embodiment, the multimodal emotion recognition data includes audio data and video data.

[0097] The preprocessing of the multimodal emotion recognition data includes:

[0098] Frame images are extracted from video data at uniform intervals, and the audio data is converted into a two-dimensional spectral map using Fast Fourier Transform. The two-dimensional spectral map is then expanded to generate a three-dimensional spectral map. For example, in PyTorch, the unsqueeze() method is used to add a new dimension to a tensor without changing the data in the tensor.

[0099] The frame image and the three-dimensional spectral map are used as preprocessing results.

[0100] Step S2: Construct a multimodal emotion recognition model. Input the preprocessed training samples into the constructed multimodal emotion recognition model for training. Update the model parameters based on the gradient dynamic adjustment strategy until the model parameters converge or the maximum number of iterations is reached.

[0101] like Figure 2 As shown, in this specific embodiment, the multimodal emotion recognition model includes a feature extraction module, a feature fusion module, and a linear classifier;

[0102] The feature extraction module employs a 3DResNet18 network, using the multi-input channel 3DResNet18 network as the feature extraction network for the preprocessed video data. v The 3DResNet18 network with a single input channel is used as the feature extraction network for the preprocessed audio data. a ;

[0103] The feature fusion module fuses the feature extraction network φ using a splicing method. v and the feature extraction network φ α Extracted feature vector φ v (θ v ,x v (i) and eigenvector φ a (θ a ,x a (i)) generates a fused feature vector [φ v (θ v ,x v (i));φ a (θ a ,x a (i))], where, θ v θ a For the feature extraction network φ v and feature extraction network φ a Network parameters, x v (i), x a (i) represents the preprocessed video and audio data of the i-th training sample;

[0104] The linear classifier performs classification processing on the fused feature vector to generate prediction results.

[0105]

[0106] In the formula, For the prediction result of the i-th training sample, W = [W a W v [W] is the weight matrix of the linear classifier. a Wv The eigenvectors φ are respectively a (θ a ,x a (i) and eigenvector φ v (θ v ,x v (i) is the weight matrix, and b is the bias vector.

[0107] like Figure 3 As shown, specifically in this embodiment, the step of updating model parameters based on the gradient dynamic adjustment strategy includes:

[0108] Step 201: Calculate the loss value L using the cross-entropy loss function. t :

[0109]

[0110] In the formula, L t Let y(i) be the loss value for the t-th iteration. Let be the true label of the i-th training sample and the prediction result of the t-th iteration; For the prediction results Let y(i) be the probability of the predicted result. For the prediction results Let N be the probability of the k-th sentiment category; N and M are the number of training samples and the number of sentiment categories, respectively.

[0111] Step 202: Calculate the gradients of the model parameters corresponding to each mode based on the loss value and using the chain rule:

[0112]

[0113]

[0114]

[0115]

[0116] In the formula, For the feature extraction network φ in the t-th iteration α and feature extraction network φ v Network parameters, Let be the weight matrix of the linear classifier in the t-th iteration;

[0117] Step 203: Calculate the gradient difference ratio of the model parameters for each mode using the ratio of the gradient 2-norm of the model parameters.

[0118]

[0119]

[0120]

[0121]

[0122] In the formula, ||·|2 is the 2-norm. for The corresponding gradient difference ratio; the gradient difference ratio reflects the difference in convergence speed between different modes;

[0123] Step 204: Calculate the adjustment factors for the model parameters based on the gradient difference ratio:

[0124]

[0125] In the formula, Let p be the adjustment factor for the parameter p in the t-th iteration, where p∈{W} a ,θ a W v ,θ v}, where α is the adjustment coefficient;

[0126] Step 205: Update the model parameters based on the adjustment factor:

[0127]

[0128] In the formula, p t+1 p t Let be the model parameters for the (t+1)th and tth iterations. r is the learning rate, β1 and β2 are the first and second exponential decay rates, respectively. t β1 t-j β1 is the power of t and tj, β2 is the power of the first exponential decay rate β1. t β2 t-j For the second exponential decay rate β2 raised to the powers of t and tj; The gradient of the model parameters, is the adjustment factor for parameter p in the j-th iteration.

[0129] Step S3: Obtain the multimodal emotion recognition data to be identified, and preprocess the multimodal emotion recognition data.

[0130] Step S4: Input the preprocessed multimodal emotion recognition data into the trained multimodal emotion recognition model to obtain the multimodal emotion recognition result.

[0131] To verify the multimodal emotion recognition method based on gradient dynamic adjustment proposed in this embodiment, 7440 training samples were taken, including 6 major common emotion categories: anger, happiness, sadness, neutrality, abandonment, disgust and fear. There were 1240 samples for each emotion category, and the length of each audio and video segment was 10 seconds.

[0132] The preprocessing involves sampling three frames of images at uniform intervals from each video segment; and converting the audio data into a two-dimensional spectral map of size 257×1004 using a Fast Fourier Transform of length 512.

[0133] The adjustment coefficient α is set to 0.6, the learning rate r is set to 0.01, and the exponential decay rates β1 and β2 are set to 0.9 and 0.999, respectively.

[0134] like Figure 4 As shown, this is the training loss curve using the traditional Adam optimization strategy; Figure 5 The figure shows the training loss curves using the gradient dynamic adjustment strategy. The solid line (video-audio model) represents the loss obtained by inputting video and audio into the multimodal model; the dotted line (video) represents the loss obtained by inputting the video feature vector from the multimodal model into the classifier; and the dashed line (audio) represents the loss obtained by inputting the audio feature vector from the multimodal model into the classifier. Experimental results show that using the traditional Adam optimization strategy, the video modality loss decreases much faster than the audio modality loss during training, playing a dominant role in the optimization process. At convergence, the audio modality loss is significantly smaller than the video modality loss, indicating that the audio modality is not sufficiently optimized. However, using the gradient dynamic adjustment strategy of this invention, the video modality loss decreases more slowly during training, and the difference between the audio and video modal losses is significantly reduced at convergence, indicating that both modalities are optimized in a balanced manner.

[0135] like Figure 6 As shown, this is the test accuracy curve using the traditional Adam optimization strategy; Figure 7 As shown, the test accuracy curves using the gradient dynamic adjustment strategy are presented. The solid line (video-audio model) represents the test accuracy obtained by using video and audio as inputs to the multimodal model; the dotted line (video) represents the test accuracy obtained by using video feature vectors from the multimodal model as inputs to the classifier; and the dashed line (audio) represents the test accuracy obtained by using audio feature vectors from the multimodal model as inputs to the classifier. The results show that the multimodal model using the traditional Adam optimization strategy achieves a similar overall test accuracy to the video modality because the video modality dominates the entire training process. However, the multimodal model using the gradient dynamic adjustment strategy proposed in this invention, while exhibiting lower test accuracies for both the video and audio modalities, demonstrates a higher overall test accuracy, indicating that this method effectively integrates information from different modalities.

[0136] like Figure 8 The figure shows the test accuracy of the gradient dynamic adjustment strategy and the traditional Adam optimization strategy in this embodiment on different multimodal fusion methods. Evaluation results show that the gradient dynamic adjustment strategy of this method not only achieves higher test accuracy than the traditional Adam optimization strategy in the splicing fusion method, but also achieves higher test accuracy in some other common multimodal fusion methods. The improvement in test accuracy is most significant in the FiLM fusion method, with a 22.4% improvement compared to the traditional Adam optimization strategy. This indicates that the gradient dynamic adjustment strategy of this method can reasonably adjust the training process of video and audio modalities, thus optimizing the overall performance of the multimodal model.

[0137] Example 2:

[0138] This invention provides a multimodal emotion recognition device based on gradient dynamic adjustment, the multimodal emotion recognition device comprising:

[0139] The data processing module is used to acquire the multimodal emotion recognition data to be identified and to preprocess the multimodal emotion recognition data.

[0140] The emotion recognition module is used to input the preprocessed multimodal emotion recognition data into the trained multimodal emotion recognition model to obtain the multimodal emotion recognition result;

[0141] The sample processing module is used to acquire multimodal emotion recognition data with real labels as training samples and to preprocess the training samples.

[0142] The model training module is used to construct a multimodal emotion recognition model. The preprocessed training samples are input into the constructed multimodal emotion recognition model for training. The model parameters are updated based on a gradient dynamic adjustment strategy until the model parameters converge or the maximum number of iterations is reached.

[0143] Specifically,

[0144] The multimodal emotion recognition data includes audio data and video data;

[0145] The multimodal emotion recognition model includes a feature extraction module, a feature fusion module, and a linear classifier;

[0146] The feature extraction module employs a 3DResNet18 network, using the multi-input channel 3DResNet18 network as the feature extraction network for the preprocessed video data. v The 3DResNet18 network with a single input channel is used as the feature extraction network for the preprocessed audio data. a ;

[0147] The feature fusion module fuses the feature extraction network φ using a splicing method. v and the feature extraction network φ α Extracted feature vector φ v (θ v ,x v (i) and eigenvector φ a (θ a ,x a (i)) generates a fused feature vector [φ v (θ v ,x v (i));φ a (θ a ,x a (i))], where, θ v θ a For the feature extraction network φ v and feature extraction network φ a Network parameters, x v (i), x a (i) represents the preprocessed video and audio data of the i-th training sample;

[0148] The linear classifier performs classification processing on the fused feature vector to generate prediction results.

[0149]

[0150] In the formula, For the prediction result of the i-th training sample, W = [W a W v [W] is the weight matrix of the linear classifier. a W v The eigenvectors φ are respectively a (θ a ,x a (i) and eigenvector φ v (θ v ,x v (i) is the weight matrix, and b is the bias vector;

[0151] The gradient-based dynamic adjustment strategy for updating model parameters includes:

[0152] The loss value L is calculated using the cross-entropy loss function. t :

[0153]

[0154] In the formula, L t Let y(i) be the loss value for the t-th iteration. Let be the true label of the i-th training sample and the prediction result of the t-th iteration; For the prediction results Let y(i) be the probability of the predicted result. For the prediction results Let N be the probability of the k-th sentiment category; N and M are the number of training samples and the number of sentiment categories, respectively.

[0155] Based on the loss value, the gradients of the model parameters corresponding to each mode are calculated using the chain rule:

[0156]

[0157]

[0158]

[0159]

[0160] In the formula, For the feature extraction network φ in the t-th iteration α and feature extraction network φ v Network parameters, Let be the weight matrix of the linear classifier in the t-th iteration;

[0161] The gradient difference ratio of the model parameters for each mode is calculated using the ratio of the gradient 2-norm of the model parameters:

[0162]

[0163]

[0164]

[0165]

[0166] In the formula, ||·|2 is the 2-norm. for The corresponding gradient difference ratio; the gradient difference ratio reflects the difference in convergence speed between different modes;

[0167] The adjustment factor for the model parameters is calculated based on the gradient difference ratio:

[0168]

[0169] In the formula, Let p be the adjustment factor for the parameter p in the t-th iteration, where p∈{W} a ,θ a W v ,θ v}, where α is the adjustment coefficient;

[0170] Update the model parameters based on the adjustment factor:

[0171]

[0172] In the formula, p t+1 p t Let be the model parameters for the (t+1)th and tth iterations. r is the learning rate, β1 and β2 are the first and second exponential decay rates, respectively. t β1 t-j β1 is the power of t and tj, β2 is the power of the first exponential decay rate β1. t β2 t-j For the second exponential decay rate β2 raised to the powers of t and tj; The gradient of the model parameters, is the adjustment factor for parameter p in the j-th iteration.

[0173] Example 3:

[0174] Based on Embodiment 1, this embodiment of the invention provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the multimodal emotion recognition method described above.

[0175] Example 4:

[0176] Based on Embodiment 1, this embodiment of the invention provides a computer device / equipment / system, including:

[0177] Memory, used to store computer programs / instructions;

[0178] A processor for executing the computer program / instructions to implement the steps of the multimodal emotion recognition method described above.

[0179] Example 5:

[0180] Based on Embodiment 1, this embodiment of the invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multimodal emotion recognition method described above.

[0181] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0183] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0185] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method based on gradient dynamic adjustment, characterized in that, include: Acquire the multimodal emotion recognition data to be identified, and preprocess the multimodal emotion recognition data; The preprocessed multimodal emotion recognition data is input into the trained multimodal emotion recognition model to obtain the multimodal emotion recognition results; The training process of the multimodal emotion recognition model includes: Obtain multimodal emotion recognition data with real labels as training samples, and preprocess the training samples; A multimodal emotion recognition model is constructed. The preprocessed training samples are input into the constructed multimodal emotion recognition model for training. The model parameters are updated based on a gradient dynamic adjustment strategy until the model parameters converge or the maximum number of iterations is reached. The gradient-based dynamic adjustment strategy for updating model parameters includes: The loss value is calculated using the cross-entropy loss function. : In the formula, For the first The loss value of the next iteration. For the first The true label of the training sample and the first training sample The prediction results of the next iteration; For the prediction results For the prediction results The probability of; For the prediction results For the first The probability of a certain emotion category; The number of training samples and the number of sentiment categories; Based on the loss value, the gradients of the model parameters corresponding to each mode are calculated using the chain rule: In the formula, For the first Sub-iteration feature extraction network and feature extraction network Network parameters, For the first The weight matrix of the linear classifier in the next iteration; The gradient difference ratio of the model parameters for each mode is calculated using the ratio of the gradient 2-norm of the model parameters: In the formula, It is a 2-norm. for The corresponding gradient difference ratio; Adjustment factors for the model parameters based on contribution calculation: In the formula, For the first Next iteration parameters Regulatory factors, , This is the adjustment coefficient; Update the model parameters based on the adjustment factor: In the formula, For the first Model parameters for the next iteration , For learning rate, The first exponential decay rate and the second exponential decay rate, The first exponential decay rate of Power of 1 The second exponential decay rate of Power; The gradient of the model parameters, , For the first Next iteration parameters The regulatory factor.

2. The multimodal emotion recognition method based on gradient dynamic adjustment according to claim 1, characterized in that, The multimodal emotion recognition data includes audio data and video data.

3. The multimodal emotion recognition method based on gradient dynamic adjustment according to claim 2, characterized in that, The preprocessing of the multimodal emotion recognition data includes: Frame images are extracted from video data at uniform intervals, and the audio data is converted into a two-dimensional spectral map using fast Fourier transform. The two-dimensional spectral map is then expanded to generate a three-dimensional spectral map. The frame image and the three-dimensional spectral map are used as preprocessing results.

4. The multimodal emotion recognition method based on gradient dynamic adjustment according to claim 2, characterized in that, The multimodal emotion recognition model includes a feature extraction module, a feature fusion module, and a linear classifier; The feature extraction module employs a 3DResNet18 network, using the multi-input channel 3DResNet18 network as the feature extraction network for the preprocessed video data. The 3DResNet18 network with a single input channel is used as the feature extraction network for the preprocessed audio data. ; The feature fusion module fuses the feature extraction network using a splicing method. and the feature extraction network Extracted feature vectors and eigenvectors Generate fused feature vectors ,in, For feature extraction network and feature extraction network Network parameters, For the first The preprocessed video and audio data of each training sample; The linear classifier performs classification processing on the fused feature vector to generate prediction results. : In the formula, For the first Prediction results for each training sample Here is the weight matrix of the linear classifier. respectively, feature vectors and eigenvectors The weight matrix, This is the bias vector.

5. A multimodal emotion recognition device based on gradient dynamic adjustment, characterized in that, The multimodal emotion recognition device includes: The data processing module is used to acquire the multimodal emotion recognition data to be identified and to preprocess the multimodal emotion recognition data. The emotion recognition module is used to input the preprocessed multimodal emotion recognition data into the trained multimodal emotion recognition model to obtain the multimodal emotion recognition result; The sample processing module is used to acquire multimodal emotion recognition data with real labels as training samples and to preprocess the training samples. The model training module is used to construct a multimodal emotion recognition model. The preprocessed training samples are input into the constructed multimodal emotion recognition model for training. The model parameters are updated based on a gradient dynamic adjustment strategy until the model parameters converge or the maximum number of iterations is reached. The step of updating model parameters based on gradient dynamic adjustment strategy includes: The loss value is calculated using the cross-entropy loss function. : In the formula, For the first The loss value of the next iteration. For the first The true label of the training sample and the first training sample The prediction results of the next iteration; For the prediction results For the prediction results The probability of; For the prediction results For the first The probability of a certain emotion category; The number of training samples and the number of sentiment categories; Based on the loss value, the gradients of the model parameters corresponding to each mode are calculated using the chain rule: In the formula, For the first Sub-iteration feature extraction network and feature extraction network Network parameters, For the first The weight matrix of the linear classifier in the next iteration; The gradient difference ratio of the model parameters for each mode is calculated using the ratio of the gradient 2-norm of the model parameters: In the formula, It is a 2-norm. for The corresponding gradient difference ratio; Adjustment factors for the model parameters based on contribution calculation: In the formula, For the first Next iteration parameters Regulatory factors, , This is the adjustment coefficient; Update the model parameters based on the adjustment factor: In the formula, For the first Model parameters for the next iteration , For learning rate, The first exponential decay rate and the second exponential decay rate, The first exponential decay rate of Power of 1 The second exponential decay rate of Power; The gradient of the model parameters, , For the first Next iteration parameters The regulatory factor.

6. The multimodal emotion recognition device based on gradient dynamic adjustment according to claim 5, characterized in that, The multimodal emotion recognition data includes audio data and video data; The multimodal emotion recognition model includes a feature extraction module, a feature fusion module, and a linear classifier; The feature extraction module employs a 3DResNet18 network, using the multi-input channel 3DResNet18 network as the feature extraction network for the preprocessed video data. The 3DResNet18 network with a single input channel is used as the feature extraction network for the preprocessed audio data. ; The feature fusion module fuses the feature extraction network using a splicing method. and the feature extraction network Extracted feature vectors and eigenvectors Generate fused feature vectors ,in, For feature extraction network and feature extraction network Network parameters, For the first The preprocessed video and audio data of each training sample; The linear classifier performs classification processing on the fused feature vector to generate prediction results. : In the formula, For the first Prediction results for each training sample Here is the weight matrix of the linear classifier. respectively, feature vectors and eigenvectors The weight matrix, This is the bias vector.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the multimodal emotion recognition method according to any one of claims 1-4.

8. A computer device / equipment / system, characterized in that, include: Memory, used to store computer programs / instructions; A processor for executing the computer program / instructions to implement the steps of the multimodal emotion recognition method according to any one of claims 1-4.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the multimodal emotion recognition method according to any one of claims 1-4.

Images