Emotion recognition model joint training method under modal lacking condition

Abstract

The invention relates to the technical field of machine learning, and particularly discloses an emotion recognition model joint training method under a mode lacking condition, which comprises the following steps: a server issues a global model to a training user; performing multi-modal training on the users participating in training to obtain local fusion features and a local model; the users participating in training upload the local model parameters and the fused emotion features to a server; the server performs feature aggregation to obtain global features and issues local model parameters to the test user, and the test user performs feature extraction by using a model of the training user and uploads the features to the server; and the server calculates the difference between the test features and the training features, and aggregates the local models according to the difference. The server issues the global model and the global features to the training user; and repeating the steps until the model converges. According to the method, the problems of incomplete modal data and heterogeneous user data are solved, so that the accuracy and robustness of the model when the incomplete modal data are processed are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a joint training method for an emotion recognition model under a modality-free condition. Background Art

[0002] In educational scenarios, emotion recognition technology is crucial to improving teaching interactions and student experience. Especially in the context of online education and distance learning, students' emotional states can be analyzed through multimodal data (such as video, audio, and text) to optimize teaching methods and content. However, in practical applications, especially in school environments, it is often impossible to obtain all modal data due to technical limitations or privacy policy constraints. For example, in some cases, students may only be able to provide text feedback but not video or audio data, which leads to the loss of modal data.

[0003] In addition, the data provided by different students may vary greatly in nature and quality due to different technical access conditions and personal privacy choices of each student. The lack of such modal data and its heterogeneity pose significant challenges to the training and application of emotion recognition models, especially in sentiment analysis that requires the integration of different types of modal data. Traditional emotion recognition methods usually rely on the integrity of multimodal data, so the performance and accuracy of the model may be greatly compromised in the absence of certain modal data. Summary of the invention

[0004] In response to the above-mentioned problems in the prior art, the present invention provides a joint training method for emotion recognition models suitable for modal-absent conditions in school education scenarios, which can effectively enhance the emotion recognition ability of the model when data is incomplete, strengthen the adaptability and robustness of the model in heterogeneous educational data environments, further optimize the performance of the global model, and achieve more accurate and effective emotion recognition in diversified educational environments.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A joint training method for emotion recognition model under modality-free conditions, the steps are as follows:

[0007] S1. The server sends the global model to the training user;

[0008] S2. The user initializes the parameters of the local model, extracts features from the multimodal data, and uses the cross-modal attention mechanism to fuse features to generate fused features. The fused features are classified and the supervised loss is calculated. At the same time, the modal alignment loss between different modal features is calculated.

[0009] S3, the user uploads the model parameters and fusion features to the server;

[0010] S4, the server aggregates the uploaded fusion features to form global features, and sends the collected local model set to the target user;

[0011] S5, the target users use each local model to extract features and upload the features to the server;

[0012] S6. The server calculates the difference between the test features and the training features, aggregates the local model according to the difference, and sends the global model and global features to the training user;

[0013] S7, the training user updates the model parameters and repeats S2, while calculating the global contrast loss;

[0014] S8. Repeat S3-S7 until the model converges.

[0015] Preferably, in S2, the cross-modal attention mechanism is as follows:

[0016] S21, use convolutional neural network to extract features from local multimodal data, denoted as z v ,z t ,z f , missing modal features are padded with zeros;

[0017] S22, concatenate the multimodal features and map them to the same dimension as the unimodal features, denoted as z con =[z v ||z t ||z f ];

[0018] S23, input the modal features obtained by S21-S22 into the Transformer of layer L to generate features;

[0019] S24, through CMA(X) = Attention(W Q X,W K Y,W V Y) is the query vector re-weighted to generate a single-mode fusion feature, and the features z of the text mode and image mode are calculated in the same way t,l+1 and z f,l+1 ;

[0020] Where vectors X and Y are given by CMA, X is the query vector from a single modality, Y is the key and value from the fused modality, and W Q , W K , W V Y is the weight matrix of query vector, key vector and value vector respectively;

[0021] S25: concatenate the single-mode fusion features obtained in S24 to obtain the final modal fusion features.

[0022] Preferably, in S2, each Transformer layer includes a cross-modal multi-head attention mechanism CMA, a layer normalization LN and a multi-layer perceptron MLP connected by a residual network.

[0023] Preferably, in S2, the process of Transformer generating speech features is v,l+1 = Transformer(z v,l ) are as follows:

[0024]

[0025]

[0026] In the formula, y v,l is the output of layer l, z v,l is the input of the lth layer, is the l-th layer fusion modality feature, z v,l+1 is the output of the l+1th layer.

[0027] Preferably, in S2, the calculation formula of the final modality fusion feature is:

[0028] Z fusion =[z v,l+1 ||z t,l+1 ||z f,l+1 ].

[0029] Preferably, in S2, the formula for calculating the modality alignment loss is as follows:

[0030] L d =Σ i≠j ||z i -z j || 2 ;

[0031] In the formula, z i and z j Represents the features of different modes, i and j represent different modes.

[0032] Preferably, in S6, the MMD distance is used to measure and minimize the distribution difference between the training data and the test data. The MMD calculates the difference between the two distributions using the embedding in the RKHS, and the calculation formula is:

[0033]

[0034] In the formula, represents the training features, Represents the test feature, represents the function mapped to RKHS, N tr Represents the amount of training data, N te represents the amount of test data, ||·|| H represents the norm in RKHS.

[0035] Preferably, in S6, the aggregation weight is The aggregation formula is:

[0036]

[0037] In the formula, E n,t represents the local model of user n in round t, E G,t represents the global model at round t.

[0038] Preferably, in S7, the global contrast loss is calculated to bring local features of the same category closer to global features, and to increase the distance between local features of different categories and global features.

[0039] Preferably, in S7, the global contrast loss calculation formula is:

[0040]

[0041] Where K is the number of categories, τ is the temperature coefficient, and Z k is the multimodal fusion representation of the kth category of the local user, Z j is the multimodal fusion representation of the jth category of the local user, P k is the global representation of the k-th category, where k is an integer.

[0042] The present invention has the following characteristics and advantages:

[0043] (1) By integrating students’ multimodal data (such as text, audio, video, etc.), this technology can more accurately identify students’ emotional states, thereby helping teachers adjust teaching methods to better meet students’ emotional and cognitive needs. This personalized teaching method can significantly improve students’ learning experience and learning outcomes;

[0044] (2) The present invention adopts the Transformer structure and context fusion attention mechanism to dynamically adapt to and strengthen the feature fusion between modalities, effectively dealing with the incompleteness and inconsistency of modal data; refines the model training by calculating the modal alignment loss and the global contrast loss to ensure the consistency and synchronization between different modal data;

[0045] (3) The present invention uses advanced loss functions such as contrastive loss, which can further enhance the model's ability to distinguish different student data when dealing with imbalance problems caused by modality missing or data heterogeneity, significantly improve the accuracy of emotion recognition, and ensure that each student can receive appropriate educational support and response, thereby improving the quality of education and student satisfaction.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments, but the embodiments should not be construed as limiting the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of an embodiment of a joint training method of an emotion recognition model under a modality-deficient condition of the present invention;

[0048] Figure 2 It is an algorithm framework diagram of an embodiment of a joint training method for emotion recognition models under modality-free conditions of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "include" or "comprise" and the like used in the present invention mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0051] like Figure 1 As shown in FIG. 1 , a joint training method for emotion recognition model under modality-free conditions is shown in FIG. 1 , and the steps are as follows:

[0052] S1. The server sends the global model to the training user;

[0053] S2. The user initializes the parameters of the local model, extracts features from the multimodal data, and uses the cross-modal attention mechanism to fuse features to generate fused features. The fused features are classified and the supervised loss is calculated. At the same time, the modal alignment loss between different modal features is calculated.

[0054] The cross-modal attention mechanism described in step S2 above is as follows:

[0055] S21, using a convolutional neural network to extract features from local multimodal data. The present invention takes speech, text and image as examples, denoted by zv , z t , z f , missing modal features are padded with zeros;

[0056] S22, concatenate the multimodal features and map them to the same dimension as the unimodal features, denoted as z con =[z v ||z t ||z f ];

[0057] S23, input the modal features obtained by S21-S22 into the L-layer Transformer to generate features, where each Transformer layer contains a cross-modal multi-head attention mechanism CMA, layer normalization LN and a multi-layer perceptron MLP, and the above modules are connected by a residual network;

[0058] S24, through CMA(X) = Attention(W Q X,W K Y,W V Y) is the query vector re-weighted to generate the single-mode fusion feature, and the features z of the text mode and image mode are calculated in the same way t,l+1 and z f,l+1 ;

[0059] Where vectors X and Y are given by CMA, X is the query vector from a single modality, Y is the key and value from the fused modality, and W Q , W K , W V Y is the weight matrix of query vector, key vector and value vector respectively;

[0060] S25, the single-mode fusion features obtained by S24 are concatenated to obtain the final modal fusion features, and the calculation formula is:

[0061] Z fusion =[z v,l+1 ||z t,l+1 ||z f,l+1 ];

[0062] In S2, the process of Transformer generating speech features z v,l+1 = Transformer(z v,l ) are as follows:

[0063]

[0064]

[0065] In the formula, y v,l is the output of layer l, z v,lis the input of the lth layer, is the l-th layer fusion modality feature, z v,l+1 is the output of the l+1th layer;

[0066] In S2, the modal alignment loss between different modal features is calculated. The formula for calculating the modal alignment loss is as follows:

[0067] L d =∑ i≠j ||z i -z j || 2 ;

[0068] In the formula, z i and z j Represents the characteristics of different modes, i and j represent different modes;

[0069] S3, the user uploads the model parameters and fusion features to the server;

[0070] S4, the server aggregates the uploaded fusion features to form global features, and sends the collected local model set to the target user;

[0071] S5, the target users use each local model to extract features and upload the features to the server;

[0072] S6. The server calculates the difference between the test features and the training features, aggregates the local model according to the difference, and sends the global model and global features to the training user;

[0073] In S6, the MMD distance is used to measure and minimize the distribution difference between training data and test data. Specifically, MMD uses the embedding in RKHS to calculate the difference between the two distributions, thereby providing an effective way to measure and bridge the statistical distance between different data sources. The calculation formula is:

[0074]

[0075] In the formula, represents the training features, Represents the test feature, represents the function mapped to RKHS, N tr Represents the amount of training data, N te represents the amount of test data, ||·|| H represents the norm in RKHS;

[0076] In S6, the aggregation weight is The aggregation formula is as follows:

[0077]

[0078] In the formula, E n,t represents the local model of user n in round t, E G,t is the global model of round t;

[0079] S7, the training user updates the model parameters and repeats S2, while calculating the global contrast loss;

[0080] The calculation of the global contrast loss described in S7 refers to bringing the distance between local features and global features of the same category closer and the distance between local features and global features of different categories farther. The calculation formula is as follows:

[0081]

[0082] Where K is the number of categories, τ is the temperature coefficient, and Z k is the multimodal fusion representation of the kth category of the local user, Z j is the multimodal fusion representation of the jth category of the local user, P k is the global representation of the k-th category, k is an integer;

[0083] S8. Repeat S3-S7 until the model converges.

[0084] The overall algorithm framework is shown in the figure Figure 2 shown.

[0085] This embodiment is applied to student sentiment analysis in an educational scenario, and the specific operations are as follows:

[0086] First, the server sends the global emotion recognition model to the teaching terminals in multiple classrooms. Since some classrooms may lack video or audio acquisition equipment, the teaching terminals need to initialize local model parameters based on the available modal data (such as text or images).

[0087] Secondly, each classroom terminal extracts features from the collected student feedback data, such as extracting language features from text feedback or facial expression features from images. After that, a cross-modal attention mechanism is applied to fuse these features to generate comprehensive emotional features, and supervised classification loss and modality alignment loss are calculated to adjust the model.

[0088] Thirdly, the teaching terminal uploads the local model parameters and the fused emotional features to the server. This step ensures that the server can obtain the emotional analysis data from a global perspective for further processing.

[0089] Then, after receiving the data uploaded by each classroom, the server aggregates the features to form a more comprehensive global emotional feature. The server will also send the updated global model and feature set to each classroom terminal so that each classroom can adjust and optimize its local model.

[0090] Finally, the classroom terminal updates the local model parameters according to the instructions sent by the server and re-executes the data processing and feature fusion steps. This process is repeated, and local training is performed each time based on the global features and updated model parameters until the model performance reaches a satisfactory stable state.

[0091] This embodiment effectively addresses the emotion recognition challenges caused by the lack of modal data in educational scenarios through the method of model joint training, and improves the adaptability and accuracy of the model to complex educational environments.

[0092] In summary, the method proposed in the present invention effectively improves the model performance and generalization ability, solves the problems of incomplete modal data and heterogeneous user data, thereby improving the accuracy and robustness of the model in processing incomplete modal data, and at the same time further helps educational institutions optimize resource allocation and improve teaching quality.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A joint training method for emotion recognition model under modality-free conditions, characterized in that: Here are the steps: S1. The server sends the global model to the training user; S2. The user initializes the parameters of the local model, extracts features from the multimodal data, and uses the cross-modal attention mechanism to fuse features to generate fused features. The fused features are classified and the supervised loss is calculated. At the same time, the modal alignment loss between different modal features is calculated. S3, the user uploads the model parameters and fusion features to the server; S4, the server aggregates the uploaded fusion features to form global features, and sends the collected local model set to the target user; S5, the target users use each local model to extract features and upload the features to the server; S6. The server calculates the difference between the test features and the training features, aggregates the local model according to the difference, and sends the global model and global features to the training user; S7, the training user updates the model parameters and repeats S2, while calculating the global contrast loss; S8. Repeat S3-S7 until the model converges.

2. The method for joint training of emotion recognition models under modality-free conditions according to claim 1, characterized in that: In S2, the cross-modal attention mechanism is as follows: S21, use convolutional neural network to extract features from local multimodal data, denoted as z v , z t , z f , missing modal features are padded with zeros; S22, concatenate the multimodal features and map them to the same dimension as the unimodal features, denoted as z con =[z v ||z t ||z f ]; S23, input the modal features obtained by S21-S22 into the Transformer of layer L to generate features; S24, through CMA(X) = Attention(W Q X,W K Y,W V Y) is the query vector re-weighted to generate a single-mode fusion feature, and the features z of the text mode and image mode are calculated in the same way t,l+1 and z f,l+1 ; Where vectors X and Y are given by CMA, X is the query vector from a single modality, Y is the key and value from the fused modality, and W Q , W K , W V Y is the weight matrix of query vector, key vector and value vector respectively; S25: concatenate the single-mode fusion features obtained in S24 to obtain the final modal fusion features.

3. The method for joint training of emotion recognition models under modality-free conditions according to claim 2, characterized in that: In S2, each Transformer layer contains a cross-modal multi-head attention mechanism CMA, layer normalization LN and a multi-layer perceptron MLP connected by a residual network.

4. The method for joint training of emotion recognition models under modality-free conditions according to claim 2, characterized in that: In S2, the process of Transformer generating speech features z v,l+1 = Transformer(z v,l ) are as follows: In the formula, y v,l is the output of layer l, z v,l is the input of the lth layer, is the l-th layer fusion modality feature, z v,l+1 is the output of the l+1th layer.

5. The method for joint training of emotion recognition models under modality-free conditions according to claim 2, characterized in that: In S2, the calculation formula of the final modal fusion feature is: WITH fusion =[of v,l+1 ||from t,l+1 ||from f,l+1 ]。 6. The method for joint training of emotion recognition models under modality-free conditions according to claim 1, characterized in that: In S2, the formula for calculating the modality alignment loss is as follows: L d =∑ i≠j ||from i -With j || 2 ; In the formula, z i and z j Represents the features of different modes, i and j represent different modes.

7. The method for joint training of emotion recognition models under modality-free conditions according to claim 1, characterized in that: In S6, the MMD distance is used to measure and minimize the distribution difference between the training data and the test data. The MMD uses the embedding in RKHS to calculate the difference between the two distributions. The calculation formula is: In the formula, represents the training features, Represents the test feature, represents the function mapped to RKHS, N tr Represents the amount of training data, N te represents the amount of test data, ||·|| H represents the norm in RKHS.

8. The method for joint training of emotion recognition models under modality-free conditions according to claim 1, characterized in that: In S6, the aggregation weight is The aggregation formula is: In the formula, E n,t represents the local model of user n in round t, E G,t represents the global model at round t.

9. The method for joint training of emotion recognition models under modality-free conditions according to claim 1, characterized in that: In S7, the global contrast loss is calculated to shorten the distance between local features and global features of the same category and to increase the distance between local features and global features of different categories.

10. The method for joint training of emotion recognition models under modality-free conditions according to claim 1, characterized in that: In S7, the global contrast loss calculation formula is: Where K is the number of categories, τ is the temperature coefficient, and Z k is the multimodal fusion representation of the kth category of the local user, Z j is the multimodal fusion representation of the jth category of the local user, P k is the global representation of the k-th category, where k is an integer.

Images