Emotion recognition method and system for children programming teaching

Abstract

The invention discloses an emotion recognition method and system for children programming teaching. The method comprises the steps of data acquisition, data preprocessing, facial emotion recognition, physiological emotion recognition and emotion recognition report generation. The invention belongs to the technical field of emotion recognition, and particularly relates to an emotion recognition method and system for children programming teaching. According to the scheme, an emotion recognition bimodal classification model is adopted, the cross-culture and cross-language problems can be effectively solved, and the emotion recognition model is more universal; the multi-mode emotion recognition method is adopted, emotion information of different modes is combined, the emotion of the child can be better captured, and the emotion recognition accuracy and the context understanding ability are improved.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition technology, and specifically refers to an emotion recognition method and system for children's programming teaching. Background Art

[0002] Emotion recognition has widespread applications in mental health, entertainment, education, and other fields. In education, it aims to help educators better understand and grasp students' emotional states in the classroom, thereby adjusting teaching strategies to improve teaching quality and cultivate students' learning motivation. However, existing emotion recognition processes face technical challenges, such as significant differences in emotional expression across different cultures and languages, which limits the model's universality across different environments. Furthermore, traditional emotion recognition models struggle to accurately capture the emotional characteristics of children, given their volatile emotions and the significant differences in their emotional expressions from adults. Summary of the Invention

[0003] In response to the above situation, in order to overcome the shortcomings of the existing technology, the present invention provides an emotion recognition method and system for children's programming teaching. In view of the technical problem that in the process of emotion recognition, there are large differences in the expression of emotions in different cultural and language regions, which leads to limited universality of the model in different environments, this solution adopts an emotion recognition bimodal classification model, which can effectively overcome cross-cultural and cross-language problems and make the emotion recognition model more universal; in view of the technical problem that in the process of emotion recognition, children's emotions are changeable and the expression of emotions between children and adults is very different, and traditional emotion recognition models are difficult to accurately capture the emotional characteristics of children, this solution adopts a multimodal emotion recognition method, which combines emotional information of different modalities, can better capture children's emotions, and improve the accuracy of emotion recognition and context understanding ability.

[0004] The technical solution adopted by the present invention is as follows: The present invention provides an emotion recognition method for children's programming teaching, which includes the following steps:

[0005] Step S1: data collection;

[0006] Step S2: data preprocessing;

[0007] Step S3: facial emotion recognition, specifically, using a deep convolutional neural network to build a model, optimizing model training by constructing a global loss function, obtaining a facial emotion recognition model Model1, and performing facial emotion recognition using the facial emotion recognition model Model1 to obtain facial emotion information;

[0008] Step S4: Physiological emotion recognition, specifically, using a bidirectional long short-term memory neural network to build a model, constructing a forward long short-term memory neural network unit and a backward long short-term memory neural network unit to perform model training, thereby obtaining a physiological emotion recognition model Model 2, and using the physiological emotion recognition model Model 2 to perform physiological emotion recognition and obtain physiological emotion recognition information;

[0009] Step S5: Emotion recognition report generation, specifically combining the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model and the emotion recognition bimodal classification model, performing multimodal emotion recognition, obtaining children's emotion recognition information, and generating an emotion recognition report.

[0010] As a further improvement of this solution, in step S1, the data collection is specifically to obtain emotion recognition data, and the emotion recognition data includes facial image data, children's EEG signal data and emotion labels, and the emotion labels include joy, anger, sadness, fear and neutrality.

[0011] As a further improvement of this solution, in step S2, the data preprocessing includes the following steps:

[0012] Step S21: normalizing, enhancing, denoising and detecting faces on facial image data to obtain smoothed image data;

[0013] Step S22: Using a bandpass filter to perform noise reduction processing on the children's EEG signal data to obtain noise-reduced signal data.

[0014] As a further improvement of this solution, in step S3, the facial emotion recognition includes the following steps:

[0015] Step S31: facial feature extraction, specifically using a local GaborXOR pattern to extract facial features, wherein the local GaborXOR pattern specifically refers to a local Gabor XOR pattern for facial feature extraction;

[0016] The facial feature extraction comprises the following steps:

[0017] Step S311: selecting a smoothed image from the smoothed image data to obtain a smoothed image A;

[0018] Step S312: Select a pixel point from the smoothed image A as the center pixel to obtain the center pixel a, and add all adjacent pixels centered on the center pixel a to an adjacent pixel set B, where the adjacent pixel set B contains m adjacent pixels b;

[0019] Step S313: Calculating facial feature vectors, including the following steps:

[0020] Step S3131: Calculate the local Gabor weight, the calculation formula is:

[0021] ;

[0022] Where Wt is the local Gabor weight, i is the adjacent pixel index, m is the number of adjacent pixels, and Wb i is the weight of the i-th neighboring pixel, dis i is the distance between the ith adjacent pixel and the center pixel;

[0023] Step S3132: Calculate the facial feature vector based on the local Gabor weights for facial feature representation. The calculation formula is:

[0024] ;

[0025] Where, is the facial feature vector, is the Gabor filter scale factor, is the Gabor filter directional factor, a is the center pixel, is a facial feature subvector composed of m adjacent pixels, is a facial feature subvector composed of m-1 adjacent pixels, It is a facial feature subvector composed of 1 adjacent pixels, and binary is a binary operator;

[0026] Step S314: Repeat steps S312 and S313 to traverse all center pixels on the smoothed image A to obtain a subset of facial feature vectors;

[0027] Step S315: Repeat steps S311, S312, S313, and S314 to calculate a facial feature vector subset for each smoothed image in the smoothed image data, and merge all facial feature vector subsets to obtain a facial feature vector set;

[0028] Step S32: constructing a facial emotion recognition model, specifically using a deep convolutional neural network to construct the model, including the following steps:

[0029] Step S321: Construct a multi-label focus loss function to optimize the prediction effect. The calculation formula is:

[0030] ;

[0031] Where, is the multi-label focal loss function, ε is the balance parameter used to adjust the recall rate, is the focus parameter, which is used to adjust the model's attention to samples with different labels, zi is the target expected value, which is used to represent the true label of the target category, u is the category prediction probability, which is used to represent the prediction result of the target category, j is the emotion category index, Mi is the number of emotion categories, and u j is the prediction result of the jth emotion category, zi j is the true label of the jth emotion category;

[0032] Step S322: Construct a regression loss function to reduce the prediction error. The calculation formula is:

[0033] ;

[0034] Where, is the regression loss function, is the loss function parameter, za is the dimension prediction probability, ui is the dimension expectation value, k is the data dimension index, Ni is the number of data dimensions, za k is the prediction result of the kth data dimension, ui k is the true label of the kth data dimension;

[0035] Step S323: Construct a global loss function for optimizing model training. The calculation formula is:

[0036] ;

[0037] Where, is the global loss function, is a global loss function weight factor, which is used to adjust the relative importance of the multi-label focus loss function and the regression loss function in the global loss function;

[0038] Step S33: deep convolutional neural network model training, specifically, training the model by constructing a global loss function to obtain a facial emotion recognition model Model1;

[0039] Step S34: using the facial emotion recognition model Model1 to perform facial emotion recognition to obtain facial emotion information.

[0040] As a further improvement of this solution, in step S4, the physiological emotion recognition includes the following steps:

[0041] Step S41: Extracting EEG signal features, specifically extracting features from the noise reduction signal data, including the following steps:

[0042] Step S411: Calculate the probability density function based on the noise reduction signal data. The calculation formula of the probability density function is:

[0043] ;

[0044] Where g(c) is the probability density function, c is the value of the noise reduction signal data, and e is the base of the natural exponential. is the mean of the noise reduction signal data, is the variance of the denoised signal data;

[0045] Step S412: Calculate the differential entropy of the noise reduction signal based on the probability density function to represent the characteristics of the EEG signal. The calculation formula is:

[0046] ;

[0047] Where G(C) is the differential entropy of the denoised signal;

[0048] Step S42: constructing a physiological emotion recognition model, specifically using a bidirectional long short-term memory neural network to construct the model, including the following steps:

[0049] Step S421: obtaining EEG signal features from the noise reduction signal differential entropy to obtain an EEG signal feature vector set;

[0050] Step S421: Construct a forward long short-term memory neural network unit to capture the sequential correlation of the EEG signal feature vector. The calculation formula is:

[0051] ;

[0052] Where t is the time step, is the hidden state of the forward LSTM neural network unit at the current time step, LSTM() is the LSTM neural network operation, y l,t is the lth EEG signal feature vector at the current time step, is the hidden state of the forward LSTM neural network unit at the previous time step;

[0053] Step S422: Construct a backward long short-term memory neural network unit to capture the reverse order correlation of the EEG signal feature vector. The calculation formula is:

[0054] ;

[0055] Where, is the hidden state of the backward long short-term memory neural network unit at the current time step, is the hidden state of the backward long short-term memory neural network unit at the next time step;

[0056] Step S423: Calculate the output of the bidirectional long short-term memory neural network. The calculation formula is:

[0057] ;

[0058] Where, is the output of the bidirectional long short-term memory neural network, W1 is the forward long short-term memory neural network unit weight, W2 is the backward long short-term memory neural network unit weight, is a bias vector of the bidirectional long short-term memory neural network, and the bias vector of the bidirectional long short-term memory neural network is used to adjust the deviation of the output of the bidirectional long short-term memory neural network;

[0059] Step S43: constructing a forward long short-term memory neural network unit and a backward long short-term memory neural network unit to perform model training to obtain a physiological emotion recognition model Model 2;

[0060] Step S44: using the physiological emotion recognition model Model2 to perform physiological emotion recognition and obtain physiological emotion recognition information.

[0061] As a further improvement of this solution, in step S5, the emotion recognition report is generated, including the following steps:

[0062] Step S51: Constructing an emotion recognition bimodal classification model, including the following steps:

[0063] Step S511: obtaining a bimodal classification model by integrating a deep convolutional neural network and a bidirectional long short-term neural network;

[0064] Step S512: inputting the facial feature vector set and the EEG signal feature vector set into the bimodal classification model to perform model training to obtain the emotion recognition bimodal classification model Model 3;

[0065] Step S52: Multimodal emotion recognition, specifically combining the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model, and the emotion recognition bimodal classification model to perform multimodal emotion recognition, including the following steps:

[0066] Step S521: Calculate the normalization coefficient. The calculation formula is:

[0067] ;

[0068] Wherein, NC is the normalization coefficient, P is the emotion category joy, Q is the emotion category anger, R1 is the emotion category fear, R2 is the emotion category sadness, Model1(P) is the facial emotion information, specifically the prediction result of the facial emotion classification model for the emotion category joy, Model2(Q) is the physiological emotion recognition information, specifically the prediction result of the physiological emotion recognition model for the emotion category anger, Model3(R1, R2) is the prediction result of the emotion recognition bimodal classification model for the emotion categories fear and sadness;

[0069] Step S522: Calculate the probability distribution of children's emotion recognition based on the normalization coefficient. The calculation formula is:

[0070] ;

[0071] Where, is the probability distribution of children's emotion recognition, are five categories of emotions, wherein the values ​​of the five categories of emotions include joy, anger, sadness, fear and neutral;

[0072] Step S523: Select the emotion category with the largest probability value from the children's emotion recognition probability distribution to obtain children's emotion recognition information;

[0073] Step S53: Generate an emotion recognition report based on the children's emotion recognition information.

[0074] The present invention provides an emotion recognition system for children's programming teaching, comprising: a data acquisition module, a data preprocessing module, a facial emotion recognition module, a physiological emotion recognition module and an emotion recognition report generation module;

[0075] The data acquisition module is used for data acquisition, specifically collecting data related to children's emotion recognition, obtaining emotion recognition data through acquisition, and sending the emotion recognition data to the data preprocessing module;

[0076] The data preprocessing module is used for data preprocessing, specifically preprocessing the emotion recognition data to obtain smoothed image data and noise reduction signal data, and sending the smoothed image data to the facial emotion recognition module, and sending the noise reduction signal data to the physiological emotion recognition module;

[0077] The facial emotion recognition module is used for facial emotion recognition, specifically extracting features from the smoothed image data and using a deep convolutional neural network to perform facial emotion recognition, obtaining facial emotion recognition information, and sending the facial emotion recognition information to the emotion recognition report generation module.

[0078] The physiological emotion recognition module is used for physiological emotion recognition, specifically performing feature extraction on the noise reduction signal data and using a bidirectional long short-term memory neural network to perform physiological emotion recognition to obtain physiological emotion recognition information, and sending the physiological emotion recognition information to the emotion recognition report generation module;

[0079] The emotion recognition report generation module is used to generate emotion recognition reports. Specifically, it constructs an emotion recognition bimodal classification model to obtain an emotion recognition bimodal classification model, combines the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model and the emotion recognition bimodal classification model, performs multimodal emotion recognition, obtains children's emotion recognition information, and generates an emotion recognition report.

[0080] The beneficial effects achieved by the present invention using the above scheme are as follows:

[0081] (1) In order to address the technical problem that in the emotion recognition process, there are large differences in the expression of emotions in different cultural and language regions, which leads to the limited universality of the model in different environments, this solution adopts an emotion recognition bimodal classification model, which can effectively overcome cross-cultural and cross-language problems and make the emotion recognition model more universal.

[0082] (2) In view of the technical problem that children’s emotions are changeable and their emotional expressions are very different from those of adults during the emotion recognition process, traditional emotion recognition models find it difficult to accurately capture the emotional characteristics of children, this solution adopts a multimodal emotion recognition method that combines emotional information from different modalities, which can better capture children’s emotions and improve the accuracy of emotion recognition and context understanding ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A flowchart of an emotion recognition method for children's programming teaching provided by the present invention;

[0084] Figure 2 A schematic diagram of an emotion recognition system for children's programming teaching provided by the present invention;

[0085] Figure 3 Schematic diagram of the process of step S2;

[0086] Figure 4 Schematic diagram of the process of step S3;

[0087] Figure 5 Schematic diagram of the process of step S4;

[0088] Figure 6 is a flow chart of step S5.

[0089] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0090] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0091] In the description of the present invention, it should be understood that terms such as "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0092] Example 1, see Figure 1 The present invention provides an emotion recognition method for children's programming teaching, which includes the following steps:

[0093] Step S1: data collection;

[0094] Step S2: data preprocessing;

[0095] Step S3: facial emotion recognition, specifically, using a deep convolutional neural network to build a model, optimizing model training by constructing a global loss function, obtaining a facial emotion recognition model Model1, and performing facial emotion recognition using the facial emotion recognition model Model1 to obtain facial emotion information;

[0096] Step S4: Physiological emotion recognition, specifically, using a bidirectional long short-term memory neural network to build a model, constructing a forward long short-term memory neural network unit and a backward long short-term memory neural network unit to perform model training, thereby obtaining a physiological emotion recognition model Model 2, and using the physiological emotion recognition model Model 2 to perform physiological emotion recognition and obtain physiological emotion recognition information;

[0097] Step S5: Emotion recognition report generation, specifically combining the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model and the emotion recognition bimodal classification model, performing multimodal emotion recognition, obtaining children's emotion recognition information, and generating an emotion recognition report.

[0098] Example 2, see Figure 1 In step S1, the data collection specifically involves acquiring emotion recognition data, which includes facial image data, children's EEG signal data, and emotion labels. The emotion labels include joy, anger, sadness, fear, and neutrality.

[0099] Example 3, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S2, the data preprocessing includes the following steps:

[0100] Step S21: normalizing, enhancing, denoising and detecting faces on facial image data to obtain smoothed image data;

[0101] Step S22: Using a bandpass filter to perform noise reduction processing on the children's EEG signal data to obtain noise-reduced signal data.

[0102] Example 4, see Figure 1 and Figure 4 This embodiment is based on the above embodiment. In step S3, the facial emotion recognition includes the following steps:

[0103] Step S31: facial feature extraction, specifically using a local GaborXOR pattern to extract facial features, wherein the local GaborXOR pattern specifically refers to a local Gabor XOR pattern for facial feature extraction;

[0104] The facial feature extraction comprises the following steps:

[0105] Step S311: selecting a smoothed image from the smoothed image data to obtain a smoothed image A;

[0106] Step S312: Select a pixel point from the smoothed image A as the center pixel to obtain the center pixel a, and add all adjacent pixels centered on the center pixel a to an adjacent pixel set B, where the adjacent pixel set B contains m adjacent pixels b;

[0107] Step S313: Calculating facial feature vectors, including the following steps:

[0108] Step S3131: Calculate the local Gabor weight, the calculation formula is:

[0109] ;

[0110] Where Wt is the local Gabor weight, i is the adjacent pixel index, m is the number of adjacent pixels, and Wb i is the weight of the i-th neighboring pixel, dis i is the distance between the ith adjacent pixel and the center pixel;

[0111] Step S3132: Calculate the facial feature vector based on the local Gabor weights for facial feature representation. The calculation formula is:

[0112] ;

[0113] Where, is the facial feature vector, is the Gabor filter scale factor, is the Gabor filter directional factor, a is the center pixel, is a facial feature subvector composed of m adjacent pixels, is a facial feature subvector composed of m-1 adjacent pixels, It is a facial feature subvector composed of 1 adjacent pixels, and binary is a binary operator;

[0114] Step S314: Repeat steps S312 and S313 to traverse all center pixels on the smoothed image A to obtain a subset of facial feature vectors;

[0115] Step S315: Repeat steps S311, S312, S313, and S314 to calculate a facial feature vector subset for each smoothed image in the smoothed image data, and merge all facial feature vector subsets to obtain a facial feature vector set;

[0116] Step S32: constructing a facial emotion recognition model, specifically using a deep convolutional neural network to construct the model, including the following steps:

[0117] Step S321: Construct a multi-label focus loss function to optimize the prediction effect. The calculation formula is:

[0118] ;

[0119] Where, is the multi-label focal loss function, ε is the balance parameter used to adjust the recall rate, is the focus parameter, which is used to adjust the model's attention to samples with different labels, zi is the target expected value, which is used to represent the true label of the target category, u is the category prediction probability, which is used to represent the prediction result of the target category, j is the emotion category index, Mi is the number of emotion categories, and u j is the prediction result of the jth emotion category, zi j is the true label of the jth emotion category;

[0120] Step S322: Construct a regression loss function to reduce the prediction error. The calculation formula is:

[0121] ;

[0122] Where, is the regression loss function, is the loss function parameter, za is the dimension prediction probability, ui is the dimension expectation value, k is the data dimension index, Ni is the number of data dimensions, za k is the prediction result of the kth data dimension, ui k is the true label of the kth data dimension;

[0123] Step S323: Construct a global loss function for optimizing model training. The calculation formula is:

[0124] ;

[0125] Where, is the global loss function, is a global loss function weight factor, which is used to adjust the relative importance of the multi-label focus loss function and the regression loss function in the global loss function;

[0126] Step S33: deep convolutional neural network model training, specifically, training the model by constructing a global loss function to obtain a facial emotion recognition model Model1;

[0127] Step S34: using the facial emotion recognition model Model1 to perform facial emotion recognition to obtain facial emotion information.

[0128] Example 5, see Figure 1 and Figure 5 This embodiment is based on the above embodiment. In step S4, the physiological emotion recognition includes the following steps:

[0129] Step S41: Extracting EEG signal features, specifically extracting features from the noise reduction signal data, including the following steps:

[0130] Step S411: Calculate the probability density function based on the noise reduction signal data. The calculation formula of the probability density function is:

[0131] ;

[0132] Where g(c) is the probability density function, c is the value of the noise reduction signal data, and e is the base of the natural exponential. is the mean of the noise reduction signal data, is the variance of the denoised signal data;

[0133] Step S412: Calculate the differential entropy of the noise reduction signal based on the probability density function to represent the characteristics of the EEG signal. The calculation formula is:

[0134] ;

[0135] Where G(C) is the differential entropy of the denoised signal;

[0136] Step S42: constructing a physiological emotion recognition model, specifically using a bidirectional long short-term memory neural network to construct the model, including the following steps:

[0137] Step S421: obtaining EEG signal features from the noise reduction signal differential entropy to obtain an EEG signal feature vector set;

[0138] Step S421: Construct a forward long short-term memory neural network unit to capture the sequential correlation of the EEG signal feature vector. The calculation formula is:

[0139] ;

[0140] Where t is the time step, is the hidden state of the forward LSTM neural network unit at the current time step, LSTM() is the LSTM neural network operation, y l,t is the lth EEG signal feature vector at the current time step, is the hidden state of the forward LSTM neural network unit at the previous time step;

[0141] Step S422: Construct a backward long short-term memory neural network unit to capture the reverse order correlation of the EEG signal feature vector. The calculation formula is:

[0142] ;

[0143] Where, is the hidden state of the backward long short-term memory neural network unit at the current time step, is the hidden state of the backward long short-term memory neural network unit at the next time step;

[0144] Step S423: Calculate the output of the bidirectional long short-term memory neural network. The calculation formula is:

[0145] ;

[0146] Where, is the output of the bidirectional long short-term memory neural network, W1 is the forward long short-term memory neural network unit weight, W2 is the backward long short-term memory neural network unit weight, is a bias vector of the bidirectional long short-term memory neural network, and the bias vector of the bidirectional long short-term memory neural network is used to adjust the deviation of the output of the bidirectional long short-term memory neural network;

[0147] Step S43: constructing a forward long short-term memory neural network unit and a backward long short-term memory neural network unit to perform model training to obtain a physiological emotion recognition model Model 2;

[0148] Step S44: using the physiological emotion recognition model Model2 to perform physiological emotion recognition and obtain physiological emotion recognition information.

[0149] Example 6, see Figure 1 and Figure 6 This embodiment is based on the above embodiment. In step S5, the emotion recognition report is generated, including the following steps:

[0150] Step S51: Constructing an emotion recognition bimodal classification model, including the following steps:

[0151] Step S511: obtaining a bimodal classification model by integrating a deep convolutional neural network and a bidirectional long short-term neural network;

[0152] Step S512: inputting the facial feature vector set and the EEG signal feature vector set into the bimodal classification model to perform model training to obtain the emotion recognition bimodal classification model Model 3;

[0153] Step S52: Multimodal emotion recognition, specifically combining the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model, and the emotion recognition bimodal classification model to perform multimodal emotion recognition, including the following steps:

[0154] Step S521: Calculate the normalization coefficient. The calculation formula is:

[0155] ;

[0156] Wherein, NC is the normalization coefficient, P is the emotion category joy, Q is the emotion category anger, R1 is the emotion category fear, R2 is the emotion category sadness, Model1(P) is the facial emotion information, specifically the prediction result of the facial emotion classification model for the emotion category joy, Model2(Q) is the physiological emotion recognition information, specifically the prediction result of the physiological emotion recognition model for the emotion category anger, Model3(R1, R2) is the prediction result of the emotion recognition bimodal classification model for the emotion categories fear and sadness;

[0157] Step S522: Calculate the probability distribution of children's emotion recognition based on the normalization coefficient. The calculation formula is:

[0158] ;

[0159] Where, is the probability distribution of children's emotion recognition, are five categories of emotions, wherein the values ​​of the five categories of emotions include joy, anger, sadness, fear and neutral;

[0160] Step S523: Select the emotion category with the largest probability value from the children's emotion recognition probability distribution to obtain children's emotion recognition information;

[0161] Step S53: generating an emotion recognition report based on the children's emotion recognition information;

[0162] By performing the above operations, we address the technical problem that emotion recognition models have limited universality due to significant differences in emotional expressions across different cultures and languages. This solution uses a bimodal emotion recognition classification model to effectively overcome cross-cultural and cross-linguistic challenges, making the emotion recognition model more universal.

[0163] In the process of emotion recognition, children's emotions are changeable and the way children express emotions is very different from that of adults. Traditional emotion recognition models find it difficult to accurately capture the emotional characteristics of children. This solution adopts a multimodal emotion recognition method that combines emotional information from different modalities. It can better capture children's emotions and improve the accuracy of emotion recognition and context understanding ability.

[0164] Example 7, see Figure 2 This embodiment is based on the above embodiment, and is an emotion recognition system for children's programming teaching, comprising: a data acquisition module, a data preprocessing module, a facial emotion recognition module, a physiological emotion recognition module and an emotion recognition report generation module;

[0165] The data acquisition module is used for data acquisition, specifically collecting data related to children's emotion recognition, obtaining emotion recognition data through acquisition, and sending the emotion recognition data to the data preprocessing module;

[0166] The data preprocessing module is used for data preprocessing, specifically preprocessing the emotion recognition data to obtain smoothed image data and noise reduction signal data, and sending the smoothed image data to the facial emotion recognition module and the noise reduction signal data to the physiological emotion recognition module;

[0167] The facial emotion recognition module is used for facial emotion recognition, specifically extracting features from the smoothed image data and using a deep convolutional neural network to perform facial emotion recognition, obtaining facial emotion recognition information, and sending the facial emotion recognition information to the emotion recognition report generation module.

[0168] The physiological emotion recognition module is used for physiological emotion recognition, specifically performing feature extraction on the noise reduction signal data and using a bidirectional long short-term memory neural network to perform physiological emotion recognition to obtain physiological emotion recognition information, and sending the physiological emotion recognition information to the emotion recognition report generation module;

[0169] The emotion recognition report generation module is used to generate emotion recognition reports. Specifically, it constructs an emotion recognition bimodal classification model to obtain an emotion recognition bimodal classification model, combines the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model and the emotion recognition bimodal classification model, performs multimodal emotion recognition, obtains children's emotion recognition information, and generates an emotion recognition report.

[0170] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0171] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0172] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. An emotion recognition method for children's programming teaching, characterized by: The method comprises the following steps: Step S1: data collection; Step S2: data preprocessing; Step S3: facial emotion recognition, specifically, using a deep convolutional neural network to build a model, optimizing model training by constructing a global loss function, obtaining a facial emotion recognition model Model1, and performing facial emotion recognition using the facial emotion recognition model Model1 to obtain facial emotion information; Step S4: Physiological emotion recognition, specifically, using a bidirectional long short-term memory neural network to build a model, constructing a forward long short-term memory neural network unit and a backward long short-term memory neural network unit to perform model training, thereby obtaining a physiological emotion recognition model Model 2, and using the physiological emotion recognition model Model 2 to perform physiological emotion recognition and obtain physiological emotion recognition information; Step S5: Emotion recognition report generation, specifically combining the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model and the emotion recognition bimodal classification model, performing multimodal emotion recognition, obtaining children's emotion recognition information, and generating an emotion recognition report.

2. The emotion recognition method for children's programming teaching according to claim 1 is characterized by: In step S3, the facial emotion recognition includes the following steps: Step S31: facial feature extraction, specifically using a local GaborXOR pattern to extract facial features, wherein the local GaborXOR pattern specifically refers to a local Gabor XOR pattern for facial feature extraction; The facial feature extraction comprises the following steps: Step S311: selecting a smoothed image from the smoothed image data to obtain a smoothed image A; Step S312: Select a pixel point from the smoothed image A as the center pixel to obtain the center pixel a, and add all adjacent pixels centered on the center pixel a to an adjacent pixel set B, where the adjacent pixel set B contains m adjacent pixels b; Step S313: Calculating facial feature vectors, including the following steps: Step S3131: Calculate the local Gabor weight, the calculation formula is: ； Where Wt is the local Gabor weight, i is the adjacent pixel index, m is the number of adjacent pixels, and Wb i is the weight of the i-th neighboring pixel, dis i is the distance between the ith adjacent pixel and the center pixel; Step S3132: Calculate the facial feature vector based on the local Gabor weights for facial feature representation. The calculation formula is: ； Where, is the facial feature vector, is the Gabor filter scale factor, is the Gabor filter directional factor, a is the center pixel, is a facial feature subvector composed of m adjacent pixels, is a facial feature subvector composed of m-1 adjacent pixels, It is a facial feature subvector composed of 1 adjacent pixels, and binary is a binary operator; Step S314: Repeat steps S312 and S313 to traverse all center pixels on the smoothed image A to obtain a subset of facial feature vectors; Step S315: Repeat steps S311, S312, S313, and S314 to calculate a facial feature vector subset for each smoothed image in the smoothed image data, and merge all facial feature vector subsets to obtain a facial feature vector set; Step S32: constructing a facial emotion recognition model, specifically using a deep convolutional neural network to construct the model, including the following steps: Step S321: Construct a multi-label focus loss function to optimize the prediction effect. The calculation formula is: ； Where, is the multi-label focal loss function, ε is the balance parameter used to adjust the recall rate, is the focus parameter, which is used to adjust the model's attention to samples with different labels, zi is the target expected value, which is used to represent the true label of the target category, u is the category prediction probability, which is used to represent the prediction result of the target category, j is the emotion category index, Mi is the number of emotion categories, and u j is the prediction result of the jth emotion category, zi j is the true label of the jth emotion category; Step S322: Construct a regression loss function to reduce the prediction error. The calculation formula is: ； Where, is the regression loss function, is the loss function parameter, za is the dimension prediction probability, ui is the dimension expectation value, k is the data dimension index, Ni is the number of data dimensions, za k is the prediction result of the kth data dimension, ui k is the true label of the kth data dimension; Step S323: Construct a global loss function for optimizing model training. The calculation formula is: ； Where, is the global loss function, is a global loss function weight factor, which is used to adjust the relative importance of the multi-label focus loss function and the regression loss function in the global loss function; Step S33: deep convolutional neural network model training, specifically, training the model by constructing a global loss function to obtain a facial emotion recognition model Model1; Step S34: using the facial emotion recognition model Model1 to perform facial emotion recognition to obtain facial emotion information.

3. The emotion recognition method for children's programming teaching according to claim 2 is characterized by: In step S4, the physiological emotion recognition includes the following steps: Step S41: Extracting EEG signal features, specifically extracting features from the noise reduction signal data, including the following steps: Step S411: Calculate the probability density function based on the noise reduction signal data. The calculation formula of the probability density function is: ； Where g(c) is the probability density function, c is the value of the noise reduction signal data, and e is the base of the natural exponential. is the mean of the noise reduction signal data, is the variance of the denoised signal data; Step S412: Calculate the differential entropy of the noise reduction signal based on the probability density function to represent the characteristics of the EEG signal. The calculation formula is: ； Where G(C) is the differential entropy of the denoised signal; Step S42: constructing a physiological emotion recognition model, specifically using a bidirectional long short-term memory neural network to construct the model, including the following steps: Step S421: obtaining EEG signal features from the noise reduction signal differential entropy to obtain an EEG signal feature vector set; Step S421: Construct a forward long short-term memory neural network unit to capture the sequential correlation of the EEG signal feature vector. The calculation formula is: ； Where t is the time step, is the hidden state of the forward LSTM neural network unit at the current time step, LSTM() is the LSTM neural network operation, y l,t is the lth EEG signal feature vector at the current time step, is the hidden state of the forward LSTM neural network unit at the previous time step; Step S422: Construct a backward long short-term memory neural network unit to capture the reverse order correlation of the EEG signal feature vector. The calculation formula is: ； Where, is the hidden state of the backward long short-term memory neural network unit at the current time step, is the hidden state of the backward long short-term memory neural network unit at the next time step; Step S423: Calculate the output of the bidirectional long short-term memory neural network. The calculation formula is: ； Where, is the output of the bidirectional long short-term memory neural network, W1 is the forward long short-term memory neural network unit weight, W2 is the backward long short-term memory neural network unit weight, is a bias vector of the bidirectional long short-term memory neural network, and the bias vector of the bidirectional long short-term memory neural network is used to adjust the deviation of the output of the bidirectional long short-term memory neural network; Step S43: constructing a forward long short-term memory neural network unit and a backward long short-term memory neural network unit to perform model training to obtain a physiological emotion recognition model Model 2; Step S44: using the physiological emotion recognition model Model2 to perform physiological emotion recognition and obtain physiological emotion recognition information.

4. The emotion recognition method for children's programming teaching according to claim 3 is characterized by: In step S5, the emotion recognition report is generated, including the following steps: Step S51: Constructing an emotion recognition bimodal classification model, including the following steps: Step S511: obtaining a bimodal classification model by integrating a deep convolutional neural network and a bidirectional long short-term neural network; Step S512: inputting the facial feature vector set and the EEG signal feature vector set into the bimodal classification model to perform model training to obtain the emotion recognition bimodal classification model Model 3; Step S52: multimodal emotion recognition; Step S53: Generate an emotion recognition report based on the children's emotion recognition information.

5. The emotion recognition method for children's programming teaching according to claim 4 is characterized by: In step S52, the multimodal emotion recognition is specifically performed by combining the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model, and the emotion recognition bimodal classification model, and includes the following steps: Step S521: Calculate the normalization coefficient. The calculation formula is: ； Wherein, NC is the normalization coefficient, P is the emotion category joy, Q is the emotion category anger, R1 is the emotion category fear, R2 is the emotion category sadness, Model1(P) is the facial emotion information, specifically the prediction result of the facial emotion classification model for the emotion category joy, Model2(Q) is the physiological emotion recognition information, specifically the prediction result of the physiological emotion recognition model for the emotion category anger, Model3(R1, R2) is the prediction result of the emotion recognition bimodal classification model for the emotion categories fear and sadness; Step S522: Calculate the probability distribution of children's emotion recognition based on the normalization coefficient. The calculation formula is: ； Where, is the probability distribution of children's emotion recognition, are five categories of emotions, wherein the values ​​of the five categories of emotions include joy, anger, sadness, fear and neutral; Step S523: Select the emotion category with the largest probability value from the children's emotion recognition probability distribution to obtain children's emotion recognition information.

6. The emotion recognition method for children's programming teaching according to claim 5, characterized in that: In step S1, the data collection specifically involves acquiring emotion recognition data, which includes facial image data, children's EEG signal data, and emotion labels, and the emotion labels include joy, anger, sadness, fear, and neutrality.

7. The emotion recognition method for children's programming teaching according to claim 6, characterized in that: In step S2, the data preprocessing includes the following steps: Step S21: normalizing, enhancing, denoising and detecting faces on facial image data to obtain smoothed image data; Step S22: Using a bandpass filter to perform noise reduction processing on the children's EEG signal data to obtain noise-reduced signal data.

8. An emotion recognition system for children's programming teaching, used to implement the emotion recognition method for children's programming teaching as described in any one of claims 1 to 7, characterized in that: include: It includes data acquisition module, data preprocessing module, facial emotion recognition module, physiological emotion recognition module and emotion recognition report generation module.

9. The emotion recognition system for children's programming teaching according to claim 8, characterized in that: The data acquisition module is used for data acquisition, specifically collecting data related to children's emotion recognition, obtaining emotion recognition data through acquisition, and sending the emotion recognition data to the data preprocessing module; The data preprocessing module is used for data preprocessing, specifically preprocessing the emotion recognition data to obtain smoothed image data and noise reduction signal data, and sending the smoothed image data to the facial emotion recognition module and the noise reduction signal data to the physiological emotion recognition module; The facial emotion recognition module is used for facial emotion recognition, specifically extracting features from the smoothed image data and using a deep convolutional neural network to perform facial emotion recognition, obtaining facial emotion recognition information, and sending the facial emotion recognition information to the emotion recognition report generation module. The physiological emotion recognition module is used for physiological emotion recognition, specifically performing feature extraction on the noise reduction signal data and using a bidirectional long short-term memory neural network to perform physiological emotion recognition to obtain physiological emotion recognition information, and sending the physiological emotion recognition information to the emotion recognition report generation module; The emotion recognition report generation module is used to generate emotion recognition reports. Specifically, it constructs an emotion recognition bimodal classification model to obtain an emotion recognition bimodal classification model, combines the emotion prediction results of the facial emotion recognition model, the physiological emotion recognition model and the emotion recognition bimodal classification model, performs multimodal emotion recognition, obtains children's emotion recognition information, and generates an emotion recognition report.

Images