Children's emotion detection method based on improved Transformer

Abstract

The present invention provides a method for detecting children's emotions based on an improved Transformer, which relates to the field of emotion recognition, and includes: constructing a data set; establishing an improved Transformer model, wherein the improved Transformer model includes an improved encoder and an improved decoder, the improved encoder includes at least a cross convolution layer, and the improved decoder includes at least an encoder-decoder attention mechanism; training and optimizing the improved Transformer model through the data set; obtaining a facial image of a child to be detected; and identifying the child's emotions based on the facial image of the child to be detected through the optimized improved Transformer model, which has the advantage of more accurate and reliable child emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition, and in particular to a method for detecting emotions of infants based on an improved Transformer. Background Art

[0002] Children's emotions are an important indicator of their mental health and development, reflecting their mental state and social skills. However, there are certain challenges and limitations in the application of existing emotion recognition technologies in young children. Traditional emotion recognition methods usually rely on manually designed features, which have weak generalization capabilities in different situations and are difficult to cope with the complex and changing facial expressions of young children. In addition, young children's facial expressions are often small and change rapidly, and traditional methods are difficult to accurately capture these subtle differences.

[0003] In recent years, deep learning technology has made significant progress in the fields of image processing and emotion recognition. Although deep learning methods can automatically learn feature representations and perform well in adult emotion recognition, there are still some problems in the recognition of children's facial expressions: 1. Although the method based on deep convolutional neural network has the ability to automatically learn features, it lacks the modeling of long-range dependencies and is difficult to fully mine the global information of children's facial expressions; 2. Existing deep convolutional neural network models usually use a single feature extractor and cannot capture local and global features at the same time, which affects the ability to recognize subtle changes in children's facial expressions.

[0004] Therefore, it is necessary to provide a method for detecting children's emotions based on an improved Transformer to achieve more accurate and reliable children's emotion recognition. Summary of the invention

[0005] The present invention provides a method for detecting emotions of infants based on an improved Transformer, comprising: constructing a data set; establishing an improved Transformer model, wherein the improved Transformer model comprises an improved encoder and an improved decoder, the improved encoder comprises at least a cross convolution layer, and the improved decoder comprises at least an encoder-decoder attention mechanism; training and optimizing the improved Transformer model through the data set; acquiring a facial image of an infant to be detected; and identifying the emotions of the infant based on the facial image of the infant to be detected through the optimized improved Transformer model.

[0006] Furthermore, the improved encoder includes at least an encoder input layer, an encoder cross convolution layer, and a stacked basic encoder unit, wherein the basic encoder unit includes an encoder multi-head self-attention mechanism layer, an encoder feedforward neural network, an encoder residual connection, and encoder layer normalization.

[0007] Furthermore, the cross convolution layer includes a plurality of row-separable one-dimensional convolution kernels and a plurality of column-separable one-dimensional convolution kernels.

[0008] Furthermore, the convolution operation formula of the cross convolution layer is: Where F′(i,j) is the (i,j)th position of the output feature map, D(i,j) is the input image data, D(im,jn) is the local image of the im row and jn column of the input image data, and w m (i) is the row-wise separable one-dimensional convolution kernel acting on the i-th row, w n (j) is a column-wise separable one-dimensional convolution kernel acting on the j-th column, M is the total number of rows of the input image data, and N is the total number of columns of the input image data.

[0009] Furthermore, the improved decoder includes at least a decoder input layer, a stacked basic decoder unit and a decoder output layer, wherein the basic decoder unit includes a decoder multi-head self-attention mechanism layer, an encoder-decoder attention mechanism layer, a decoder feedforward neural network, a decoder residual connection and a decoder layer normalization.

[0010] Furthermore, the encoder-decoder attention mechanism includes: calculating the correlation weight between the decoder input and each position feature in the infant facial image features output by the encoder, wherein the decoder input is an enhanced sequence representation obtained by processing the current input sequence through the decoder multi-head self-attention mechanism layer; based on the correlation weight, performing weighted summation on the infant facial image features to generate a new feature representation.

[0011] Furthermore, the correlation weight between the decoder input and each position feature in the infant facial image feature output by the encoder is calculated by the following formula: Among them, α is the correlation weight, W q is the linear transformation matrix of the query sequence, is the linear transformation matrix of the key sequence, d k is the attention dimension, V is the facial image feature of the child, and H is the enhanced sequence representation.

[0012] Furthermore, a new feature representation is generated by the following formula: Among them, Z is the new feature representation, M is the sequence length of the value sequence, and W v is the linear transformation matrix of the facial image features of the child, (V×W v ) j Represented by V×W v Get the value of the jth element in the vector.

[0013] Furthermore, the constructing of the data set includes: collecting original facial expression images of young children; preprocessing the original facial expression images of young children to obtain preprocessed facial expression images of young children, wherein the preprocessing at least includes image denoising, image enhancement, image cropping and image scaling; assigning emotion category labels to the preprocessed facial expression images of young children; and constructing a training set, a validation set and a test set.

[0014] Furthermore, the improved Transformer model is trained and optimized through the data set, including: repeatedly importing the training set into the improved Transformer model in batches, obtaining the prediction results of the improved Transformer model, calculating the prediction loss based on the prediction results of the improved Transformer model according to the loss function, and updating the parameters of the improved Transformer model based on the prediction loss using an optimization algorithm until the preset training conditions are met.

[0015] Compared with the prior art, the method for detecting children's emotions based on the improved Transformer provided by the present invention has at least the following beneficial effects:

[0016] 1. The separable one-dimensional convolution kernel in the cross convolution layer can adaptively capture the local features of different scales and shapes of children's facial expressions, avoiding the limitations brought by fixed convolution kernels, thereby significantly improving the feature extraction capability. At the same time, it reduces the number of parameters and calculations, and reduces the complexity of the model.

[0017] 2. The encoder combines the advantages of convolutional neural networks and self-attention mechanisms, and can simultaneously extract local features and model global dependencies, providing high-quality feature input for emotion detection tasks and improving the expressiveness of the model;

[0018] 3. The encoder-decoder attention mechanism enables the decoder to obtain the local information contained in the current input and mine the most relevant global image features, better understand the semantics of the input in the overall context, provide rich decoding representations for emotion prediction, and improve the accuracy of prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0020] Figure 1 is a flowchart of a method for detecting emotions of young children based on an improved Transformer according to some embodiments of this specification;

[0021] Figure 2 is a schematic diagram of the structure of an improved encoder according to some embodiments of this specification;

[0022] Figure 3 It is a schematic diagram of the structure of an improved decoder according to some embodiments of this specification. DETAILED DESCRIPTION

[0023] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0024] Figure 1 is a flow chart of a method for detecting emotions of young children based on an improved Transformer according to some embodiments of this specification, such as Figure 1 As shown, the method for detecting emotions of young children based on the improved Transformer may include the following steps.

[0025] Step 110, constructing a data set.

[0026] Specifically include:

[0027] Collect original facial expression images of children;

[0028] Preprocessing the original infant facial expression image to obtain a preprocessed infant facial expression image, wherein the preprocessing at least includes image denoising, image enhancement, image cropping and image scaling;

[0029] Assign emotion category labels to the pre-processed facial expression images of young children;

[0030] Construct training, validation, and test sets.

[0031] Specifically, in order to obtain authentic and reliable original facial expression images of children, the shooting environment is controlled and facial expression videos of children are collected. As an example only, by controlling the shooting environment, a high-definition camera with a resolution of 1080P is used to collect facial expressions of children at a video acquisition rate of 30fps. By controlling the lighting conditions and background environment, it is ensured that clear and noise-free original facial expression images of children under various emotions are collected, including original facial expression images of children in emotional states such as calm, happy, angry, sad and surprised. The resolution of the collected facial expression images is 1920×1080, and they are stored in PNG (Portable Network Graphics) lossless compression format. The size of a single image file ranges from about 500KB to 1MB. The video sampling rate is 30fps, which means that 30 frames of children's facial expression images need to be stored per second.

[0032] The Gaussian filtering algorithm is used to remove the noise in the original children's facial expression image to obtain the denoised children's facial expression image.

[0033] In Gaussian filtering, the filter kernel size is (2k+1)×(2k+1), and the mathematical expression of the Gaussian kernel G(x, y) is:

[0034]

[0035] Among them, m, n are the row and column indices of the filter kernel coordinates, x, y are the relative positions of the current pixel relative to the kernel, and σ is the standard deviation of the Gaussian kernel, which controls the smoothness of the filter.

[0036] In order to expand the diversity of the dataset and increase the robustness and generalization ability of the model, linear interpolation is performed on the denoised children's facial expression images to generate new synthetic images.

[0037] Just as an example, any two denoised children's facial expression images are linearly interpolated and mixed to generate a new synthetic children's facial expression image:

[0038] X=λX′ i +(1-λ)X′ j

[0039] Among them, X is the synthetic facial expression image of the child, X′ i is the i-th denoised facial expression image of a child, X′ j is the jth denoised child facial expression image, and λ is the mixing weight. The value range of λ is [0,1], and its function is to control the weight ratio of the two images in the composite image. When λ=0, the composite child facial expression image X is equal to X′ j, that is, only the jth denoised child facial expression image is retained. When λ = 1, the synthesized child facial expression image X is equal to X′ i , that is, only the i-th denoised child facial expression image is retained. When 0<λ<1, the composite image X is the weighted average of the two denoised child facial expression images. Different values ​​of λ will cause the composite image to transition between the two denoised child facial expression images. For example, if X′ i Contains the happy expression of a child, X' j If the expression of a child is included, then when λ gradually increases from 0 to 1, the synthesized child's facial expression image will gradually transition from sadness to happiness. This linear interpolation method can add samples of intermediate transition states to the dataset, thereby improving the improved Transformer model's ability to model continuous facial expression changes.

[0040] In order to ensure the consistency and stability of the input model, the denoised children’s facial expression images and the synthetic children’s facial expression images were cropped to a uniform size of 224×224, and the pixel values ​​were normalized to the [0, 1] interval to obtain the final preprocessed children’s facial expression images to ensure the consistency of the input.

[0041] In order to assign emotion category labels to the preprocessed facial image data of young children and prepare for the training of supervised learning models, data labeling operations are required. Five labels are defined based on the emotion type of facial expressions:

[0042] {"calm", "happy", "angry", "sad", "surprised"}, respectively, using one-hot vectors

[0043] For example, we can invite experts in child psychology and behavior analysis to analyze the preprocessed image data D.

[0044] Manual annotation is performed to generate a dataset with emotion labels.

[0045] The dataset can be divided into training set, validation set and test set according to a certain ratio. As an example, the labeled dataset is divided into training set, validation set and test set in a ratio of 8:1:1. This division can ensure that there is sufficient data for parameter adjustment and optimization during model training, and after model training, the training set, validation set and test set are optimized.

[0046] Evaluation of the validation set and final performance verification of the test set are performed to ensure the generalization ability and effectiveness of the model.

[0047] Step 120, establishing an improved Transformer model.

[0048] Among them, improving the Transformer model includes improving the encoder and improving the decoder.

[0049] The improved decoder includes at least the encoder-decoder attention mechanism.

[0050] Figure 2 is a schematic diagram of the structure of an improved encoder according to some embodiments of this specification, such as Figure 2

[0051] As shown, in some embodiments, the improved encoder includes at least an encoder input layer, an encoder cross convolution layer, and a stacked basic encoder unit, wherein the basic encoder unit includes an encoder multi-head self-attention mechanism layer, an encoder feedforward neural network, an encoder residual connection, and an encoder layer normalization. The multi-layer stacked structure can improve the depth of feature extraction and modeling and enhance the expressiveness of the model. The output of each basic encoder unit will be used as the input of the next basic encoder unit to process and enhance the input features layer by layer.

[0052] The final output of the encoder is the facial image feature of the child as V.

[0053] Specifically, the input size of the input layer is (224, 224, 3), where 224 and 224 represent

[0054] The height and width of the child's facial expression image. 3 indicates the number of channels of the child's facial expression image.

[0055] Existing convolutional neural networks use convolution kernels of fixed shape and size to perform convolution operations on the input image to extract local features. However, this approach has certain limitations. The selection of convolution kernel size usually needs to be determined manually, and the fixed shape cannot fully adapt to different feature distributions. Another approach is to use convolution kernels of multiple scales, but this requires additional parameters and increases model complexity. In order to improve feature extraction capabilities and reduce model complexity, the improved encoder uses a cross convolution layer that uses separable convolution kernels.

[0056] By performing convolution operations on the input image, features of different scales and shapes can be adaptively extracted.

[0057] In some embodiments, the cross convolution layer includes a plurality of row-wise separable one-dimensional convolution kernels and a plurality of column-wise separable one-dimensional convolution kernels, and the convolution operation formula is:

[0058]

[0059] Where F′(i,j) is the (i,j)th position of the output feature map, D(i,j) is the input image data, D(im,jn) is the local image of the im row and jn column of the input image data, and w m (i) is the row-wise separable one-dimensional convolution kernel acting on the i-th row, w n (j) is a column-wise separable one-dimensional convolution kernel acting on the j-th column, M is the total number of rows of the input image data, and N is the total number of columns of the input image data.

[0060] In the detection of facial expressions of young children, the separable convolution kernel design has significant advantages. First, by learning the parameters of the one-dimensional convolution kernel in the row and column directions respectively, this design can adaptively capture the local features of different scales and shapes of the children's facial expressions, avoiding the limitations brought by the fixed convolution kernel. Second, compared with the standard two-dimensional convolution kernel, the separable convolution kernel only needs to learn m+n one-dimensional kernels instead of m×n two-dimensional kernels, which greatly reduces the number of parameters and reduces the complexity of the model. In addition, due to the reduction in parameters, the computational complexity of the separable convolution operation is also reduced accordingly, which improves the computational efficiency of the model.

[0061] In this cross-convolution layer, the input facial expression image of the child will be convolved through multiple groups of different separable convolution kernels. Since the parameters of each group of convolution kernels are adaptively learned, different local features of the child's facial expression can be automatically captured, including different scales, shapes, and textures. This cross-convolution mechanism significantly improves the feature extraction capability and makes up for the shortcomings of traditional convolutional neural networks in facial expression detection. Finally, the feature maps extracted by different groups of convolution kernels will be spliced ​​together as the output of the cross-convolution layer, providing diversified feature representations for subsequent self-attention mechanism layers and other layers.

[0062] The core idea of ​​the multi-head self-attention mechanism layer is to calculate the correlation weight between the current position feature and all other position features, and perform weighted summation of all position features to obtain the enhanced current position feature representation. The first step is to perform self-attention calculation:

[0063]

[0064] Among them, Q, K, V are query sequence, key sequence and value sequence respectively, softmax(·) is the activation function, d k is the dimension of the attention head. The calculation result of the i-th attention head is expressed as:

[0065] head i =Attention(FQW i Q ,FKW iK ,FVW i V )

[0066] Among them, W i Q ,W i K ,W i V are the linear transformation weight matrices of query, key, and value respectively. Connect h attention heads to get a multi-head attention mechanism:

[0067] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W o

[0068] Among them, MultiHead(Q,K,V) is a multi-head attention mechanism, Concat() is a connection operation, and W o It is the linear transformation weight matrix output by the multi-head attention mechanism.

[0069] The feedforward neural network is used to perform deeper feature extraction and transformation on the output of the multi-head attention, so that the improved encoder can better capture the complex patterns and semantic information in the input sequence. The feedforward neural network consists of two linear layers and a ReLU activation function:

[0070] FNN=ReLU[max(0,MultiHead(Q,K,V)W1+b1)W2+b2]

[0071] Among them, FNN is a feedforward neural network, W1, W2 are weight matrices, b1, b2 are bias vectors. The ReLU activation function can provide nonlinear mapping while maintaining linearity when the value is positive, thus avoiding the gradient vanishing problem.

[0072] Residual connections help keep the underlying features available in the upper layers, and also facilitate the transfer of gradients in deep networks, thereby alleviating the vanishing gradient problem. Layer normalization can speed up model convergence and improve training stability. Specifically, the output of the feedforward network layer is normalized, thereby alleviating the vanishing gradient problem and improving the training stability of the model:

[0073]

[0074] Among them, LN is the normalized output, μ and σ are the mean and standard deviation of the output of the feedforward network FNN calculated on a single sample, and α and β are learnable scaling and translation parameters.

[0075] Finally, the normalized output is residually connected to the input of the multi-head self-attention mechanism layer, which helps the model capture local and global features and accelerates the training process. The input of the multi-head self-attention mechanism layer contains the original feature information, and passing it directly to the subsequent layers can ensure that the underlying features will not completely disappear in deep calculations.

[0076] The improved encoder design combines the advantages of convolutional neural networks and Transformer self-attention mechanism. The cross-convolution layer can extract multi-scale local features, and the self-attention mechanism can model the long-range dependencies between features at different positions; the feedforward network adds nonlinear mapping capabilities to the features; the residual connection and layer normalization operations can accelerate convergence and alleviate gradient problems; the stacking of multiple layers of basic encoder units gives the model enough depth to learn complex facial expression feature representations. This design architecture combines local and global feature extraction capabilities to provide high-quality feature input for the task of detecting emotions in young children.

[0077] Figure 3 is a schematic diagram of the structure of an improved decoder according to some embodiments of this specification, such as Figure 3 As shown, in some embodiments, the improved decoder includes at least a decoder input layer, a stacked basic decoder unit and a decoder output layer, wherein the basic decoder unit includes a decoder multi-head self-attention mechanism layer, an encoder-decoder attention mechanism layer, a decoder feedforward neural network, a decoder residual connection and a decoder layer normalization. The output of each basic decoder unit will be used as the input of the next basic decoder unit to process and enhance the feature representation layer by layer. Multi-layer stacked basic decoder units can improve the expressive power and modeling depth of the decoder, and the final output feature of the decoder is recorded as E.

[0078] Specifically, in the improved decoder, the initial input is a special start mark <start>, which indicates the starting point of the prediction sequence of the improved decoder. During training, the model uses the actual label as the subsequent input sequence and updates the parameters through supervised learning; during inference, the model uses the output of the previous step of the improved decoder as the input of the current step. This autoregressive method can make continuous predictions without obtaining the entire output sequence in advance.

[0079] The decoder multi-head self-attention mechanism layer uses a multi-head self-attention mechanism to process the current input sequence, capture the dependencies between elements in the sequence, and obtain the enhanced sequence representation H. The formula is as follows:

[0080] H=MultiHead(Q,K,V)

[0081] Among them, Q, K, and V are query, key, and value sequences respectively, all of which are linearly transformed from the current input sequence.

[0082] In some embodiments, the encoder-decoder attention mechanism includes:

[0083] Calculate the correlation weight between the decoder input and each position feature in the infant facial image features output by the encoder, where the decoder input is the enhanced sequence representation obtained by processing the current input sequence through the decoder multi-head self-attention mechanism layer;

[0084] Based on the relevance weights, the facial image features of the children are weighted and summed to generate a new feature representation that can simultaneously integrate the contextual information of the enhanced sequence representation H and the most relevant global image features.

[0085] In some embodiments, the enhanced sequence representation H is used as the query vector, the facial image feature V of the child extracted by the improved encoder is used as the key and value vectors, and the j-th position H in H is calculated by the scaled dot product attention mechanism. j The correlation weight α between the feature at each position in V j :

[0086]

[0087] Among them, α j is the relevance weight of the jth position, W q is the linear transformation matrix of the query sequence, is the linear transformation matrix of the key sequence, d k is the attention dimension, which is used to scale the dot product to ensure numerical stability. V is the facial image feature of the child. The softmax function normalizes the dot product result to the interval [0,1].

[0088] Using α as the weight, we perform weighted summation on the facial image features V of the child, and obtain a new feature representation Z that combines the global image information and the current input context:

[0089]

[0090] Among them, Z is the new feature representation, M is the sequence length of the value sequence, and W v is the linear transformation matrix of the facial image features of the child, (V×W v ) j Represented by V×W v Get the value of the jth element in the vector.

[0091] Through this attention fusion mechanism, the improved decoder can not only obtain the local information contained in the fused and enhanced sequence representation H, but also mine the most relevant global image features, so as to better understand the semantics of the fused and enhanced sequence representation in the overall image context and provide a richer decoding representation for the final emotion prediction.

[0092] The output of the encoder-decoder attention mechanism is further processed using a feedforward network layer, allowing the model to better capture complex patterns and semantic information in the input sequence. The feedforward network consists of two linear layers and a ReLU activation function:

[0093] FNN=ReLU[max(0,ZW1+b1)W2+b2]

[0094] Among them, W1, W2 are weight matrices, b1, b2 are bias vectors. This feedforward network adds nonlinear mapping capabilities to the decoder, enabling it to better capture high-order patterns and semantic information of the input.

[0095] Introduce residual connection and layer normalization operations. Perform layer normalization on the output of the feedforward network layer:

[0096]

[0097] Among them, μ and σ are the mean and standard deviation of the output of the feedforward network FNN calculated on a single sample, and α and β are learnable scaling and translation parameters.

[0098] Finally, the normalized output is residually connected to the input of the multi-head self-attention mechanism layer, which helps the model capture local and global features and accelerates the training process, ultimately obtaining the residually connected and layer-normalized output.

[0099] The decoder output layer of the improved decoder finally outputs a K-dimensional vector, which is mapped to the probability distribution of emotion classification through the fully connected layer, where K is the total number of emotion categories:

[0100] P(y|D)=softmax(W o E+b o )

[0101] Among them, W o and b o are the weights and biases of the fully connected layer. The softmax activation function maps the output to the range of [0,1] to obtain the probability value of each emotion category. The category with the largest value is the final emotion prediction result.

[0102] In general, the improved decoder models the current input through the self-attention mechanism, fuses global information through interaction with the improved encoder, extracts high-level features through feedforward networks and residual connections, and finally outputs the probability distribution of emotion classification. This design combines the advantages of self-attention and cross convolution, can efficiently model images and sequences, and provides powerful feature representation capabilities for the task of early childhood emotion detection.

[0103] Step 130, training and optimizing the Transformer model through the data set.

[0104] Specifically include:

[0105] Repeat the process of importing the training set into the improved Transformer model in batches to obtain the prediction results of the improved Transformer model. According to the loss function, the classification probability result P predicted by the Transformer model is compared with the label L of the child's facial expression image, the prediction loss is calculated, and the optimization algorithm is used to update the parameters of the improved Transformer model based on the prediction loss until the preset training conditions are met.

[0106] The loss function is defined as:

[0107]

[0108] Among them, loss is the loss, P i is the classification probability result of the i-th child’s facial expression image predicted by the Transformer model, L i is the label of the i-th child’s facial expression image, and N is the training batch size.

[0109] The Adam optimization algorithm is used to perform gradient updates on the encoder / decoder parameters according to the loss function:

[0110]

[0111] Among them, θ represents the parameters to be updated, including all weight matrices and bias vectors in the improved Transformer infant emotion detection model. is the learning rate, which controls the step size of each parameter update. Represents the gradient of the loss function loss to the parameter θ, which represents the rate of change of the loss function to the parameter, by multiplying it by the learning rate The parameter θ can be updated according to the direction and size of the gradient, so that the loss function is gradually reduced and the model can better adapt to the training data. The updated parameter θ′ will be used in the next iteration of the training process.

[0112] The gradient of each layer parameter of the improved Transformer model is counted, and the learning rate of the layer is dynamically adjusted according to the statistical value of the gradient of the layer. The specific process is as follows:

[0113] 1. Initialization: Set a base learning rate for the entire model And a learning rate adjustment parameter γ (0<γ<1). Each layer parameter is initialized to the basic learning rate

[0114] 2. Training process: Before each model weight update, perform the following operations on each layer parameter of the model:

[0115] a) Calculate the sum of squared gradients of all weight parameters of this layer as the statistical value of the gradient of this layer:

[0116]

[0117] Where n is the number of parameters in this layer, is the gradient of the i-th parameter to the loss function loss.

[0118] b) Calculate the L2 norm of this layer:

[0119]

[0120] c) Calculate the new learning rate of this layer based on the gradient statistics g and parameter norm ω

[0121]

[0122] d) Yes Crop it to ensure it is within a certain range and prevent it from being too large or too small:

[0123]

[0124] in, and are the pre-set minimum and maximum learning rates, Used to Restricted to and If The value is less than Set it to if The value is greater than Set it to if The value of and In between, it remains unchanged.

[0125] e) Use the new learning rate for this layer Update the parameters of this layer:

[0126]

[0127] 3. When N consecutive batches fail to reduce the loss, the training process is terminated.

[0128] The basic principle of the algorithm is: if the gradient statistics of a layer's parameters are large, it means that the parameters of this layer have a greater impact on the loss function, and learning needs to be accelerated, so a higher learning rate is assigned; conversely, if the gradient statistics of a layer are small, it means that the parameters of this layer have a smaller impact on the loss function, and the learning rate can be appropriately slowed down to protect the features obtained by this layer from being destroyed, so a lower learning rate is assigned.

[0129] The γ parameter can be used to control the magnitude of the learning rate adjustment. The smaller γ is, the more aggressive the learning rate adjustment is. and The setting can prevent instability caused by too large or too small learning rate.

[0130] This algorithm implements the need to assign different learning rates to different layers of the improved Transformer, which helps to improve the stability and convergence of model training while protecting the basic features obtained at the bottom layer from being destroyed.

[0131] Use the validation set and test set to verify the trained improved Transformer model. Save the final model parameters at the end of training as the deployment model for toddler facial emotion detection. Deploy the trained improved Transformer model as a toddler emotion detection model to a server or cloud platform for other systems or applications to call, and realize the online service of toddler emotion detection.

[0132] Step 140, obtaining a facial image of a child to be detected.

[0133] Specifically, the facial expression video of the child can be obtained from the video acquisition device, using a high-definition camera with a resolution of 1080P and a video acquisition rate of 30fps, and shooting in an environment with controlled light and background interference to ensure that clear and noise-free facial expression image data of the child is collected.

[0134] The Gaussian filtering algorithm is used to remove noise from the children's facial expression image data to obtain denoised children's facial expression image data. The denoised children's facial expression image data is cropped to a uniform size of 224×224, and the pixel values ​​are normalized to the [0,1] interval to obtain the final input image.

[0135] Step 150, identifying the child's emotions based on the facial image of the child to be detected by using the optimized improved Transformer model.

[0136] The preprocessed facial images of infants are input into the trained improved Transformer model for inference. The improved Transformer model will output the corresponding emotion category prediction results, including five emotional states: {"calm", "happy", "angry", "sad", "surprised"}.

[0137] According to the emotion prediction results of the improved Transformer model, the current emotional state of the child is judged and analyzed, and the corresponding emotion category results are output. For example, if the model prediction result is [0,1,0,0,0], the "happy" emotion is output.

[0138] According to the results of the children's emotion detection, corresponding follow-up education and psychological intervention operations can be carried out. For example, for children who are detected to be "angry", appropriate emotional counseling measures can be taken; for children who are detected to be "sad", appropriate care and support can be given.

[0139] Through the above steps, the trained child emotion detection model can be used to effectively detect children's facial expressions, obtain the current emotional state of the children, and provide important reference for subsequent educational counseling and psychological intervention.

[0140] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.< / start>

Claims

1. The method for detecting children's emotions based on improved Transformer is characterized by: include: Build a dataset; Establishing an improved Transformer model, wherein the improved Transformer model includes an improved encoder and an improved decoder, the improved encoder includes at least a cross convolution layer, and the improved decoder includes at least an encoder-decoder attention mechanism; Training and optimizing the improved Transformer model through the data set; Obtain the facial image of the child to be detected; Recognize the emotion of the child based on the facial image of the child to be detected by using the optimized improved Transformer model; The improved encoder comprises at least an encoder input layer, a cross convolution layer, and a stacked basic encoder unit, wherein the basic encoder unit comprises an encoder multi-head self-attention mechanism layer, an encoder feedforward neural network, an encoder residual connection, and an encoder layer normalization; The cross convolution layer includes a plurality of row-direction separable one-dimensional convolution kernels and a plurality of column-direction separable one-dimensional convolution kernels; The convolution operation formula of the cross convolution layer is: Where F′(i,j) is the (i,j)th position of the output feature map, D(i,j) is the input image data, D(im,jn) is the local image of the im row and jn column of the input image data, and w m (i) is the row-wise separable one-dimensional convolution kernel acting on the i-th row, w n (j) is a column-wise separable one-dimensional convolution kernel acting on the j-th column, M is the total number of rows of the input image data, and N is the total number of columns of the input image data; The improved decoder comprises at least a decoder input layer, a stacked basic decoder unit and a decoder output layer, wherein the basic decoder unit comprises a decoder multi-head self-attention mechanism layer, an encoder-decoder attention mechanism layer, a decoder feedforward neural network, a decoder residual connection and a decoder layer normalization; The encoder-decoder attention mechanism includes: Calculating the correlation weight between the input of the decoder and each position feature in the infant facial image features output by the encoder, wherein the input of the decoder is an enhanced sequence representation obtained by processing the current input sequence through the multi-head self-attention mechanism layer of the decoder; Based on the correlation weights, weighted summation is performed on the features of the infant's facial image to generate a new feature representation.

2. The method for detecting children's emotions based on improved Transformer according to claim 1, characterized in that: The correlation weight between the decoder input and each position feature in the encoder output child facial image features is calculated by the following formula: Among them, α j is the relevance weight of the jth position, W q is the linear transformation matrix of the query sequence, is the linear transformation matrix of the key sequence, d k is the attention dimension, and V is the facial image feature of the child.

3. The method for detecting children's emotions based on improved Transformer according to claim 2, characterized in that: The new feature representation is generated by the following formula: Among them, Z is the new feature representation, M is the sequence length of the value sequence, and W v is the linear transformation matrix of the facial image features of the child, (V×W v ) j Represented by V×W v The value of the jth element in the obtained vector.

4. The method for detecting children's emotions based on an improved Transformer according to any one of claims 1 to 3, characterized in that: The constructing of the data set comprises: Collect original facial expression images of children; Preprocessing the original child facial expression image to obtain a preprocessed child facial expression image, wherein the preprocessing at least includes image denoising, image enhancement, image cropping and image scaling; Assign emotion category labels to the pre-processed facial expression images of young children; Construct training, validation, and test sets.

5. The method for detecting children's emotions based on improved Transformer according to claim 4, characterized in that: Training and optimizing the improved Transformer model through the data set includes: Repeat the process of importing the training set into the improved Transformer model in batches, obtaining the prediction results of the improved Transformer model, calculating the prediction loss based on the prediction results of the improved Transformer model according to the loss function, and updating the parameters of the improved Transformer model based on the prediction loss using an optimization algorithm until the preset training conditions are met.

Images