Face emotion recognition network model training method and device, equipment and medium

Abstract

The present application provides a kind of face emotion recognition network model training method, device, equipment and medium, the face emotion recognition network model includes main network, feature fusion module, detection module and emotion classification module, method includes: obtaining data set;Training image in data set is input to main network, and first feature image is obtained;First feature image is input to feature fusion module and is carried out feature fusion, and second feature image is obtained;Second feature image is input to detection module and is carried out multiple different scale face detection, and face feature image is obtained;Face feature image is input to emotion classification module, and model prediction result is obtained;According to model prediction result, face emotion recognition network model is optimized, and the face emotion recognition network model after optimization is obtained.The model trained by the present application can directly classify and regress the extracted features to obtain face emotion classification, with high recognition accuracy and improved processing efficiency.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of artificial intelligence, and particularly relates to a face emotion recognition network model training method and device, equipment and medium. BACKGROUND

[0002] The emotion recognition method based on the convolutional neural network is relatively mainstream, and a supervised scoring estimator (SSE) based on a basic framework of a convolutional neural network (CNN) is proposed; or models can be combined in sequence to form a cascade network. In reality, face emotions coexist with various factors, such as head posture and illumination, in order to solve this problem, multi-task learning is introduced, such as introducing face key point positioning and face action unit set (AUS) detection; a generative adversarial network (GAN) is also applied to face emotion recognition, but such a model is relatively complex to build.

[0003] However, such a network that integrates multiple relatively small networks in parallel or in series significantly increases the computing cost and storage requirement. In addition, the weights of each sub-network are usually learned according to the performance of the original training data, leading to overfitting to new emerging invisible test data. The disadvantage of multi-task network combination is that it requires labeled data from all tasks, and as the number of tasks involved increases, training becomes more and more troublesome. SUMMARY

[0004] The embodiment of the present application provides a face emotion recognition network model training method, device, equipment and medium to solve the problems of complex construction of the existing face emotion recognition network model, low recognition accuracy and low processing efficiency.

[0005] To solve the above technical problems, the present application is implemented as follows:

[0006] In a first aspect, the embodiment of the present application provides a face emotion recognition network model training method, the face emotion recognition network model comprising a backbone network, a feature fusion module, a detection module and an emotion classification module, and the method comprising:

[0007] obtaining a data set;

[0008] inputting a training image in the data set into the backbone network to obtain a plurality of first feature images of different scales about the training image;

[0009] inputting the plurality of second feature images of different scales into a detection module for multi-scale face detection to obtain a face feature image;

[0010] inputting the plurality of second feature images of different scales into a detection module for multi-scale face detection to obtain a face feature image;

[0011] inputting the plurality of second feature images of different scales into a detection module for multi-scale face detection to obtain a face feature image;

[0012] According to the model prediction result, the face emotion recognition network model is optimized to obtain an optimized face emotion recognition network model.

[0013] Optionally, the obtaining of the data set comprises:

[0014] obtaining an initial data set, wherein the initial data set comprises a plurality of training images;

[0015] generating a grid mask on at least part of the training images, wherein the grid mask comprises a horizontal continuous mask and a vertical continuous mask;

[0016] randomly rotating the grid mask by the same angle in each dimension of an RGB channel to obtain an expanded training image;

[0017] obtaining the data set according to the training images in the initial data set and the expanded training image.

[0018] Optionally, the backbone network comprises:

[0019] a plurality of convolution layers, each of which comprises a plurality of convolution blocks;

[0020] Each of the convolution blocks adopts a depth separable convolution;

[0021] The convolution block comprises a normal convolution block and a skip convolution block, and the skip convolution block is associated with other convolution blocks in a skip connection manner.

[0022] Optionally, the inputting of the training image in the data set into the backbone network to obtain a plurality of first feature images of different scales about the training image comprises:

[0023] feature extraction is performed on different input feature images from the first convolution block to the last convolution block to obtain output results of different convolution blocks in sequence;

[0024] The input feature image of a first convolutional block is the training image, and if the convolutional block is the last convolutional block in the convolutional layer, the output result of the convolutional block is the output result of the convolutional layer.

[0025] Different scale feature images are extracted from the output results of the plurality of convolutional layers and the plurality of convolutional blocks as a plurality of different scale first feature images of the training image.

[0026] Optionally, the input of the plurality of different scale first feature images into a feature fusion module for feature fusion to obtain a plurality of different scale second feature images comprises:

[0027] The plurality of different scale first feature images are input into a 1x1 convolution to generate a plurality of different scale sub-feature images of a unified dimension;

[0028] From the highest scale sub-feature image to the lowest scale sub-feature image, the up-sampling feature fusion is performed one by one to sequentially obtain different scale second feature images, wherein the highest scale second feature image is the highest scale sub-feature image.

[0029] The up-sampling feature fusion of the j-1 scale sub-feature image comprises:

[0030] The j scale second feature image is up-sampled to obtain an up-sampled feature map;

[0031] The up-sampled feature map and the j-1 scale sub-feature image are added to obtain a second feature image before the j-1 scale;

[0032] The second feature image before the j-1 scale is input into a 3x3 convolution to obtain the j-1 scale second feature image.

[0033] Optionally, the detection module comprises:

[0034] A plurality of sub-detection modules, the number of the sub-detection modules is the same as the number of the second feature image scales, and the network structures of the plurality of sub-detection modules are the same;

[0035] The network structure comprises a 3x3 convolution module, a context module, a face emotion classification module, a bounding box regression module, and a key point regression module.

[0036] Each of the sub-detection modules comprises three different prior boxes, and the prior boxes are used for detecting faces.

[0037] Optionally, the inputting the plurality of second feature images of different scales into the detection module for multi-scale face detection to obtain face feature images comprises:

[0038] For each scale of the second feature map, the following operations are performed to obtain a face feature image at the scale:

[0039] The second feature images are respectively input into a 3x3 convolution module and a context module of the corresponding sub-detection module to obtain first detection output results and second detection output results;

[0040] The first detection output results and the second detection output results are spliced in the dimension to obtain the face feature image at the scale.

[0041] Optionally, the emotion classification module comprises a face emotion classification module, a bounding box regression module, and a key point regression module,

[0042] The inputting the face feature image into the emotion classification module to obtain a model prediction result comprises:

[0043] For each scale of the face feature image, the following operations are performed to obtain a model prediction result at the scale:

[0044] The face feature image is respectively input into three 1x1 convolutional adjustment channels to obtain three classification input feature images;

[0045] The three classification input feature images are respectively input into the face emotion classification module, the bounding box regression module, and the key point regression module to obtain classification scores of the face emotion, coordinate information of the face bounding box, and coordinate information of the face key points at the scale.

[0046] Optionally, the optimizing the face emotion recognition network model according to the model prediction result to obtain an optimized face emotion recognition network model comprises:

[0047] A multi-task network loss function is obtained, the multi-task network loss function comprising: a classification loss of the face emotion, a regression loss of the coordinate information of the face bounding box, and a regression loss of the coordinate information of the face key points;

[0048] The face emotion recognition network model is optimized according to the classification scores of the face emotion, the coordinate information of the face bounding box, the coordinate information of the face key points, and the multi-task network loss function to obtain an optimized face emotion recognition network model.

[0049] In a second aspect, an embodiment of the present application provides a device for training a facial emotion recognition network model, the facial emotion recognition network model comprising a backbone network, a feature fusion module, a detection module and an emotion classification module, the device comprising:

[0050] an acquisition module configured to acquire a dataset;

[0051] a first feature module configured to input a training image in the dataset into the backbone network to obtain a plurality of first feature images of different scales about the training image;

[0052] a second feature module configured to input the plurality of first feature images of different scales into the feature fusion module for feature fusion to obtain a plurality of second feature images of different scales;

[0053] a facial feature module configured to input the plurality of second feature images of different scales into the detection module for a plurality of different scale face detection to obtain a facial feature image;

[0054] a classification module configured to input the facial feature image into the emotion classification module to obtain a model prediction result, the model prediction result comprising a classification score of facial emotion, coordinate information of a face frame and coordinate information of facial key points;

[0055] an optimization module configured to optimize the facial emotion recognition network model according to the model prediction result to obtain an optimized facial emotion recognition network model.

[0056] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory and a program stored in the memory and executable in the processor, the program being executed by the processor to implement the steps of the method for training a facial emotion recognition network model according to the first aspect.

[0057] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the method for training a facial emotion recognition network model according to the first aspect.

[0058] The face emotion recognition network model provided by the embodiment of the present application comprises a backbone network, a feature fusion module, a detection module and an emotion classification module. Based on the backbone network and training images in a data set, first feature images of multiple different scales about the training images are obtained. Then, second feature images of multiple different scales are obtained according to the feature fusion module and the first feature images. The detection module is used for multiple different scale face detection to obtain face feature images. The multiple different scale face detection can recognize face feature images in the images, and the detection of different scales can improve the accuracy of face feature image recognition. Then, the emotion classification module is used for analyzing the face feature images to obtain model prediction results. The model prediction results obtained based on the face feature images after the detection of different scales are more accurate, and the recognition of emotions is more accurate. Finally, the face emotion recognition network model is optimized according to the model prediction results to obtain an optimized face emotion recognition network model. Through the face emotion recognition network model optimized by the method of the embodiment of the present application, the image can realize an end-to-end face emotion recognition process in a natural scene, the extracted features can be classified and regressed to obtain face emotion classification through the face emotion recognition network model only once, the recognition accuracy is high, and the processing efficiency is also improved. BRIEF DESCRIPTION OF DRAWINGS

[0059] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings provided herein are for illustrative purposes only and are not considered a limitation of the present application. Moreover, like reference numerals are used to designate identical components throughout the several views. In the drawings:

[0060] Figure 1 is a flowchart of a training method of a face emotion recognition network model provided by the embodiment of the present application;

[0061] Figure 2 is a structural schematic diagram of a face emotion recognition network model provided by the embodiment of the present application;

[0062] Figure 3A is a schematic diagram of input image data before enhancement provided by the embodiment of the present application;

[0063] Figure 3B is a schematic diagram of a bidirectional grid mask provided by the embodiment of the present application;

[0064] Figure 3C is a schematic diagram of a grid mask provided by the embodiment of the present application;

[0065] Figure 4 is a schematic diagram of a depth separable convolution provided by the embodiment of the present application;

[0066] Figure 5 is a schematic diagram of a convolutional block structure provided by an embodiment of the present application;

[0067] Figure 6 is a schematic diagram of an inverse residual structure provided by an embodiment of the present application;

[0068] Figure 7 is a schematic diagram of an attention mechanism structure provided by an embodiment of the present application;

[0069] Figure 8 is a network structure diagram of a detection module provided by an embodiment of the present application;

[0070] Figure 9 is a structure diagram of a context module provided by an embodiment of the present application;

[0071] Figure 10 is a schematic diagram of a prior box provided by an embodiment of the present application;

[0072] Figure 11 is a structure schematic diagram of a training device of a facial emotion recognition network model provided by an embodiment of the present application;

[0073] Figure 12 is a structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0075] Please refer to Figure 1 and Figure 2 The present application provides a training method of a facial emotion recognition network model, the facial emotion recognition network model comprising a backbone network, a feature fusion module, a detection module and an emotion classification module, the method comprising:

[0076] Step 11: obtaining a data set;

[0077] Step 12: inputting a training image in the data set into the backbone network to obtain a plurality of first feature images of different scales about the training image;

[0078] Step 13: inputting the plurality of first feature images of different scales into the feature fusion module for feature fusion to obtain a plurality of second feature images of different scales;

[0079] Step 14: input the plurality of second feature images of different scales into a detection module for a plurality of different scale face detection, to obtain a face feature image;

[0080] Step 15: input the face feature image into an emotion classification module to obtain a model prediction result, the model prediction result including: a classification score of a face emotion, coordinate information of a face frame and coordinate information of a face key point;

[0081] Step 16: optimize the face emotion recognition network model according to the model prediction result, to obtain an optimized face emotion recognition network model.

[0082] The face emotion recognition network model provided by the embodiment of the application includes a backbone network, a feature fusion module, a detection module and an emotion classification module, based on the backbone network and training images in a data set, a plurality of first feature images of different scales about the training images are obtained; then, the second feature images of different scales are obtained according to the feature fusion module and the first feature images; the face feature image is obtained by using the detection module for a plurality of different scale face detection; the plurality of different scale face detection can recognize the face feature image in the image, and the detection of different scales can improve the accuracy of face feature image recognition; then, the model prediction result is obtained by analyzing the face feature image according to the emotion classification module, the model prediction result based on the face feature image after different scale detection is more accurate, and the emotion recognition is more accurate, finally, the face emotion recognition network model is optimized according to the model prediction result, to obtain an optimized face emotion recognition network model. The face emotion recognition network model optimized by the method of the embodiment of the application can realize the end-to-end face emotion recognition process of the image in a natural scene, realize the classification and regression of the extracted features by using the face emotion recognition network model only once, has high recognition accuracy, and can also improve the processing efficiency.

[0083] The deep learning neural network training process is easily affected by the size of the training data, and if there is not enough training data to learn new parameters, the process will not perform well. Usually, the original image is preprocessed to generate similar sample data, and data enhancement is simple, only the input image needs to be processed, without changing the network structure or adding additional parameters. If a reasonable preprocessing method is used, the robustness of the model can be improved, and the problems of overfitting and sample inequality can be avoided. Data enhancement has become an indispensable step in neural network training.

[0084] Information blocking is currently considered to be the most effective data enhancement method, but simple information blocking also has some shortcomings, such as incompatibility with small targets, weak ability to learn local features, etc.

[0085] The key of image sequence information deletion is to avoid over-deletion and retention of continuous regions, and a reasonable information deletion method should balance between deletion and retention of region information of images. If random deletion is used, a lot of useful information will be lost, resulting in incorrect model training.

[0086] Therefore, the embodiment of the present application provides a method for realizing data enhancement by using a grid mask, please refer to Figures 3A-3C , specifically, optionally, the obtaining data set comprises:

[0087] obtaining an initial data set, the initial data set comprising a plurality of training images;

[0088] generating a grid mask on at least part of the training images, the grid mask comprising a horizontal continuous mask and a vertical continuous mask;

[0089] randomly rotating the grid mask by the same angle in each dimension of the RGB channel to obtain an expanded training image;

[0090] obtaining the data set according to the training images in the initial data set and the expanded training image.

[0091] Optionally, the initial data set can use the Cohn-Kanade (CK+) data set, which is based on the Cohn-Kanade expansion, and the data set includes 593 image sequences from 123 objects under laboratory strips, which are classified into 8 emotion categories. The data set is one of the common methods for developing and evaluating facial expression analysis algorithms. Generally, researchers use leave-one-out cross-validation strategy or random training / test set configuration to use the data set. However, due to the small amount of data, it is difficult to measure the reliability of the algorithm by these methods.

[0092] Therefore, in the prior art, data enhancement is also an important step before training the neural network, which generates more useful data from the existing data to improve the accuracy of neural network training and prevent overfitting. Common data enhancement methods mainly include spatial geometric transformation, such as random cropping, random scaling, random flipping, etc.; color transformation, including noise, changing transparency, brightness and hue, etc.; information deletion, such as random deletion to help network learn features. There are many data enhancement methods, random erasing is to randomly set the continuous region of the input image to zero; Has method is to randomly delete several regions in the image; MixUP method is a data enhancement based on multi-source image fusion.

[0093] Optionally, the embodiment of the present application uses a grid mask to reasonably delete at least part of the plurality of training images in the initial data set.

[0094] The grid mask is an effective data enhancement method. It is not to remove some pixels in the image randomly, nor to generate rectangular blocks, but to generate continuous masks and fill certain values. Unlike other methods, the mask blocks of the grid mask have better sparsity and regularity, which helps to segment more object features and reduce the total amount of masks, but the richness and randomness of the masks are insufficient, therefore, the embodiment of the present application further improves the grid mask, specifically:

[0095] Please refer to Figure 3B , assuming W x represents the width of the horizontal mask, W y represents the width of the vertical mask, G x represents the distance width of adjacent masks in the horizontal direction, and G y represents the distance width of adjacent masks in the vertical direction.

[0096] A bidirectional (vertical and horizontal) grid mask is generated by using (W x , W y ) and (G x , G y ) respectively, and then they are multiplied to obtain a complete grid mask.

[0097] In order to enrich the distribution of the mask blocks and improve the randomness of the mask blocks, the grid mask is randomly rotated by the same angle in each dimension of the RGB channel, which can be a clockwise random rotation, and the rotation angle range can be from 0 degrees to 30 degrees.

[0098] Taking the input training image in Figure 3A as an example, a bidirectional grid mask as shown in Figure 3B is generated, and the grid mask in Figure 3B is randomly rotated to obtain Figure 3C .

[0099] Further, the value range of (W max , W min ) and (G x , G y ) can be further set by a random value method, and the values of (W min , W max , G x , G y ) are set in a range, and these parameters take random values. W min represents the maximum value of the width of the horizontal mask, W max represents the minimum value of the width of the vertical mask, G min represents the maximum value of the distance width of adjacent masks in the horizontal direction, and G max represents the minimum value of the distance width of adjacent masks in the vertical direction.

[0100] G x ,G y =random(G min ,G max );

[0101] W x ,W y =random(W min ,W max );

[0102] Further, the grid mask can be represented using 5 parameters:

[0103] (W min ,W max ,g min ,g min ,F);

[0104] The F vector represents the fill value of the mask in the RGB channel, and optionally, the fill value can be set to zero to represent the mask. The pixel value distribution of the enhanced image can be set by the value of F, and when the mask block occupies a large proportion in the image, the parameters of the mask block can be adjusted to retain the information of the original image.

[0105] The improved grid mask algorithm provided in the embodiment of the application can not only maintain the data deletion rate, but also effectively retain the subtle data change information in the image, is effective for blurred faces, and will not cause additional calculation.

[0106] In the embodiment of the application, the backbone network optionally comprises:

[0107] a plurality of convolution layers, each of the convolution layers comprising a plurality of convolution blocks;

[0108] Each of the convolution blocks adopts deep separable convolution.

[0109] The convolution block comprises a normal convolution block and a skip convolution block, and the skip convolution block is associated with other convolution blocks in a skip connection manner.

[0110] In some embodiments, the backbone network design optionally adopts a convolutional neural network as a basis.

[0111] Please refer to Figure 2 In some embodiments, the backbone network optionally constructs 7 convolution layers.

[0112] In order to achieve the purpose of lightweight, in some embodiments, deep separable convolution is optionally used instead of normal convolution. Please refer to Figure 4The embodiment of the present application provides a depth separable convolution structure, which is decomposed into a depth convolution and a point convolution from a standard convolution, uses the depth separable convolution to replace a traditional convolution layer, uses a single-channel convolution to fuse spatial information first, and then uses a point-by-point convolution to fuse channel information of features. The depth separable convolution provided by the embodiment of the present application can effectively separate regions and channels of an image, greatly reduces the amount of calculation, greatly improves the efficiency, and makes the model more lightweight. Specifically,

[0113] Please refer to Figure 4 , assuming that the input data size is WxWxC, first, a group convolution is performed by using C 3x3x1 convolution kernels; then, feature information of each channel is fused by using N 1x1xC point convolution operations. Assuming that the size of the convolution kernel is DxD and the step is 1, the parameter amount of the standard convolution is DxDxCxN, and the parameter amount of the depth separable convolution is DxDxC+CxN.

[0114] The amount of calculation of the standard convolution is as follows:

[0115] Amount(conv) = DxDxCxWxWxN

[0116] The amount of calculation of the depth separable convolution is as follows:

[0117] Amount(dsconv) = DxDxCxWxW+CxWxWxN

[0118] It can be seen that the ratio of the parameter amounts of the two is equal to the ratio of the amounts of calculation. Assuming that an 80x80x24 feature map is input, a 40x40x40 feature map is output after 3x3 convolution, and the parameter amount and the amount of calculation of the depth separable convolution in the above case are only about 1 / 7 of those of the traditional convolution. Obviously, the network model calculation speed is greatly improved.

[0119] Please refer to Figure 2 In some embodiments, optionally, the backbone network is composed of Conv1-7, and optionally, 15 convolution blocks Block are constructed, and each Conv includes a plurality of Block.

[0120] Please refer to Figure 2 In the embodiment of the present application, the ordinary convolution block can include Block1, Block2, Block4, Block7 and Block14, and the remaining convolution blocks are jump convolution blocks. In the embodiment of the present application, when the convolution block is a jump convolution block, the convolution block is jump connected. Please refer to Figure 5 for a structural diagram of the jump convolution block.

[0121] Please refer to Figure 2In the embodiment of the present application, optionally, the inputting the training image in the data set into the backbone network to obtain the first feature image of multiple different scales about the training image comprises:

[0122] The feature extraction is performed on different input feature images from the first convolution block to the last convolution block, and the output results of different convolution blocks are sequentially obtained.

[0123] The input feature image of the first convolution block is the training image, and if the convolution block is the last convolution block of the convolution layer where the convolution block is located, the output result of the convolution block is the output result of the convolution layer.

[0124] The multiple feature images of different scales are extracted from the output results of the multiple convolution layers and the multiple convolution blocks as the first feature images of multiple different scales of the training image.

[0125] In the embodiment of the present application, first, the feature extraction is performed on different input feature images from the first convolution block to the last convolution block, and the output results of different convolution blocks are sequentially obtained.

[0126] Specifically, optionally, please refer to Figure 5 When the convolution block is the i-th convolution block, first, the input feature image is input into the inverse residual network:

[0127] The feature channels of the input feature image of the i-th convolution block are expanded using the 1*1 and 3*3 convolution in the i-th convolution block, and the expanded feature image is obtained.

[0128] The expanded feature image is compressed using the 1*1 convolution, and the compressed feature image is obtained.

[0129] Please refer to Figure 6 The inverse residual structure used in the embodiment of the present application is to first expand the feature channels of the input data and then compress the feature channels, and the last layer uses a linear function to replace Relu, avoiding a large amount of information loss when converting from high dimension to low dimension. When the feature channels of the input and output are inconsistent, the output feature channels need to be added to the input through 1*1 convolution in the skip connection to obtain the final output. The inverse residual structure in the embodiment of the present application can also reduce the parameters while obtaining deeper image features.

[0130] Then the attention mechanism is introduced. When observing an image, an observer usually allocates limited attention efficiently, and pays more attention to the key area. In deep learning, this way is imitated to obtain more interesting information from a lot of feature information. The main process of the attention mechanism is that the input feature map is firstly used to obtain overall features for channels, then the weight of the overall feature of each channel is learned, and the correlation between channels is established to enhance the influence of high response features. Therefore, please refer to Figure 7 The convolution block in the embodiment of the application further introduces the attention mechanism, specifically:

[0131] The compressed feature image is down-sampled by using 2x2 maximum pooling to obtain a down-sampled feature image;

[0132] The down-sampled feature image is convolved four times by using a 1x1 convolution to obtain an attention feature image;

[0133] The attention feature image is taken as the output result of the i-th convolution block;

[0134] The output result of the i-th convolution block is taken as the input feature image of the i+1-th convolution block;

[0135] The attention feature is added to the feature image before the input inverse residual network to obtain the output result of the i-th convolution block.

[0136] Please refer to Figure 7 The attention mechanism in the embodiment of the application adopts a fully convolutional neural network, and uses a 1x1 convolution instead of a fully connected layer, so that the network can input image data of any size. The input feature is subjected to global average pooling (GAP), and the output size is 1x1xC. After being subjected to the first 1x1 convolution layer, the output size is 1x1xC / 4. After being subjected to the second 1x1 convolution layer, the output size is 1x1xC. Finally, a Scale operation is performed. The Scale operation is the result of multiplying the learned weight of each channel with the input C channels. Through the attention model provided in the embodiment of the application, the model can learn more important features and suppress unimportant features.

[0137] Finally, in the embodiment of the application, the feature images obtained by the feature extraction stages Conv2, Conv3, Block11 and Conv7 are taken as the first feature images of four scales of the training image.

[0138] Further, the first feature images of different scales are input into a feature fusion module for feature fusion, specifically:

[0139] The multiple first feature images of different scales are input into a 1×1 convolution to generate multiple sub-feature images of different scales with the same dimension.

[0140] From the highest-scale sub-feature image to the lowest-scale sub-feature image, upsampling feature fusion is performed one by one to obtain second feature images of different scales in sequence, wherein the second feature image of the highest scale is the sub-feature image of the highest scale;

[0141] The upsampling feature fusion of the sub-feature image at scale j-1 includes:

[0142] Upsample the second feature image at the j-th scale to obtain the upsampled feature map;

[0143] The upsampled feature map is added to the sub-feature image at the (j-1)th scale to obtain the first second feature image at the (j-1)th scale;

[0144] The second feature image at the (j-1)th scale is input into a 3×3 convolution to obtain the second feature image at the (j-1)th scale.

[0145] It is worth noting that in real-world applications, the images captured by cameras often show significant variations in facial scale, thus requiring face detection algorithms to have good robustness across multiple scales.

[0146] Optional, please refer to Figure 2 In the backbone network of this embodiment, there are four feature extraction stages: Stage 1, Stage 2, Stage 3, and Stage 4 extract four feature maps of different scales, namely the feature maps output by Conv2, Conv3, Block11, and Conv7. Optionally, the downsampling factor corresponding to the input image is 4x, 8x, 16x, and 32x, and the feature map sizes are 160×160×24, 80×80×40, 40×40×112, and 20×20×160, respectively. Among them, the feature map output by Conv2 has a smaller receptive field and is used to detect small faces, the feature map output by Conv3 is used to detect medium-sized faces, the feature map output by Block11 is used to detect medium-sized faces, and the feature map output by Conv7 has a larger receptive field and is used to detect large faces.

[0147] In the embodiment of the present application, the feature fusion part unifies the feature maps output by Conv2, Conv3, Block11 and Conv7 to have a dimension of 64 through point convolution, and the outputs are denoted as B1, B2, B3 and B4 layers respectively, B4 is P4 layer, B4 layer is up-sampled and added to B3 layer, in order to eliminate the aliasing effect of up-sampling, 3*3 convolution is needed to obtain P3 layer; similarly, P3 layer is up-sampled and added to B2 layer, and 3*3 convolution is needed to obtain P2 layer; P2 layer is up-sampled and added to B1 layer, and 3*3 convolution is needed to obtain P1 layer, thus the feature fusion is completed.

[0148] Further, P1, P2, P3 and P4 are input to the detection module for subsequent classification and regression work.

[0149] Specifically, refer to Figure 8 Optionally, the detection module comprises:

[0150] a plurality of sub-detection modules, the number of the sub-detection modules is the same as the number of the second feature image scales, and the network structures of the plurality of sub-detection modules are the same;

[0151] The network structure comprises: a 3*3 convolution module, a context module, a face emotion classification module, a frame body regression module and a key point regression module.

[0152] Each of the sub-detection modules comprises three different prior frames, and the prior frames are used for detecting faces.

[0153] Please refer to Figure 2 In the embodiment of the present application, the detection module comprises four sub-detection modules D1, D2, D3 and D4, that is, the number of the second feature image scales is also four, which means four scale feature modules, that is, the face size is divided into four levels, and small faces, medium-small faces, medium-large faces and large faces are identified according to different receptive fields. Specifically:

[0154] Please refer to Figure 8 Optionally, the down-sampling multiples of P1, P2, P3 and P4 after feature fusion using the feature maps output by Conv2, Conv3, Block11 and Conv7 are different, and it can be obtained that the input sizes of D1, D2, D3 and D4 modules are 160*160*64, 80*80*64, 40*40*64 and 20*20*64 respectively, and it is worth mentioning that the network structures of D1, D2, D3 and D4 are the same, only the sizes of the input feature maps are different, and small faces, medium-small faces, medium-large faces and large faces are identified according to different receptive fields.

[0155] In the embodiment of the present application, four sub-detection modules, each sub-detection module has a corresponding three prior boxes (Anchor), taking the D1 module as an example, the network output is 160x160x51, each channel of this feature map can be regarded as 160x160 pixel points of the original input image, each pixel point corresponds to three prior boxes of different scales, and the center position of the prior box is also the center position of the pixel. Please refer to Figure 10 , the dashed box represents the real position of the face frame, the center box with filling represents the grid responsible for predicting the face frame, and the three thick line boxes from large to small represent three prior boxes of different sizes centered on the center pixel of the feature map. The face is detected using multi-scale prior boxes.

[0156] Further, in the embodiment of the present application, optionally, the plurality of second feature images of different scales are input into the detection module for multi-scale face detection to obtain a face feature image.

[0157] For each scale of the second feature map, the following operations are performed to obtain a face feature image at the scale:

[0158] The second feature image is input into the 3x3 convolution module and the context module of the corresponding sub-detection module respectively to obtain a first detection output result and a second detection output result.

[0159] The first detection output result and the second detection output result are spliced in the dimension to obtain the face feature image at the scale.

[0160] Please refer to Figure 8 , in the embodiment of the present application, different scales are identified to obtain more rich context feature information, a 3x3 convolution and a context module Context Module are used for each module D1, D2, D3 and D4, and the two outputs are spliced in the dimension to obtain a size of WxWxC.

[0161] Please refer to Figure 9 , the context module expands the receptive field by introducing a large-scale filter to obtain more rich context information, and in the embodiment of the present application, three 3x3 filters are cascaded instead of a 7x7 filter, which increases the receptive field and reduces the calculation amount by 28%.

[0162] The emotion classification module includes a face emotion classification module, a frame regression module and a key point regression module,

[0163] The face feature image is input into the emotion classification module to obtain a model prediction result, which includes:

[0164] The following operations are performed on each scale face feature image to obtain a model prediction result at the scale:

[0165] The face feature image is input into three 1x1 convolutional adjustment channels respectively to obtain three classification input feature images;

[0166] The three classification input feature images are input into the face emotion classification module, the frame regression module and the key point regression module respectively to obtain the classification score of the face emotion, the coordinate information of the face frame and the coordinate information of the face key point at the scale.

[0167] The face emotion classification module in the embodiment of the application can output classification scores of different face emotions, and optionally, the face emotion classification can be divided into six basic emotions: happiness, anger, surprise, fear, disgust and sadness, and of course, the classification can be adjusted according to actual application requirements.

[0168] In the embodiment of the application, three 1x1 convolutions are used to adjust the channel number as input for face emotion classification, frame regression and key point regression. WxWx(3x3) face emotion classification scores, WxWx(3x4) frame regression coordinates and WxWx(3x10) key point regression coordinates are obtained.

[0169] Please refer to Figure 7 A key point regression branch is added to the recognition module, and since the related tasks promote each other, a key point regressor is added for joint learning, which can improve the recognition accuracy, and the key point recognition can correct the face, and face alignment is a necessary step in face recognition, and the key point recognition provides a prerequisite for subsequent recognition. Each recognition module is provided with three prior frames of different scales and a length-width ratio of 1.

[0170] Therefore, the feature map of 160*160 size generates 160*160*3 prior boxes, that is, the D1, D2, D3 and D4 modules generate 76800, 19200, 4800 and 1200 prior boxes respectively, and the total number of prior boxes is 102000. In the training stage, the Intersection over Union (IOU) of the prior boxes and the real boxes needs to be calculated, and the threshold is set to 0.35. The prior boxes with the IOU greater than 0.35 are classified as target prior boxes, and the prior boxes with the IOU less than 0.35 are classified as background as negative samples. Since the number of faces on a picture is limited and the number of anchors is large, the problem of unbalanced positive and negative samples may occur. Therefore, the Online Hard Example Mining (OHEM) is used to select the negative samples from high to low order according to the loss value of the anchors determined as negative samples, and the positive and negative ratio is 1:7. Each target prior box has a real box responsible for prediction. According to the offset parameters of the corresponding box body regression and key point regression output by the network, the position of the predicted box and the position of the key point are corrected.

[0171] Therefore, it can be seen that it is crucial to obtain prior boxes of suitable size. In the embodiment of the present application, the K-Means method is used to cluster the data set, so that prior boxes of suitable size for the data set can be obtained. Generally, the Euclidean distance is used for clustering analysis, but the Euclidean distance of a large box is larger, so the Euclidean distance cannot be used, and the IOU between the cluster center and other boxes is used as the Euclidean distance. The distance formula of clustering is shown in the following formula. The use of 1-IOU can ensure that the higher the coincidence degree is, the shorter the distance should be, and the effect is enhanced.

[0172] dist(i,center)=1=IOU(i,center);

[0173] Three anchors are set for each stage feature map, so k=12. The k-means clustering algorithm can be used to calculate 12 prior box sizes.

[0174] In the embodiment of the present application, the face emotion recognition network model is optimized according to the model prediction result, and an optimized face emotion recognition network model is obtained.

[0175] A multi-task network loss function is obtained, and the multi-task network loss function includes a face emotion classification loss, a face box coordinate information regression loss and a face key point coordinate information regression loss.

[0176] The face emotion recognition network model is optimized according to the face emotion classification score, the face box coordinate information, the face key point coordinate information and the multi-task network loss function, and an optimized face emotion recognition network model is obtained.

[0177] In the embodiment of the present application, a multi-task network loss function is used, which is divided into three parts in total, the first part is the classification loss of the face emotion, the second part is the regression loss of the coordinate information of the face frame, and the third part is the coordinate information regression loss of the face key points. The formula of the multi-task loss function is as follows:

[0178]

[0179] L c represents the face classification loss, and the softmax loss is used for classification, p i represents the classification prediction result in the prior frame, represents the classification of the true label.

[0180] The specific optimization process is as follows: first, the predicted value is calculated by using the softmax to calculate the probability, and the sum of the probabilities of classification is 1, the formula of the softmax function is shown below, represents the probability output value of the i th classification of the softmax, and k represents k classes.

[0181]

[0182] Then, the classification error is calculated by using the cross-entropy function to calculate the true value and the predicted classification in the network, and the cross-entropy function is shown below.

[0183] C=-∑ i y i loga i ;

[0184] L b represents the regression loss of the face frame coordinate position, L l represents the regression loss of the position of the five key points of the face, both of the two regression losses use loss function, q i represents whether it is a face, 0 represents the background, and 1 represents other situations, when the data set is manually labeled, part of the faces are not labeled with key points due to being too small or other reasons, therefore t i 0 represents that the face has no key point label, and 1 represents that the face has available key point information for calculating the regression loss, b i represents the position information of the face frame predicted by the model relative to the grid of the feature map, represents the position information of the face frame labeled by the true label relative to the grid of the feature map, and l i represents the position information of the face key point predicted by the model relative to the grid of the feature map, represents the position information of the face key point labeled by the true label relative to the grid of the feature map.

[0185] L b For example, the regression loss function is shown in the formula below.

[0186]

[0187]

[0188] Wherein, λ1 and λ2 are weight parameters, respectively 0.25 and 0.15, which increases the importance of face frame positioning and face key point positioning in supervised learning, and the key point weight is small, which can avoid large deviation of key point prediction leading to excessive loss.

[0189] According to the classification score of the face emotion, the coordinate information of the face frame, the coordinate information of the face key point and the multi-task network loss function in the embodiment of the application, the face emotion recognition network model is optimized to obtain the optimized face emotion recognition network model.

[0190] The face emotion recognition network model proposed in the embodiment of the application mainly includes a backbone network, a feature fusion module, a detection module and an emotion classification module. The backbone network is composed of Conv1~7, each Conv contains a plurality of blocks containing convolutional layers, and some blocks are associated with each other through a skip connection, which can solve the grid degradation problem caused by a deep network, and batch normalization (Batch Normal) is used to accelerate model convergence, prevent gradient disappearance and explosion, and deep separable convolution is used instead of traditional convolution to achieve the purpose of lightness. Furthermore, four feature layers of different scales are extracted from the backbone network to construct a feature pyramid for feature fusion, and then the four layers are sent to four face detection modules (D1, D2, D3, D4) respectively to detect small, medium, small, medium, large and large target faces. Further, the emotion classification module is based on the SSH algorithm and adopts a multi-task learning strategy, including face classification, frame regression and face key point regression operations, and a context module structure is used in the emotion classification module to expand the receptive field and obtain more rich context information. The face emotion recognition network model trained in the embodiment of the application does not contain a full connection layer, so there is no limitation on the resolution of the input picture, and the efficiency is higher.

[0191] Please refer to Figure 11 The embodiment of the application also provides a training device 20 of a face emotion recognition network model, the face emotion recognition network model comprising a backbone network, a feature fusion module, a detection module and an emotion classification module, the device comprising:

[0192] The acquisition module 21 is configured to acquire a data set.

[0193] The first feature module 22 is configured to input a training image in the data set into the backbone network to obtain a plurality of first feature images of different scales about the training image.

[0194] The second feature module 23 is configured to input the plurality of first feature images of different scales into a feature fusion module for feature fusion to obtain a plurality of second feature images of different scales.

[0195] The face feature module 24 is configured to input the plurality of second feature images of different scales into a detection module for a plurality of different scale face detection to obtain a face feature image.

[0196] The classification module 25 is configured to input the face feature image into an emotion classification module to obtain a model prediction result, wherein the model prediction result comprises a classification score of a face emotion, coordinate information of a face frame and coordinate information of a face key point.

[0197] The optimization module 26 is configured to optimize the face emotion recognition network model according to the model prediction result to obtain an optimized face emotion recognition network model.

[0198] In the embodiment of the present application, optionally, the acquisition module 21 comprises:

[0199] An acquisition sub-module is configured to acquire an initial data set, wherein the initial data set comprises a plurality of training images.

[0200] A grid mask is generated on at least part of the training images, wherein the grid mask comprises a horizontal continuous mask and a vertical continuous mask.

[0201] The grid mask is randomly rotated by the same angle on each dimension of the RGB channel to obtain an expanded training image.

[0202] The data set is obtained according to the training images in the initial data set and the expanded training image.

[0203] In the embodiment of the present application, optionally, the backbone network comprises:

[0204] A plurality of convolution layers, each of which comprises a plurality of convolution blocks.

[0205] Each of the convolution blocks adopts a depth separable convolution.

[0206] The convolution block comprises a normal convolution block and a skip convolution block, and the skip convolution block is associated with other convolution blocks in a skip connection manner.

[0207] In the embodiment of the present application, optionally, the first feature module 22 comprises:

[0208] The first feature submodule is configured to extract features from different input feature images from the first convolutional block to the last convolutional block, and sequentially obtain output results of different convolutional blocks.

[0209] The input feature image of the first convolutional block is the training image, and if the convolutional block is the last convolutional block of the convolutional layer in which the convolutional block is located, the output result of the convolutional block is the output result of the convolutional layer.

[0210] The output results of the plurality of convolutional layers and the plurality of convolutional blocks are extracted to obtain a plurality of first feature images of different scales as the training image.

[0211] In the embodiment of the application, the second feature module 23 comprises:

[0212] The second feature submodule is configured to input the plurality of first feature images of different scales into 1x1 convolution to generate a plurality of sub-feature images of different scales of a uniform dimension.

[0213] The sub-feature images of different scales are sequentially obtained by performing upsampling feature fusion on the sub-feature image of the highest scale to the sub-feature image of the lowest scale, wherein the sub-feature image of the highest scale is the sub-feature image of the highest scale.

[0214] The upsampling feature fusion on the sub-feature image of the j-1th scale comprises:

[0215] The jth-scale second feature image is upsampled to obtain an upsampled feature map.

[0216] The upsampled feature map and the sub-feature image of the j-1th scale are added to obtain a second feature image of the j-1th scale.

[0217] The second feature image of the j-1th scale is obtained by inputting the second feature image of the j-1th scale into 3x3 convolution.

[0218] In the embodiment of the application, the detection module comprises:

[0219] A plurality of sub-detection modules, the number of the sub-detection modules is the same as the number of the second feature image scales, and the network structures of the plurality of sub-detection modules are the same.

[0220] The network structure comprises a 3x3 convolution module, a context module, a face emotion classification module, a bounding box regression module, and a key point regression module.

[0221] Each of the sub-detection modules comprises three different prior boxes, and the prior boxes are used for detecting a face.

[0222] In the embodiment of the present application, the face feature module 24 comprises:

[0223] The face feature sub-module is configured to perform the following operations on the second feature map of each scale to obtain a face feature image at the scale:

[0224] The second feature image is input into a 3x3 convolution module and a context module of the corresponding sub-detection module respectively to obtain a first detection output result and a second detection output result.

[0225] The first detection output result and the second detection output result are spliced in the dimension to obtain the face feature image at the scale.

[0226] In the embodiment of the present application, the emotion classification module comprises a face emotion classification module, a bounding box regression module and a key point regression module,

[0227] The classification module 25 comprises:

[0228] The classification sub-module is configured to perform the following operations on the face feature image of each scale to obtain a model prediction result at the scale:

[0229] The face feature image is input into three 1x1 convolution adjustment channels respectively to obtain three classification input feature images.

[0230] The three classification input feature images are input into the face emotion classification module, the bounding box regression module and the key point regression module respectively to obtain a classification score of the face emotion, coordinate information of the face bounding box and coordinate information of the face key point at the scale.

[0231] In the embodiment of the present application, the optimization module 26 comprises:

[0232] The optimization sub-module is configured to obtain a multi-task network loss function, wherein the multi-task network loss function comprises a classification loss of the face emotion, a regression loss of the coordinate information of the face bounding box and a regression loss of the coordinate information of the face key point.

[0233] The face emotion recognition network model is optimized according to the classification score of the face emotion, the coordinate information of the face bounding box, the coordinate information of the face key point and the multi-task network loss function to obtain an optimized face emotion recognition network model.

[0234] The training device for the face emotion recognition network model provided by the embodiment of the present application can realize Figure 1The method embodiments of the present application implement the various processes and achieve the same technical effects, and thus, details are not repeated here.

[0235] The present application provides an electronic device 30, as shown in Figure 12 Figure 12 The present application provides an electronic device 30, as shown in

[0236] The present application provides an electronic device 30, as shown in

[0237] The computer readable medium includes permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0238] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0239] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. ​

[0240] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software product, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc), including a number of instructions to make a service classification device (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0241] The above-mentioned is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principle of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A training method for a facial emotion recognition network model, characterized in that, The facial emotion recognition network model includes a backbone network, a feature fusion module, a detection module, and an emotion classification module. The method includes: Obtain the dataset; The training images in the dataset are input into the backbone network to obtain first feature images of the training images at multiple different scales; The multiple first feature images of different scales are input into the feature fusion module for feature fusion to obtain multiple second feature images of different scales; The multiple second feature images at different scales are input into the detection module to perform face detection at multiple different scales, thereby obtaining face feature images; The facial feature image is input into the emotion classification module to obtain the model prediction result, which includes: the classification score of facial emotion, the coordinate information of the face bounding box, and the coordinate information of the facial key points. The facial emotion recognition network model is optimized based on the prediction results of the model to obtain the optimized facial emotion recognition network model. The detection module includes: Multiple sub-detection modules, the number of which is the same as the number of scales in the second feature image, and the network structure of the multiple sub-detection modules is the same; The network structure includes: a 3×3 convolutional module, a context module, a face emotion classification module, a bounding box regression module, and a key point regression module; Each of the sub-detection modules contains three different prior boxes, which are used for face detection; The step of inputting the multiple second feature images of different scales into the detection module to perform face detection at multiple different scales to obtain face feature images includes: Perform the following operations on the second feature map at each scale to obtain the face feature image at that scale: The second feature image is input into the 3×3 convolution module and context module of the corresponding sub-detection module to obtain the first detection output result and the second detection output result. The first detection output and the second detection output are concatenated dimensionally to obtain the facial feature image at the specified scale.

2. The method according to claim 1, characterized in that, The acquisition of the dataset includes: Obtain an initial dataset, which includes multiple training images; A grid mask is generated on at least a portion of the training images, the grid mask comprising a continuous mask in the horizontal direction and a continuous mask in the vertical direction; The grid mask is randomly rotated by the same angle in each dimension of the RGB channels to obtain the augmented training image; The dataset is obtained based on the training images in the initial dataset and the expanded training images.

3. The method according to claim 1, characterized in that, The backbone network includes: Multiple convolutional layers, each of which contains a plurality of convolutional blocks; Each of the convolutional blocks employs depthwise separable convolution; The convolutional block includes ordinary convolutional blocks and skip convolutional blocks, and the skip convolutional block is associated with other convolutional blocks in a skip connection manner.

4. The method according to claim 3, characterized in that, The step of inputting the training images from the dataset into the backbone network to obtain first feature images of the training images at multiple different scales includes: From the first convolutional block to the last convolutional block, features are extracted from different input feature images, and the output results of different convolutional blocks are obtained sequentially. Wherein, the input feature image of the first convolutional block is the training image, and if the convolutional block is the last convolutional block of the convolutional layer in which the convolutional block is located, the output result of the convolutional block is the output result of the convolutional layer; Multiple feature images of different scales are extracted from the output results of the multiple convolutional layers and multiple convolutional blocks as the first feature images of multiple different scales of the training image.

5. The method according to claim 1, characterized in that, The step of inputting the multiple first feature images of different scales into the feature fusion module for feature fusion to obtain multiple second feature images of different scales includes: The multiple first feature images of different scales are input into a 1×1 convolution to generate multiple sub-feature images of different scales with the same dimension. From the highest-scale sub-feature image to the lowest-scale sub-feature image, upsampling feature fusion is performed one by one to obtain second feature images of different scales in sequence, wherein the second feature image of the highest scale is the sub-feature image of the highest scale; The upsampling feature fusion of the sub-feature image at scale j-1 includes: Upsample the second feature image at scale j to obtain the upsampled feature map; The upsampled feature map is added to the sub-feature image at the (j-1)th scale to obtain the first second feature image at the (j-1)th scale; The second feature image at the (j-1)th scale is input into a 3×3 convolution to obtain the second feature image at the (j-1)th scale.

6. The method according to claim 1, characterized in that, The emotion classification module includes a face emotion classification module, a frame regression module, and a key point regression module. The step of inputting the facial feature image into the emotion classification module to obtain the model prediction result includes: Perform the following operations on the facial feature images at each scale to obtain the model prediction results at that scale: The facial feature images are input into three 1×1 convolutional layers with adjusted channel numbers to obtain three classification input feature images; The three classification input feature images are respectively input into the face emotion classification module, the bounding box regression module, and the key point regression module to obtain the face emotion classification score, the coordinate information of the face bounding box, and the coordinate information of the face key points at the specified scale.

7. The method according to claim 1, characterized in that, The step of optimizing the facial emotion recognition network model based on the prediction results of the model to obtain the optimized facial emotion recognition network model includes: Obtain the multi-task network loss function, which includes: the classification loss of facial emotion, the regression loss of the coordinate information of the face bounding box, and the regression loss of the coordinate information of the facial key points; Based on the facial emotion classification score, the coordinate information of the face bounding box, the coordinate information of the facial key points, and the multi-task network loss function, the facial emotion recognition network model is optimized to obtain the optimized facial emotion recognition network model.

8. A training device for a facial emotion recognition network model, characterized in that, The facial emotion recognition network model includes a backbone network, a feature fusion module, a detection module, and an emotion classification module; the device includes: The acquisition module is used to acquire datasets; The first feature module is used to input the training images in the dataset into the backbone network to obtain multiple first feature images of different scales about the training images; The second feature module is used to input the multiple first feature images of different scales into the feature fusion module for feature fusion to obtain multiple second feature images of different scales. The face feature module is used to input the multiple second feature images of different scales into the detection module to perform face detection at multiple different scales and obtain face feature images; The classification module is used to input the facial feature image into the emotion classification module to obtain the model prediction result. The model prediction result includes: the classification score of facial emotion, the coordinate information of the face bounding box, and the coordinate information of facial key points. An optimization module is used to optimize the facial emotion recognition network model based on the model prediction results, so as to obtain an optimized facial emotion recognition network model. The detection module includes: Multiple sub-detection modules, the number of which is the same as the number of scales in the second feature image, and the network structure of the multiple sub-detection modules is the same; The network structure includes: a 3×3 convolutional module, a context module, a face emotion classification module, a bounding box regression module, and a key point regression module; Each of the sub-detection modules contains three different prior boxes, which are used for face detection; The facial feature module includes: The face feature submodule is used to perform the following operations on the second feature map at each scale to obtain a face feature image at said scale: The second feature image is input into the 3×3 convolution module and context module of the corresponding sub-detection module to obtain the first detection output result and the second detection output result. The first detection output and the second detection output are concatenated dimensionally to obtain the facial feature image at the specified scale.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the training method for a facial emotion recognition network model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on a computer-readable storage medium, which, when executed by a processor, implements the steps of the training method for a facial emotion recognition network model as described in any one of claims 1 to 7.

Images