Expression recognition method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an expression recognition method and device, computer equipment, a readable storage medium and a program product, and relates to the field of expression recognition. The method comprises the steps of inputting a target face image into an expression recognition model in response to an expression recognition instruction for the target face image; the expression recognition model comprises a main recognition network and a branch recognition network; the main recognition network is obtained based on attention mechanism training; the branch identification network is obtained by training based on an output result of the main identification network; performing expression recognition on the target face image through a branch recognition network to obtain a first expression recognition result; performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result; and performing weighted summation on the first expression recognition result and the second expression recognition result to obtain an expression recognition result of the target face image. By adopting the method, the facial expression recognition accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of expression recognition, and in particular to an expression recognition method, apparatus, computer equipment, readable storage medium and program product. Background Art

[0002] Facial expression recognition has always been one of the research hotspots in the field of computer vision. Facial expression recognition is an important way to convey emotional information and has a wide range of applications in human-computer interaction, recommendation systems, medical research and other fields.

[0003] At present, the research on facial expression recognition is mainly based on two methods: traditional manual feature extraction and machine learning algorithm. Traditional manual feature extraction is too complicated and inefficient, so this method is gradually replaced by machine learning algorithm-based methods. However, due to the complexity and diversity of facial expressions, most existing machine learning algorithms cannot extract discriminative features, resulting in low accuracy of facial expression recognition. Summary of the invention

[0004] Based on this, it is necessary to provide an expression recognition method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of facial expression recognition in order to solve the above technical problems.

[0005] In a first aspect, the present application provides an expression recognition method, comprising: in response to an expression recognition instruction for a target facial image, inputting the target facial image into an expression recognition model; the expression recognition model comprises a main recognition network and a branch recognition network; the main recognition network is trained based on an attention mechanism; the branch recognition network is trained based on the output result of the main recognition network; performing expression recognition on the target facial image through the branch recognition network to obtain a first expression recognition result; performing expression recognition on the target facial image through the main recognition network to obtain a second expression recognition result; performing weighted summation of the first expression recognition result and the second expression recognition result to obtain an expression recognition result of the target facial image.

[0006] In one embodiment, the branch recognition network includes multiple intermediate residual blocks; the branch recognition network is used to perform expression recognition on a target facial image to obtain a first expression recognition result, including: performing feature extraction on the target facial image through each intermediate residual block to obtain a branch feature map output by each intermediate residual block; and using each branch feature map together as the first expression recognition result.

[0007] In one of the embodiments, the branch recognition network also includes a target residual block connected to the fully connected layer of the branch recognition network; the method also includes: obtaining a target feature map output by the target residual block; performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result, including: inputting the target feature map into the main recognition network, and pooling the target feature map through the main recognition network to obtain a pooled feature map; determining the channel attention weight based on the pooled feature map; multiplying the channel attention weight by the target feature map to obtain a channel attention feature map, and using the channel attention feature map as the second expression recognition result.

[0008] In one of the embodiments, a target feature map is pooled to obtain a pooled feature map, including: performing pooling processing on the target feature map in an image width direction to obtain a first feature image of the target feature map in the image width direction; performing pooling processing on the target feature map in an image height direction to obtain a second feature image of the target feature map in the image height direction; and using the first feature image and the second feature image together as the pooled feature map.

[0009] In one of the embodiments, determining the channel attention weight based on the pooled feature map includes: aggregating the first feature image and the second feature image to obtain an aggregated feature map; and generating the channel attention weight based on the aggregated feature map.

[0010] In one embodiment, the training process of the expression recognition model includes: obtaining multiple facial expression images for model training; based on the facial expression images and the attention mechanism, training a first pre-trained network until the network loss value of the first pre-trained network is lower than the loss threshold, thereby obtaining a main recognition network; inputting the facial expression images into the main recognition network, performing facial expression prediction, and obtaining expression prediction results; based on the expression prediction results, training a second pre-trained network until the training stop condition is met, thereby obtaining a branch recognition network; the main recognition network and the branch recognition network together constitute the expression recognition model.

[0011] In the second aspect, the present application also provides an expression recognition device, including: a model input module, which is used to input the target face image into the expression recognition model in response to the expression recognition instruction for the target face image; the expression recognition model includes a main recognition network and a branch recognition network; the main recognition network is trained based on the attention mechanism; the branch recognition network is trained based on the output result of the main recognition network; a first expression recognition module, which is used to perform expression recognition on the target face image through the branch recognition network to obtain a first expression recognition result; a second expression recognition module, which is used to perform expression recognition on the target face image through the main recognition network to obtain a second expression recognition result; a model output module, which is used to perform weighted summation of the first expression recognition result and the second expression recognition result to obtain the expression recognition result of the target face image.

[0012] In a third aspect, the present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: in response to an expression recognition instruction for a target facial image, inputting the target facial image into an expression recognition model; the expression recognition model comprises a main recognition network and a branch recognition network; the main recognition network is trained based on an attention mechanism; the branch recognition network is trained based on the output result of the main recognition network; performing expression recognition on the target facial image through the branch recognition network to obtain a first expression recognition result; performing expression recognition on the target facial image through the main recognition network to obtain a second expression recognition result; performing weighted summation on the first expression recognition result and the second expression recognition result to obtain an expression recognition result of the target facial image.

[0013] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the following steps when the computer program is executed by a processor: in response to an expression recognition instruction for a target facial image, inputting the target facial image into an expression recognition model; the expression recognition model comprises a main recognition network and a branch recognition network; the main recognition network is trained based on an attention mechanism; the branch recognition network is trained based on the output result of the main recognition network; performing expression recognition on the target facial image through the branch recognition network to obtain a first expression recognition result; performing expression recognition on the target facial image through the main recognition network to obtain a second expression recognition result; performing weighted summation on the first expression recognition result and the second expression recognition result to obtain an expression recognition result of the target facial image.

[0014] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the following steps when executed by a processor: in response to an expression recognition instruction for a target facial image, inputting the target facial image into an expression recognition model; the expression recognition model comprises a main recognition network and a branch recognition network; the main recognition network is trained based on an attention mechanism; the branch recognition network is trained based on the output result of the main recognition network; performing expression recognition on the target facial image through the branch recognition network to obtain a first expression recognition result; performing expression recognition on the target facial image through the main recognition network to obtain a second expression recognition result; performing weighted summation on the first expression recognition result and the second expression recognition result to obtain an expression recognition result of the target facial image.

[0015] The above-mentioned expression recognition method, device, computer equipment, computer-readable storage medium and computer program product, in response to the expression recognition instruction for the target face image, input the target face image into the expression recognition model including the main recognition network and the branch recognition network. Since the main recognition network is trained based on the attention mechanism, the attention mechanism can guide the model to pay more attention to the key areas in the facial expression and quantify the contribution of different areas to the expression recognition task, thereby improving the model's ability to extract expression features, and then improve the expression recognition accuracy. The branch recognition network is trained based on the output results of the main recognition network. The branch recognition network can learn richer knowledge from the main recognition network. The combination of the two can enhance the generalization ability of the expression recognition model, so that it can maintain good recognition performance in different scenarios and conditions. Finally, the expression recognition results output by the branch recognition network and the main recognition network are weighted and summed to obtain the expression recognition result of the target face image. Therefore, the present application can effectively improve the accuracy of facial expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0017] FIG1 is a diagram showing an application environment of a method for facial expression recognition in one embodiment;

[0018] FIG2 is a schematic diagram of a flow chart of a method for facial expression recognition in one embodiment;

[0019] FIG3 is a schematic diagram of a process of extracting features from a residual block in one embodiment;

[0020] FIG4 is a schematic diagram of a process for generating a channel attention feature map in one embodiment;

[0021] FIG5 is a schematic diagram of a flow chart of a pooling process in one embodiment;

[0022] FIG6 is a schematic diagram of a process of feature graph aggregation in one embodiment;

[0023] FIG7 is a flow chart of training an expression recognition model in one embodiment;

[0024] FIG8 is a schematic diagram of an expression recognition model in a specific embodiment;

[0025] FIG9 is a prediction flow chart of a branch identification network in a specific embodiment;

[0026] FIG10 is a prediction flow chart of a main recognition network in a specific embodiment;

[0027] FIG11 is a block diagram of a facial expression recognition device according to an embodiment;

[0028] FIG. 12 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The expression recognition method provided in the embodiment of the present application can be applied in the application environment as shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and Internet of Things devices. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0031] Specifically, when the server 104 receives the expression recognition instruction for the target face image initiated by the terminal 102, the target face image is input into the pre-trained expression recognition model. The expression recognition model includes a main recognition network and a branch recognition network. The main recognition network is trained based on the attention mechanism. The branch recognition network is trained based on the output result of the main recognition network. The expression recognition of the target face image is performed through the branch recognition network and the main recognition network respectively. The expression recognition results output by the two networks are weighted and summed to obtain the expression recognition result of the final target face image.

[0032] In an exemplary embodiment, as shown in FIG. 2 , a method for facial expression recognition is provided, which is described by taking the method applied to the server 104 in FIG. 1 as an example, and includes:

[0033] Step S202, in response to an expression recognition instruction for a target facial image, inputs the target facial image into an expression recognition model; the expression recognition model includes a main recognition network and a branch recognition network; the main recognition network is trained based on an attention mechanism; and the branch recognition network is trained based on an output result of the main recognition network.

[0034] The target facial image may refer to a facial image that needs to be subjected to expression recognition, and may be a static image or a dynamic video image. This embodiment does not limit the number and form of expressions of the target facial image. The expression recognition instruction may refer to an instruction for performing expression recognition on the target facial image. Expression recognition refers to identifying facial expressions from a given facial image, thereby determining the emotion of the object being recognized. The recognized expressions may include but are not limited to any one of joy, sadness, anger, surprise, disgust, fear, etc. The expression recognition model is a deep learning model used to recognize facial expressions, such as a convolutional neural network. In this embodiment, the expression recognition model consists of a main recognition network and a branch recognition network.

[0035] The main recognition network is pre-trained based on the attention mechanism. The attention mechanism is a mechanism that can select key information for processing and ignore other unimportant information when processing a large amount of input information, similar to the human focus function when processing information. This mechanism enables the model to respond to different tasks and input data more flexibly.

[0036] The branch recognition network is trained based on the output results of the main recognition network. This process can be understood as knowledge distillation, which means that during the model training process, different versions of the model itself (or outputs at different stages) are used as "teachers" to guide its own learning process in order to achieve the purpose of optimizing the model effect. In other words, in this embodiment, the output results of the main recognition network are used as "teachers" to guide the learning of the branch recognition network, so that the branch recognition network can learn richer knowledge from the main recognition network. Thereby improving the generalization ability of the entire expression recognition model, so that it can maintain good recognition performance in different scenarios and conditions.

[0037] Exemplarily, when the server receives a target facial image for which expression recognition is required and sent by the terminal, the server may input the target facial image into a pre-trained expression recognition model for expression recognition.

[0038] In one example, after the server obtains the target face image, it can also pre-process the target face image before performing expression recognition. For example, the target face image can be grayed to reduce the amount of calculation and avoid interference from complex backgrounds. It is also possible to use methods such as median filtering, mean filtering or Gaussian filtering to remove noise in the image and improve image quality. The target face image can also be transformed to a uniform angle or posture to reduce the impact of posture and expression differences on expression recognition. Specifically, it can be by extracting key feature points such as eyes, nose, mouth, etc. in the target face image, and then aligning the face in the target face image to a standard template through similarity transformations such as rotation, scaling and translation of these key points. Of course, in practical applications, a suitable image pre-processing method can be formulated according to demand, and this embodiment does not limit this.

[0039] Step S204, performing expression recognition on the target face image through a branch recognition network to obtain a first expression recognition result.

[0040] Among them, the branch recognition network can be a residual network (Residual Network, ResNet). The residual network introduces residual blocks and jump connections to make the network easier to optimize and maintain a high accuracy even when the depth is increased. In addition, the residual network allows information to be directly transferred from one layer to another through jump connections, which can alleviate the common gradient vanishing problem in deep neural networks. The first expression recognition result refers to the expression feature map output by the branch recognition network.

[0041] In one example, the branch identification network may specifically be a ResNet34 network.

[0042] Exemplarily, after the server inputs the target facial image into the expression recognition model, the branch recognition network in the expression recognition model will perform expression recognition on the target facial image and output the corresponding expression recognition result.

[0043] Step S206, performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result.

[0044] Among them, the second expression recognition result refers to the expression feature map output by the main recognition network.

[0045] Exemplarily, after the server inputs the target face image into the expression recognition model, the main recognition network in the expression recognition model will also perform expression recognition on the target face image and output the corresponding expression recognition result. In one example, the branch recognition network can perform recognition first, and the main recognition network can perform recognition later.

[0046] Step S208, performing weighted summation on the first expression recognition result and the second expression recognition result to obtain an expression recognition result of the target facial image.

[0047] Among them, the first expression recognition result output by the branch recognition network and the second expression recognition result output by the main recognition network can both correspond to a weight, and the weight can be determined according to the model contribution of the two recognition networks. The greater the contribution, the greater the weight. Of course, the weights of the two recognition results can also be customized according to actual conditions. The expression recognition result can include the expression category and expression probability of the target face image.

[0048] Exemplarily, after the branch recognition network and the main recognition network have completed recognition and output corresponding expression recognition results, the first expression recognition result and the second expression recognition result can be multiplied by their respective corresponding weights and then added together through the output layer of the expression recognition model to obtain the expression category to which the final target facial image belongs and the probability of belonging to that expression.

[0049] In this embodiment, the server responds to the expression recognition instruction for the target face image initiated by the terminal, and inputs the target face image into the expression recognition model including the main recognition network and the branch recognition network. Since the main recognition network is trained based on the attention mechanism, the attention mechanism can guide the model to pay more attention to the key areas in the facial expressions, and quantify the contribution of different areas to the expression recognition task, thereby improving the model's ability to extract expression features, and then improve the accuracy of expression recognition. The branch recognition network is trained based on the output results of the main recognition network. The branch recognition network can learn more abundant knowledge from the main recognition network. The combination of the two can enhance the generalization ability of the expression recognition model, so that it can maintain good recognition performance in different scenarios and conditions. Finally, the expression recognition results output by the branch recognition network and the main recognition network are weighted and summed to obtain the expression recognition result of the target face image. Therefore, this embodiment can effectively improve the accuracy of facial expression recognition.

[0050] In an exemplary embodiment, as shown in FIG3 , expression recognition is performed on a target face image through a branch recognition network to obtain a first expression recognition result, including:

[0051] Step S302, performing feature extraction on the target face image through each intermediate residual block respectively, and obtaining a branch feature map output by each intermediate residual block.

[0052] Among them, the branch recognition network is composed of multiple residual blocks, which include multiple intermediate residual blocks. The intermediate residual block can be understood as a residual block located in the middle, or in other words, a residual block other than the last residual block. A new residual block is connected after each residual block to extract features separately. Each intermediate residual block corresponds to a branch. It should be noted that the number of branches can be determined according to actual conditions. In this embodiment, three branches are introduced, which corresponds to three intermediate residual blocks. The branch feature map refers to the feature map output after each residual block extracts expression features and average pooling on the target face image.

[0053] For example, after the target face image is input into the branch recognition network, each intermediate residual block in the branch recognition network will extract features of the target face image in turn and perform global average pooling. Each intermediate residual block corresponds to a branch, and each branch can eventually output a corresponding feature map.

[0054] Step S304: taking the feature graphs of each branch together as the first expression recognition result.

[0055] Exemplarily, the expression feature maps output by each branch in the branch recognition network are collectively used as the first expression recognition result of the branch recognition network.

[0056] In this embodiment, using a residual network as a branch recognition network can further improve the accuracy of the expression recognition model, thereby improving the accuracy of the entire facial expression recognition.

[0057] In an exemplary embodiment, the branch recognition network also includes a target residual block connected to the fully connected layer of the branch recognition network, and the target residual block can also be understood as the last residual block in the entire branch recognition network. In other words, the intermediate residual block and the target residual block together constitute the branch recognition network. Exemplarily, the main recognition network obtains the target feature map output by the target residual block, that is, the expression feature map output by the target residual block, as the input of the main recognition network.

[0058] In an exemplary embodiment, as shown in FIG4 , expression recognition is performed on the target face image through the main recognition network to obtain a second expression recognition result, including:

[0059] Step S402: input the target feature map into the main recognition network, and perform pooling processing on the target feature map through the main recognition network to obtain a pooled feature map.

[0060] Among them, pooling is an operation to compress images, which is used to reduce the dimension of feature maps, improve the abstraction ability of the network, reduce the amount of calculation and prevent overfitting. The pooled feature map is the feature map obtained after the target feature map is pooled.

[0061] Exemplarily, after the main recognition network obtains the target feature map output by the target residual block in the branch recognition network, the target feature map can be subjected to maximum pooling processing and average pooling processing, thereby obtaining a feature map obtained by maximum pooling processing and a feature map obtained by average pooling processing. The feature maps obtained by the two pooling processes are combined through a concat operation to form a pooled feature map.

[0062] In one example, the maximum pooling process selects the maximum value in the local area as the output value, thereby retaining the most significant expression features. The average pooling process calculates the average value of all pixel values ​​in the local area as the output, so as to capture the average state of the overall expression features. Through these two pooling processes in this embodiment, even if the input features undergo a slight displacement or offset, the output features after the pooling operation can remain stable and not be affected by these slight changes, thereby improving the accuracy of expression recognition. The concat operation can be used to connect two feature maps together in a certain dimension to generate a feature map with a larger number of channels or dimensions, so as to better capture the relationship between different features in subsequent layers and improve the richness and diversity of features. ‌

[0063] Step S404: determine the channel attention weight based on the pooled feature map.

[0064] Among them, channel attention weights refer to the weights on the feature maps of each channel to emphasize the channels that contribute most to the task and suppress irrelevant or redundant channels.

[0065] Exemplarily, after the main recognition network obtains the pooled feature map, a convolution operation can be performed on the pooled feature map. In this embodiment, three 3*3 convolution operations are selected. Compared with the 7*7 convolution kernel, this embodiment can reduce more parameters, and multiple convolution kernels also provide more activation functions, increasing the nonlinear transformation capability of the network. The feature map generated by the convolution operation is dimensionally permuted, which can also be understood as dimensional exchange to obtain a permuted feature map. A global adaptive pooling operation is performed on the permuted feature map to output a one-dimensional feature map. Based on the one-dimensional feature map, the corresponding channel attention can be obtained.

[0066] Step S406, multiply the channel attention weight by the target feature map to obtain a channel attention feature map, and use the channel attention feature map as the second expression recognition result.

[0067] Among them, the channel attention feature map refers to the feature map processed by the channel attention mechanism.

[0068] For example, after calculating the channel attention weight, the channel attention weight can be multiplied element by element with each channel in the original feature map, that is, the target feature map, to obtain the channel attention feature map after attention weighting. In this way, important channel features are amplified, while unimportant channel features are suppressed, which improves the expression quality of the features and thus improves the accuracy of expression recognition.

[0069] In this embodiment, by introducing the main recognition network with the attention mechanism for expression recognition, the model can be guided to pay more attention to the key areas in the facial expressions and quantify the contribution of different areas to the expression recognition task, thereby improving the model's ability to extract expression features and further improving the accuracy of expression recognition.

[0070] In an exemplary embodiment, as shown in FIG5 , a pooling process is performed on the target feature map to obtain a pooled feature map, including:

[0071] Step S502, performing pooling processing on the target feature map in the image width direction to obtain a first feature image of the target feature map in the image width direction.

[0072] Among them, the first feature image refers to the feature image obtained after the target feature map is pooled in the image width direction, and the first feature image includes the feature map obtained by performing maximum pooling processing on the target feature map in the image width direction, and the feature map obtained by performing average pooling processing on the target feature map in the image width direction.

[0073] Exemplarily, the main recognition network may perform maximum pooling and average pooling on the input target feature map in the image width direction, or in the image width dimension, to obtain two feature maps, and the two feature maps may be combined to obtain a first feature image.

[0074] Step S504, performing pooling processing on the target feature map in the image height direction to obtain a second feature image of the target feature map in the image height direction.

[0075] Step S506: The first feature image and the second feature image are used together as a pooled feature image.

[0076] Among them, the second feature image refers to the feature image obtained after the target feature map is pooled in the direction of the image height, and the second feature image includes the feature map obtained by performing maximum pooling processing on the target feature map in the direction of the image height, and the feature map obtained by performing average pooling processing on the target feature map in the direction of the image height.

[0077] Exemplarily, the main recognition network may also perform maximum pooling and average pooling on the input target feature map in the image height direction, or in the image height dimension, to obtain two feature maps, and the two feature maps are combined to obtain a second feature image. The first feature image and the second feature image are used together as the pooled feature map.

[0078] In this embodiment, by performing pooling processing on the height dimension and the width dimension on the target feature map, the features on the height dimension and the width dimension are integrated as much as possible to enhance the expressive power of the features, thereby more accurately capturing the facial expression features.

[0079] In an exemplary embodiment, as shown in FIG6 , determining the channel attention weight based on the pooled feature map includes:

[0080] Step S602: Aggregate the first feature image and the second feature image to obtain an aggregated feature image.

[0081] The aggregated feature map refers to a feature map obtained by aggregating the first feature image and the second feature image.

[0082] Exemplarily, the main recognition network averages and aggregates the first feature image and the second feature image between channels, thereby obtaining a feature map after channel aggregation.

[0083] Step S604: Generate channel attention weights based on the aggregated feature map.

[0084] For example, the feature map obtained after the first feature image and the second feature image are aggregated actually represents the channel attention weight. This is because in this process, the main recognition network learns the importance of each channel by comparing and integrating the information from the first feature image and the second feature image. Each element of this aggregated feature map corresponds to the attention weight of a channel.

[0085] In this embodiment, the channel attention weight is obtained by aggregating the first feature image and the second feature image, thereby improving the accuracy of the channel attention weight, thereby improving the accuracy of expression recognition.

[0086] In an exemplary embodiment, as shown in FIG7 , the training process of the expression recognition model includes:

[0087] Step S702, obtaining multiple facial expression images for model training.

[0088] Among them, facial expression images may refer to sample expression images used for model training, and these images carry labels of the corresponding expressions.

[0089] Exemplarily, when receiving an instruction to train an expression recognition model, the server may first obtain or construct multiple facial expression images as a training set for model training. These images may be labeled with the expression categories to which they belong to verify the model output results.

[0090] Step S704, based on the facial expression image and the attention mechanism, the first pre-trained network is trained until the network loss value of the first pre-trained network is lower than the loss threshold, thereby obtaining the main recognition network.

[0091] The first pre-trained network refers to the initial state of the main recognition network. The network loss value can be the value of the loss function of the first pre-trained network. The loss function of the first pre-trained network can include a softmax loss function and a loss function based on an attention mechanism. The expression can be:

[0092]

[0093] in, Represents the ‌softmax loss loss function, which is a combination of the softmax function and the cross entropy loss function. The softmax function converts the output of the network into a set of probability values, which are between 0 and 1, and the sum of all values ​​is 1. The cross entropy loss function is used to calculate the difference between the predicted probability distribution and the true label. represents a hyperparameter used to balance the size of these two losses. represents the loss function based on the attention mechanism, The expression can be:

[0094] L center = 1 2 ∑ i =1 n [D× x i - d y i ] 2

[0095] Among them, i represents the i-th feature vector, D represents the aggregate feature map, represents the feature center, which calculates half of the sum of the squares of the Euclidean distances between each feature vector and the corresponding feature center. The loss threshold is the error threshold defined in advance for the loss function of the first pre-trained network.

[0096] Exemplarily, the server trains the first pre-trained network based on the acquired facial expression image and the attention mechanism until the network loss value of the first pre-trained network is lower than the loss threshold, and then the main recognition network can be obtained. Of course, other training stop conditions can also be defined in practical applications, such as loss function convergence or reaching the number of training times.

[0097] Step S706, inputting the facial expression image into the main recognition network, performing facial expression prediction, and obtaining expression prediction results.

[0098] The expression prediction result refers to the result output by the main recognition network for predicting the facial expression of the facial expression image, which may be the category and probability of the expression to which the facial expression image belongs.

[0099] For example, after the main recognition network is trained, the branch recognition network can be trained. The server inputs the facial expression image into the main recognition network, and the main recognition network predicts the facial expression of the facial expression image, thereby obtaining the expression prediction result. The expression prediction result is used as the training data of the branch recognition network to transfer the knowledge learned by the main recognition network to the branch recognition network, thereby improving the accuracy of the branch recognition network.

[0100] Step S708: Based on the expression prediction result, the second pre-trained network is trained until the training stop condition is met to obtain a branch recognition network.

[0101] Step S710: the main recognition network and the branch recognition network together form an expression recognition model.

[0102] The second pre-trained network refers to the initial state of the branch recognition network. The training stop condition refers to the prerequisite for stopping the training of the branch recognition network, which can be reaching the number of training times or the convergence of the loss function of the branch recognition network. The specific training stop condition can be set according to the actual situation.

[0103] Exemplarily, the server inputs the face prediction result normally output by the main recognition network into the second pre-trained network to train the second pre-trained network until the training stop condition is met to obtain a branch recognition network. The main recognition network and the branch recognition network together constitute an expression recognition model for expression recognition.

[0104] In this embodiment, training the main recognition network based on the attention mechanism can guide the model to pay more attention to the key areas in the facial expressions and quantify the contribution of different areas to the expression recognition task, thereby improving the model's ability to extract expression features and further improving the accuracy of expression recognition. The normal prediction results of the main recognition network are further used to train the branch recognition network so that the branch recognition network can learn more abundant knowledge from the main recognition network. The combination of the two can enhance the generalization ability of the expression recognition model, so that it can maintain good recognition performance in different scenarios and conditions.

[0105] In a specific embodiment, as shown in FIG8 , the process of the expression recognition method includes: in response to an expression recognition instruction for a target face image, the target face image is input into an expression recognition model, and the expression recognition model includes a main recognition network and a branch recognition network. The main recognition network is trained based on the attention mechanism. The branch recognition network is trained based on the output result of the main recognition network.

[0106] As shown in Figure 9, the branch recognition network includes multiple intermediate residual blocks, namely residual block 1, residual block 2, residual block 3, and a target residual block, namely residual block 4. Residual block 1, residual block 2, and residual block 3 correspond to branch 1, branch 2, and branch 3, respectively. Residual block 1, residual block 2, and residual block 3 extract features from the target face image, respectively, and obtain output 2, output 3, and output 4, which are collectively used as the first expression recognition result. .

[0107] Residual block 4 outputs a C*H*W target feature map and enters the main recognition network. As shown in Figure 10, the main recognition network first converts the target feature map in channel C and width W to obtain a H*C*W feature map A. Then, maximum pooling and average pooling are performed in the image height direction to obtain two 1*C*W feature maps. The two feature maps are combined through the concat operation to obtain a 2*C*W feature map. For the obtained 2*C*W feature map, after 3 3*3 convolution operations, a 1*C*W feature map is obtained. Next, a permutation operation is performed to transform the dimension of the 1*C*W feature map to obtain a C*1*W feature map. Through the global adaptive pooling operation, the C*1*W feature map is output as a C*1*1 feature map A1, which is the second feature image. Similarly, the target feature map is converted in channel C and height H to obtain a W*H*C feature map B. Then, the maximum pooling and average pooling are performed in the width direction of the image. After the same operation as above, a C*1*1 feature map B1, i.e., the first feature image, is obtained. The feature maps A1 and B1 are averaged and aggregated to obtain an aggregated feature map representing the channel attention weight. Finally, the channel attention weight is element-wise multiplied with each channel in the target feature map to obtain the channel attention feature map, i.e., output 1, which is the second expression recognition result.

[0108] Finally, output 1, output 2, output 3 and output 4 are weightedly summed to obtain the expression recognition result of the target face image.

[0109] In this embodiment, the server responds to the expression recognition instruction for the target face image initiated by the terminal, and inputs the target face image into the expression recognition model including the main recognition network and the branch recognition network. Since the main recognition network is trained based on the attention mechanism, the attention mechanism can guide the model to pay more attention to the key areas in the facial expressions, and quantify the contribution of different areas to the expression recognition task, thereby improving the model's ability to extract expression features, and then improve the accuracy of expression recognition. The branch recognition network is trained based on the output results of the main recognition network. The branch recognition network can learn more abundant knowledge from the main recognition network. The combination of the two can enhance the generalization ability of the expression recognition model, so that it can maintain good recognition performance in different scenarios and conditions. Finally, the expression recognition results output by the branch recognition network and the main recognition network are weighted and summed to obtain the expression recognition result of the target face image. Therefore, this embodiment can effectively improve the accuracy of facial expression recognition.

[0110] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiment of the present application also provides an expression recognition device for implementing the expression recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more expression recognition device embodiments provided below can refer to the limitations on the expression recognition method above, and will not be repeated here.

[0112] In an exemplary embodiment, as shown in FIG11 , an expression recognition device is provided, including: a model input module 1102, for inputting a target face image into an expression recognition model in response to an expression recognition instruction for the target face image; the expression recognition model includes a main recognition network and a branch recognition network; the main recognition network is obtained by training based on an attention mechanism; the branch recognition network is obtained by training based on the output result of the main recognition network; a first expression recognition module 1104, for performing expression recognition on the target face image through the branch recognition network to obtain a first expression recognition result; a second expression recognition module 1106, for performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result; and a model output module 1108, for performing weighted summation of the first expression recognition result and the second expression recognition result to obtain an expression recognition result of the target face image.

[0113] In one embodiment, the branch recognition network includes multiple intermediate residual blocks; the first expression recognition module 1104 is also used to: extract features of the target facial image through each intermediate residual block respectively, to obtain branch feature maps output by each intermediate residual block; and use each branch feature map together as the first expression recognition result.

[0114] In one embodiment, the branch recognition network also includes a target residual block connected to the fully connected layer of the branch recognition network; the device is also used to: obtain a target feature map output by the target residual block; the second expression recognition module 1106 is also used to: input the target feature map into the main recognition network, and pool the target feature map through the main recognition network to obtain a pooled feature map; based on the pooled feature map, determine the channel attention weight; multiply the channel attention weight by the target feature map to obtain a channel attention feature map, and use the channel attention feature map as the second expression recognition result.

[0115] In one of the embodiments, the second expression recognition module 1106 is also used to: perform pooling processing on the target feature map in the image width direction to obtain a first feature image of the target feature map in the image width direction; perform pooling processing on the target feature map in the image height direction to obtain a second feature image of the target feature map in the image height direction; and use the first feature image and the second feature image together as the pooled feature map.

[0116] In one embodiment, the second expression recognition module 1106 is further used to: aggregate the first feature image and the second feature image to obtain an aggregated feature map; and generate a channel attention weight based on the aggregated feature map.

[0117] In one of the embodiments, the device is also used to: obtain multiple facial expression images for model training; based on the facial expression images and the attention mechanism, train the first pre-trained network until the network loss value of the first pre-trained network is lower than the loss threshold, and obtain the main recognition network; input the facial expression images into the main recognition network, perform facial expression prediction, and obtain expression prediction results; based on the expression prediction results, train the second pre-trained network until the training stop condition is met, and obtain a branch recognition network; the main recognition network and the branch recognition network together constitute an expression recognition model.

[0118] Each module in the above-mentioned expression recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0119] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG12. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store expression recognition data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an expression recognition method is implemented.

[0120] Those skilled in the art will understand that the structure shown in FIG. 12 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: in response to an expression recognition instruction for a target facial image, the target facial image is input into an expression recognition model; the expression recognition model includes a main recognition network and a branch recognition network; the main recognition network is trained based on an attention mechanism; the branch recognition network is trained based on the output result of the main recognition network; expression recognition is performed on the target facial image through the branch recognition network to obtain a first expression recognition result; expression recognition is performed on the target facial image through the main recognition network to obtain a second expression recognition result; and the first expression recognition result and the second expression recognition result are weightedly summed to obtain an expression recognition result of the target facial image.

[0122] In one embodiment, when the processor executes the computer program, the following steps are also implemented: feature extraction is performed on the target facial image through each intermediate residual block respectively to obtain a branch feature map output by each intermediate residual block; and each branch feature map is collectively used as the first expression recognition result.

[0123] In one embodiment, the processor further implements the following steps when executing the computer program: obtaining a target feature map output by the target residual block; performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result, including: inputting the target feature map into the main recognition network, and performing pooling processing on the target feature map through the main recognition network to obtain a pooled feature map; determining the channel attention weight based on the pooled feature map; multiplying the channel attention weight by the target feature map to obtain a channel attention feature map, and using the channel attention feature map as the second expression recognition result.

[0124] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing pooling processing on the target feature map in the image width direction to obtain a first feature image of the target feature map in the image width direction; performing pooling processing on the target feature map in the image height direction to obtain a second feature image of the target feature map in the image height direction; and using the first feature image and the second feature image together as the pooling feature map.

[0125] In one embodiment, when the processor executes the computer program, the following steps are also implemented: aggregating the first feature image and the second feature image to obtain an aggregated feature map; and generating a channel attention weight based on the aggregated feature map.

[0126] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining multiple facial expression images for model training; training the first pre-trained network based on the facial expression images and the attention mechanism until the network loss value of the first pre-trained network is lower than the loss threshold, thereby obtaining a main recognition network; inputting the facial expression images into the main recognition network, performing facial expression prediction, and obtaining expression prediction results; based on the expression prediction results, training the second pre-trained network until the training stop condition is met, thereby obtaining a branch recognition network; the main recognition network and the branch recognition network together constitute an expression recognition model.

[0127] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: in response to an expression recognition instruction for a target facial image, the target facial image is input into an expression recognition model; the expression recognition model includes a main recognition network and a branch recognition network; the main recognition network is obtained by training based on an attention mechanism; the branch recognition network is obtained by training based on the output result of the main recognition network; the expression recognition of the target facial image is performed through the branch recognition network to obtain a first expression recognition result; the expression recognition of the target facial image is performed through the main recognition network to obtain a second expression recognition result; the first expression recognition result and the second expression recognition result are weightedly summed to obtain the expression recognition result of the target facial image.

[0128] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: feature extraction is performed on the target facial image through each intermediate residual block respectively to obtain branch feature maps output by each intermediate residual block; and each branch feature map is collectively used as the first expression recognition result.

[0129] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a target feature map output by the target residual block; performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result, including: inputting the target feature map into the main recognition network, and performing pooling processing on the target feature map through the main recognition network to obtain a pooled feature map; determining the channel attention weight based on the pooled feature map; multiplying the channel attention weight by the target feature map to obtain a channel attention feature map, and using the channel attention feature map as the second expression recognition result.

[0130] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: performing pooling processing on the target feature map in the image width direction to obtain a first feature image of the target feature map in the image width direction; performing pooling processing on the target feature map in the image height direction to obtain a second feature image of the target feature map in the image height direction; and using the first feature image and the second feature image together as the pooling feature map.

[0131] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: aggregating the first feature image and the second feature image to obtain an aggregated feature map; and generating a channel attention weight based on the aggregated feature map.

[0132] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining multiple facial expression images for model training; training the first pre-trained network based on the facial expression images and the attention mechanism until the network loss value of the first pre-trained network is lower than the loss threshold, thereby obtaining a main recognition network; inputting the facial expression images into the main recognition network, performing facial expression prediction, and obtaining expression prediction results; based on the expression prediction results, training the second pre-trained network until the training stop condition is met, thereby obtaining a branch recognition network; the main recognition network and the branch recognition network together constitute an expression recognition model.

[0133] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: in response to an expression recognition instruction for a target facial image, the target facial image is input into an expression recognition model; the expression recognition model includes a main recognition network and a branch recognition network; the main recognition network is trained based on an attention mechanism; the branch recognition network is trained based on the output result of the main recognition network; expression recognition is performed on the target facial image through the branch recognition network to obtain a first expression recognition result; expression recognition is performed on the target facial image through the main recognition network to obtain a second expression recognition result; and the first expression recognition result and the second expression recognition result are weightedly summed to obtain an expression recognition result of the target facial image.

[0134] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: feature extraction is performed on the target facial image through each intermediate residual block respectively to obtain branch feature maps output by each intermediate residual block; and each branch feature map is collectively used as the first expression recognition result.

[0135] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a target feature map output by the target residual block; performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result, including: inputting the target feature map into the main recognition network, and performing pooling processing on the target feature map through the main recognition network to obtain a pooled feature map; determining the channel attention weight based on the pooled feature map; multiplying the channel attention weight by the target feature map to obtain a channel attention feature map, and using the channel attention feature map as the second expression recognition result.

[0136] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: performing pooling processing on the target feature map in the image width direction to obtain a first feature image of the target feature map in the image width direction; performing pooling processing on the target feature map in the image height direction to obtain a second feature image of the target feature map in the image height direction; and using the first feature image and the second feature image together as the pooling feature map.

[0137] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: aggregating the first feature image and the second feature image to obtain an aggregated feature map; and generating a channel attention weight based on the aggregated feature map.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining multiple facial expression images for model training; training the first pre-trained network based on the facial expression images and the attention mechanism until the network loss value of the first pre-trained network is lower than the loss threshold, thereby obtaining a main recognition network; inputting the facial expression images into the main recognition network, performing facial expression prediction, and obtaining expression prediction results; based on the expression prediction results, training the second pre-trained network until the training stop condition is met, thereby obtaining a branch recognition network; the main recognition network and the branch recognition network together constitute an expression recognition model.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0140] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0142] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for facial expression recognition, characterized in that: The method comprises: In response to an expression recognition instruction for a target face image, the target face image is input into an expression recognition model; the expression recognition model includes a main recognition network and a branch recognition network; the main recognition network is trained based on an attention mechanism; the branch recognition network is trained based on an output result of the main recognition network; Performing expression recognition on the target face image through the branch recognition network to obtain a first expression recognition result; Performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result; The first expression recognition result and the second expression recognition result are weightedly summed to obtain the expression recognition result of the target facial image.

2. The method according to claim 1, characterized in that The branch recognition network includes a plurality of intermediate residual blocks; the expression recognition is performed on the target face image through the branch recognition network to obtain a first expression recognition result, including: Performing feature extraction on the target face image through each of the intermediate residual blocks to obtain branch feature maps output by each of the intermediate residual blocks; The branch feature graphs are collectively used as the first expression recognition result.

3. The method according to claim 1, characterized in that The branch recognition network also includes a target residual block connected to the fully connected layer of the branch recognition network; the method also includes: Obtaining a target feature map output by the target residual block; The step of performing expression recognition on the target face image through the main recognition network to obtain a second expression recognition result includes: Inputting the target feature map into the main recognition network, and performing pooling processing on the target feature map through the main recognition network to obtain a pooled feature map; Based on the pooled feature map, determining a channel attention weight; The channel attention weight is multiplied by the target feature map to obtain a channel attention feature map, and the channel attention feature map is used as the second expression recognition result.

4. The method according to claim 3, characterized in that: The performing pooling processing on the target feature map to obtain a pooled feature map includes: Performing pooling processing on the target feature map in the image width direction to obtain a first feature image of the target feature map in the image width direction; Performing pooling processing on the target feature map in the image height direction to obtain a second feature image of the target feature map in the image height direction; The first feature image and the second feature image are used together as a pooling feature map.

5. The method according to claim 4, characterized in that The determining the channel attention weight based on the pooled feature map includes: Aggregating the first feature image and the second feature image to obtain an aggregated feature image; Based on the aggregated feature map, a channel attention weight is generated.

6. The method according to claim 1, characterized in that The training process of the expression recognition model includes: Obtain multiple facial expression images for model training; Based on the facial expression image and the attention mechanism, training the first pre-trained network until the network loss value of the first pre-trained network is lower than the loss threshold, thereby obtaining a main recognition network; Inputting the facial expression image into the main recognition network to perform facial expression prediction to obtain an expression prediction result; Based on the expression prediction result, the second pre-trained network is trained until a training stop condition is met to obtain a branch recognition network; The main recognition network and the branch recognition network together constitute the expression recognition model.

7. An expression recognition device, characterized in that: The device comprises: A model input module, for responding to an expression recognition instruction for a target face image, inputting the target face image into an expression recognition model; the expression recognition model comprises a main recognition network and a branch recognition network; the main recognition network is obtained by training based on an attention mechanism; the branch recognition network is obtained by training based on an output result of the main recognition network; A first expression recognition module, used to perform expression recognition on the target face image through the branch recognition network to obtain a first expression recognition result; A second expression recognition module, used to perform expression recognition on the target face image through the main recognition network to obtain a second expression recognition result; The model output module is used to perform weighted summation on the first expression recognition result and the second expression recognition result to obtain the expression recognition result of the target facial image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images