Facial expression recognition method and device

Abstract

The present disclosure provides a facial expression recognition method and device, which includes the steps of: obtaining a recognition target image; obtaining an image feature map of the recognition target image based on the recognition target image; determining global feature information and local feature information based on the image feature map; and determining a facial expression type of the recognition target image based on the global feature information and the local feature information. Because the global feature information of the recognition target image reflects the overall face information, and the local feature information of the recognition target image reflects the detailed information of each region of the face, the effective combination of the local feature information and the global feature information can restore more detailed information of the image of facial expressions, and thus the facial expression type of the recognition target image determined based on the global feature information and the local feature information is more accurate, that is, the accuracy of the recognition result of the facial expression type corresponding to the detection target image can be improved, and the user experience can be further improved.

Description

[Technical field]

[0001] The present disclosure relates to the computer technology field, and in particular to a facial expression recognition method and apparatus. [Background technology]

[0002] With the development of image processing technology, image recognition technology is used in more and more scenarios. Current image recognition technology has achieved very good results in facial expression recognition, but in complex scenarios, for example, due to important parts of the face being occluded or due to different body postures (such as turning the body sideways), the complete face cannot be collected, so facial expressions cannot be recognized or the facial expression recognition results are inaccurate. Therefore, a new solution for facial expression recognition is urgently needed. Summary of the Invention

[0003] In view of this, the embodiments disclosed herein provide a facial expression recognition method, an apparatus, a computer device, and a computer-readable storage medium, so as to solve the problem in the prior art that in complex scenarios, the complete face cannot be collected, for example, due to important parts of the face being occluded or due to different body postures (e.g., turning the body sideways), and thus facial expressions cannot be recognized or the facial expression recognition results are inaccurate.

[0004] In a first aspect of the disclosed embodiment, acquiring a recognition target image; obtaining an image feature map of the recognition target image based on the recognition target image; determining global and local feature information based on the image feature map; and identifying an expression type of the recognition target image based on the global feature information and the local feature information.

[0005] In a second aspect of the disclosed embodiment, an image acquisition module for acquiring a recognition target image; A first feature acquisition module for acquiring an image feature map of the recognition target image based on the recognition target image; a second feature acquisition module for identifying global feature information and local feature information based on the image feature map; and a facial expression type identification module for identifying a facial expression type of an image to be recognized based on global feature information and local feature information.

[0006] In a third aspect of an embodiment of the present disclosure, there is provided a computing device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the computing device implementing the steps of the method when the processor executes the computer program.

[0007] In a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the above method when executed by a processor.

[0008] Beneficial effects of the embodiment of the present disclosure compared to the prior art: the embodiment of the present disclosure can first obtain a recognition target image, then obtain an image feature map of the recognition target image based on the recognition target image, then characterize the global feature information and local feature information based on the image feature map, and finally identify the facial expression type of the recognition target image based on the global feature information and the local feature information. Because the global feature information of the recognition target image reflects the overall face information, and the local feature information of the recognition target image reflects the detailed information of each region of the face, the effective combination of the local feature information and the global feature information can restore more image details of facial expressions, thus making the facial expression type of the recognition target image identified based on the global feature information and the local feature information more accurate, that is, the accuracy of the recognition result of the facial expression type corresponding to the detection target image can be improved, and the user experience can be further improved. [Brief description of the drawings]

[0009] In order to more clearly explain the technical solutions in the embodiments of the present disclosure, the following briefly introduces drawings necessary for describing the embodiments or prior art. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without the need for creative work. [Figure 1] FIG. 2 is a scenario schematic diagram of an application scenario of an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of a facial expression recognition method provided in an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a schematic diagram of the network architecture of a facial expression recognition model provided in an embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a GoF modulation process provided in an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a schematic diagram of a network architecture of a Local-Global Attention structure provided in an embodiment of the present disclosure. [Figure 6]FIG. 2 is a schematic diagram of a network architecture of a global model provided in an embodiment of the present disclosure. [Figure 7] FIG. 2 is a schematic diagram of the network architecture of the Split-Attention module provided in an embodiment of the present disclosure. [Figure 8] FIG. 2 is a schematic diagram of a network architecture of a weight layer provided in an embodiment of the present disclosure. [Figure 9] FIG. 1 is a block diagram of a facial expression recognition device provided in an embodiment of the present disclosure. [Figure 10] FIG. 2 is a schematic diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In the following description, specific details, such as specific system structures, techniques, etc., are provided for the purpose of explanation, not limitation, to thoroughly understand the embodiments of the present disclosure. However, those skilled in the art should understand that the present disclosure can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present disclosure with unnecessary details.

[0011] Hereinafter, a facial expression recognition method and device according to an embodiment of the present disclosure will be described in detail with reference to the drawings.

[0012] In the prior art, in complex scenarios, the complete face cannot be collected, for example, due to important parts of the face being occluded or due to different body postures (e.g., turning the body sideways), so facial expressions cannot be recognized or the facial expression recognition results are inaccurate. Therefore, a new facial expression recognition solution is urgently needed.

[0013] In order to solve the above problem, the present invention provides a facial expression recognition method, in which the method provided in this embodiment can first obtain a recognition target image, and then obtain an image feature map of the recognition target image based on the recognition target image, and then obtain global feature information and local feature information based on the image feature map, and finally identify the facial expression type of the recognition target image based on the global feature information and the local feature information. Because the global feature information of the recognition target image reflects the overall face information, and the local feature information of the recognition target image reflects the detailed information of each region of the face, the effective combination of the local feature information and the global feature information can restore more image details of facial expressions, and thus the facial expression type of the recognition target image identified based on the global feature information and the local feature information is more accurate, that is, the accuracy of the recognition result of the facial expression type corresponding to the detection target image can be improved, and the user experience can be further improved.

[0014] By way of example, an embodiment of the invention can be applied to the application scenario depicted in Figure 1. This scenario may comprise a terminal device 1 and a server 2.

[0015] The terminal device 1 may be hardware or software. When the terminal device 1 is hardware, it may be various electronic devices that have the function of collecting images and storing images and support communication with the server 2, including, but not limited to, smartphones, tablets, laptop computers, digital cameras, monitors, video recorders, and desktop computers, and when the terminal device 1 is software, it may be installed in, for example, the electronic devices described above. The terminal device 1 may be realized as multiple software or software modules, or may be realized as a single software or software module, and the embodiments of the present disclosure are not limited thereto. Furthermore, various apps, such as an image collection app, an image storage app, and a live chat app, may be installed in the terminal device 1.

[0016] The server 2 may be a server providing various services, such as a back-end server receiving a bill sent by a terminal device that establishes a communication connection with the server, and the back-end server may process the bill sent by the terminal device, such as receiving and analyzing, to generate a processing result. The server 2 may be a single server, a server cluster consisting of several servers, or a cloud computing service center, and the embodiments disclosed herein are not limited thereto.

[0017] The server 2 may be hardware or software. If the server 2 is hardware, it may be various electronic devices that provide various services to the terminal device 1. If the server 2 is software, it may be multiple software programs or software modules that provide various services to the terminal device 1, or it may be a single software program or software module that provides various services to the terminal device 1, and the embodiment of the present disclosure is not limited to this.

[0018] The terminal device 1 and the server 2 may be communicatively connected via a network. The network may be a wired network connected using a coaxial cable, a twisted pair, or an optical fiber, or may be a wireless network capable of realizing interconnection of various communication devices without the need for wiring, such as Bluetooth, Near Field Communication (NFC), or Infrared, and the embodiments disclosed herein are not limited thereto.

[0019] Specifically, the user specifies a detection target image by the terminal device 1 and transmits the detection target image to the server 2. After the server 2 receives the detection target image, the server 2 can extract an image feature map of the recognition target image. Then, the server 2 can specify global feature information and local feature information based on the image feature map. Next, the server 2 can specify the facial expression type of the recognition target image based on the global feature information and the local feature information. In this way, the global feature information of the recognition target image reflects the overall face information, and the local feature information of the recognition target image reflects the detailed information of each region of the face, so that the effective combination of the local feature information and the global feature information can restore more image detailed information of facial expressions, and thus the facial expression type of the recognition target image specified based on the global feature information and the local feature information is more accurate, that is, the accuracy of the recognition result of the facial expression type corresponding to the detection target image can be improved, and the user experience can be further improved.

[0020] It should be noted that the specific types, numbers and combinations of the terminal device 1, the server 2 and the network can be adjusted according to the actual needs of the application scenario, and the embodiments disclosed herein are not limited thereto.

[0021] It should be noted that the above application scenarios are merely exemplified for easy understanding of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure may be used in any scenario of application.

[0022] Fig. 2 is a flowchart of a facial expression recognition method provided in an embodiment of the present disclosure. The facial expression recognition method of Fig. 2 may be performed by the terminal device and / or the server of Fig. 1. As shown in Fig. 2, the facial expression recognition method includes the following steps:

[0023] S201: The recognition target image is acquired.

[0024] In this embodiment, an image or video frame requiring facial expression recognition may be referred to as a recognition target image, and a face requiring facial expression recognition may be referred to as a target face.

[0025] For example, the terminal device can provide a web page, and the user can upload an image on the web page and click a preset button to trigger facial expression recognition on the image, and at this time, the image can be set as a recognition target image. Of course, the user can also obtain the recognition target image by taking a photo using the terminal device. In addition, the user needs to specify the target face that needs to be recognized, or the system can set the target face that needs to be recognized as a default setting.

[0026] S202: An image feature map of the recognition target image is obtained based on the recognition target image.

[0027] After obtaining the target image, an image feature map of the target image may be extracted for the target image. As can be understood, the image feature map of the target image is a complete feature map extracted from the target image, and can reflect the entire image feature information of the target image.

[0028] S203: Identify global feature information and local feature information based on the image feature map.

[0029] In complex scenarios, even if important parts of the face are occluded, or due to different body postures (e.g., turning the body sideways), the complete face cannot be collected, and the facial expression type cannot be recognized, or the recognition of the facial expression type is inaccurate. To solve the above problem, in this embodiment, after obtaining the image feature map of the recognition target image, the global feature information and local feature information of the target face can be obtained from the image feature map based on the image feature map. The global feature information of the recognition target image can reflect the overall information of the face, that is, the global feature information of the recognition target image can be understood as the global information of the target face, such as the overall information of the skin color, contour, distribution of facial organs, etc., and the local feature information of the recognition target image can reflect the detailed information of each region of the face, that is, the local feature information can reflect the detailed features of the target face, such as organ features such as scars, moles, dimples, etc., and strange facial features. As can be seen, the global feature information is used for rough matching, and the local feature information is used for detailed matching. Therefore, the effective combination of the local feature information and the global feature information can restore more image details of facial expressions.

[0030] S204: Identify the facial expression type of the recognition target image based on the global feature information and the local feature information.

[0031] In this embodiment, after the global feature information and the local feature information are identified, more image details of facial expressions can be restored using the global feature information and the local feature information, and thus the image details of the restored facial expressions can be used to identify the expression type of the target face. In this way, the expression type of the recognition target image identified based on the global feature information and the local feature information can be made more accurate, and thus the accuracy of the recognition result of the expression type corresponding to the detection target image can be improved.

[0032] Beneficial effects of the embodiment of the present disclosure compared to the prior art: The embodiment of the present disclosure can first obtain a recognition target image, then obtain an image feature map of the recognition target image based on the recognition target image, then characterize the global feature information and local feature information based on the image feature map, and finally identify the facial expression type of the recognition target image based on the global feature information and the local feature information. Because the global feature information of the recognition target image reflects the overall face information, and the local feature information of the recognition target image reflects the detailed information of each region of the face, the effective combination of the local feature information and the global feature information can restore more detailed image information of facial expressions, thus making the facial expression type of the recognition target image identified based on the global feature information and the local feature information more accurate, that is, the accuracy of the recognition result of the facial expression type corresponding to the detection target image can be improved, and the user experience can be further improved. That is, in the facial expression type recognition process, while paying attention to the local features, the context information of the global features can also be introduced to significantly improve the recognition performance.

[0033] Next, a method for implementing S202 of "obtaining an image feature map of the recognition target image based on the recognition target image" will be described. In this embodiment, the corresponding method of FIG. 2 can be applied to a facial expression recognition model (shown in FIG. 3), and the facial expression recognition model may include a first neural network model, and S202 of "obtaining an image feature map of the recognition target image based on the recognition target image" may include the following steps:

[0034] A recognition target image is input to a first neural network model to obtain an image feature map of the recognition target image.

[0035] Wherein, the first neural network model comprises a number of first convolution modules, where the number of first convolution modules are connected in sequence, and each first convolution module comprises a filter, a batch normalization layer, an MP model and an activation function. For example, as shown in Fig. 3, the first neural network model comprises three first convolution modules, where each first convolution module comprises one filter (e.g., GC module), one batch normalization layer (i.e., BN layer), one MP model (i.e., MP, Max Pooling) and one activation function (i.e., ReLU).

[0036] Here, the GC module is a convolution kernel with an added Gabor filter, and it should be emphasized that in this embodiment, the GC module can be called GoF (i.e., the Gabor filter is a Gabor directional filter modulated to a normal convolution kernel), and the GoF can enhance the robustness (orientation invariance and scale invariance) of the network against changes in orientation and scale. It should be noted that the Gabor filter has excellent properties and can not only retain frequency domain information in an image, but also has strong robustness against geometric transformations such as orientation and size, while the conventional convolutional neural network does not have such properties, so the Gabor filter is suitable for facial expression recognition in scenarios where orientation and size change frequently, while facial expression recognition focuses more on local features such as nose and mouth compared to other face tasks, and the Gabor filter can extract richer local features compared to the conventional convolutional neural network. As can be seen, the GC module can enhance the robustness of the network against changes in orientation and scale after the Gabor filter is added, while reducing the number of network layers, thereby reducing the forward inference time of the model.

[0037] In this embodiment, the modulation process of GoF is as follows:

[0038] Gabor filters have U-direction and V-scale, and to incorporate the steerable properties into GCN, the direction information is encoded in the learning filter while the scale information is embedded in different convolution layers. In GoF, Gabor filters capture the direction and scale information, so the corresponding convolution functions are enhanced.

[0039] The filter trained by the backpropagation algorithm is called the training filter. Unlike the standard convolutional neural network, the filter trained by the GCN is three-dimensional in order to encode the directional channel. Assuming that the size of the training filter is N×W×W, W×W is the size of the filter and N represents the channel. In traditional CNN, the weights of each layer are C out ×C in When expressed as C × W × W, GCN is out ×C in ×N×W×W, C out and C in Let U denote the channels that output and input the feature map, respectively. In the forward convolution process, we match the number of channels of the feature map, and select N as U, that is, the modulation number of the Gabor filter directions of this learning filter. On a given scale V, we obtain the GoF through the modulation process. JPEG2025505437000002.jpg62170

[0040] In GoF, the value of ν increases with the number of layers, which means that the ratio of Gabor filters in GoF changes with the number of layers. At each scale, the size of GoF is U×N×W×W. However, for a given Gabor filter, we only save N×W×W learning filters, which means that we do not need to increase the number of parameters and obtain enhanced functionality by such modulation.

[0041] Next, a method for implementing S203 of "determining global feature information and local feature information based on the image feature map" will be described. In this embodiment, the corresponding method of Fig. 2 can be applied to a facial expression recognition model, and as shown in Fig. 3, the facial expression recognition model has a Local-Global Attention structure, and as shown in Fig. 5, the Local-Global Attention structure has a global model and a local model, and S203 of "determining global feature information and local feature information based on the image feature map" may include the following steps:

[0042] S203a: Input the image feature map into a global model to obtain global feature information.

[0043] The conv and feature layers in Figure 5 are specifically shown in Figure 6, that is, the global model includes a first convolution layer, an H-Sigmoid activation function layer (i.e., F in Figure 6), a channel attention module (CAM, i.e., Channel attention in Figure 6), a spatial attention module (SAM, i.e., Spatial attention in Figure 6) and a second convolution layer. As can be understood, the channel attention module and the spatial attention module may be understood as one CBAM attention structure.

[0044] Specifically, the image feature map can be input into the first convolution layer to obtain the first feature map. Then, the first feature map can be input into the H-Sigmoid activation function layer to obtain the second feature map, where the H-Sigmoid activation function layer can avoid exponential calculations, so that the calculation speed can be improved. Next, the second feature map can be input into the channel attention module to obtain the channel attention map, where the role of the channel attention module is to learn one weight for each channel, and then multiply the weight by the corresponding channel, that is, multiply the features of different channels in the second feature map by their corresponding weights, respectively, to obtain the channel attention map. Immediately after, the channel attention map can be input into the spatial attention module to obtain the spatial attention map, where the spatial attention module is for the channel attention map to perform attention calculation in the feature map space (i.e., the dimensions of width and height), that is, in the channel attention map, each local region in space learns one weight, and then multiplies the weight by the channel attention map to obtain the spatial attention map. Finally, the spatial attention map can be input into the second convolutional layer to obtain global feature information.

[0045] S203b: Input the image feature map into a local model to obtain local feature information.

[0046] In this embodiment, as shown in FIG. 5 , the local model may include N local feature extraction convolutional layers (i.e., a conv layer and a feature layer, where k and N are the same number), an attention module, and N is a positive integer greater than 1.

[0047] In this embodiment, first, N local image blocks can be generated based on the image feature map. For example, the image feature map can be randomly cropped to obtain N different local image blocks, each of which includes local information of different regions. Then, each local image block can be input to each local feature extraction convolution layer respectively to obtain N local feature maps, that is, each local feature extraction convolution layer outputs one local feature map.

[0048] Then, the N local feature maps can be input to an attention module to obtain local feature information. It should be noted that in this embodiment, the attention module may be a Split-Attention module, which is shown in FIG. 7, and the attention module may include a pooling layer (i.e., Global Pooling in FIG. 7), a second convolution module (i.e., Conv+BN+ReLU in FIG. 7, that is, the second convolution module includes a convolution layer, a BN layer and a ReLU function layer), N third convolution layers (i.e., Conv layer in FIG. 7) and a normalization layer (i.e., r-Softmax layer in FIG. 7). Specifically, as shown in FIG. 7, the N local feature maps are fused to obtain a fused feature map. The fused feature map is input to a pooling layer to obtain a pooled feature map. The pooled feature map is input to a second convolution module to obtain a processed local feature map. The processed local feature map is input to each third convolution layer respectively to obtain N sub-local feature maps, i.e., the processed local feature map is input to each independent third convolution layer processing to extract each sub-local feature map. For each sub-local feature map, the sub-local feature map is input to a normalization layer to obtain a weight value corresponding to the sub-local feature map, and the local feature map corresponding to the sub-local feature map is obtained based on the weight value corresponding to the sub-local feature map and the local feature map, i.e., the normalization layer is used to perform an r-Softmax operation on the sub-local feature map to obtain a weight value corresponding to the sub-local feature map, and then the local feature map corresponding to the sub-local feature map is multiplied by the weight value corresponding to the sub-local feature map to obtain a processed local feature map corresponding to the sub-local feature map. The N local feature maps (i.e., the processed local feature maps) are fused to obtain local feature information.

[0049] Next, a method for implementing S204 of “inputting the global feature information into the first global average pooling layer to obtain the global pooling feature information” will be described. In this embodiment, S204 may include the following steps:

[0050] S204a: Input the global feature information into a first global average pooling layer to obtain global pooling feature information.

[0051] In this embodiment, as shown in FIG. 5 , a first global average pooling layer (i.e., the GAP layer in FIG. 5 ) is further connected to the global model. After the global feature information is extracted, the first global average pooling layer can perform a global average pooling (GAP) process on the global feature information to obtain global pooling feature information.

[0052] S204b: Input the local feature information into a second global average pooling layer to obtain local pooling feature information.

[0053] In this embodiment, as shown in FIG. 5 , a second global average pooling layer (i.e., the GAP layer in FIG. 5 ) is further connected to the local model. After the local feature information is extracted, the second global average pooling layer performs global average pooling (GAP) processing on the local feature information to obtain local pooling feature information.

[0054] S204c: The global pooling feature information and the local pooling feature information are input to the fully connected layer to obtain the facial expression type of the recognition target image.

[0055] In this embodiment, after obtaining the global pooling feature information and the local pooling feature information, the global pooling feature information and the local pooling feature information can be firstly fused to obtain fused pooling feature information, and then the fused pooling feature information is input to the fully connected layer (i.e., FC in FIG. 3) to obtain the facial expression type of the recognition target image output by the fully connected layer.

[0056] In addition, in one implementation of this embodiment, the facial expression recognition model may include a weight layer (i.e., Weight branch in FIG. 3) as shown in FIG. 3, and the weight layer includes two fully connected layers (i.e., FC1 and FC2 in FIG. 8) and one Sigmoid activation layer (i.e., Sigmoid in FIG. 8) as shown in FIG. 8. Specifically, the corresponding method of FIG. 2 further includes the following steps:

[0057] Step 1: Input the global pooling feature information and the local pooling feature information into the weight layer to obtain the true weight value of the facial expression type of the recognition target image.

[0058] Here, the true weight value of the facial expression type of the recognition target image is intended to represent the probability that the facial expression type is the corresponding true facial expression type of the recognition target image, and in one implementation, the true weight value may be a weight value between 0 and 1. As can be seen, the larger the true weight value of the facial expression type of the recognition target image, the higher the probability that the facial expression type is the corresponding true facial expression type of the recognition target image, and conversely, the smaller the true weight value of the facial expression type of the recognition target image, the lower the probability that the facial expression type is the corresponding true facial expression type of the recognition target image.

[0059] Step 2: Identify the expression type detection result corresponding to the recognition target image based on the truth weight value of the expression type of the recognition target image and the expression type of the recognition target image.

[0060] In this embodiment, if the true weight value of the facial expression type of the recognition target image meets a predetermined condition, for example, if the true weight value of the facial expression type of the recognition target image is greater than a predetermined threshold, the facial expression type of the recognition target image can be determined as the corresponding facial expression type detection result of the recognition target image. The predetermined threshold may be preset, or may be set by a user according to actual needs. Conversely, if the true weight value of the facial expression type of the recognition target image does not meet the predetermined condition, for example, if the true weight value of the facial expression type of the recognition target image is less than or equal to a predetermined threshold, the facial expression type of the recognition target image does not need to be determined as the corresponding facial expression type detection result of the recognition target image, and the corresponding facial expression type detection result of the recognition target image can be determined as no result.

[0061] It should be noted that the facial expression recognition dataset of the prior art has a problem of labeling uncertainty. On the one hand, such uncertainty is caused by the subjectivity of the annotator, so there will be differences after the same photo is labeled by different annotators; on the other hand, due to the ambiguity of facial expression photos, that is, one expression photo may contain various different expressions, and factors such as occlusion, lighting, and image quality problems may also affect the labeling result. Since such data training with uncertain labeling is disadvantageous to learning effective facial expression features and causes a phenomenon of non-convergence in the early stage of network training, in this embodiment, the weight layer is trained using the loss function of Attention Weighted Loss. Specifically, first, after extracting the features, each sample learns one weight by one added weight branch (i.e., weight layer), and this weight represents the degree of certainty of the current sample, and when calculating the loss, the loss of the training sample is weighted according to the calculated weight.

[0062] In the process of model training, if the calculated weight value of the weight layer is smaller than the predetermined training value, it indicates that the certainty of the current training sample is very low, which is unfavorable to the optimization of the network, so these samples are filtered by an artificially set threshold, and the calculated weight value of the weight layer is equal to or greater than the predetermined training value, and the higher the value, the stricter the requirement for certainty, so that some samples between certainty and uncertainty are removed, and the lower the weight value, the more samples participate in the loss contribution, and the samples with low certainty (i.e., the calculated weight value of the weight layer is smaller than the predetermined training value) are not used to participate in the loss calculation, so that the accuracy rate of facial expression recognition can be significantly improved. After filtering the training samples, the features of all samples and their corresponding weights can be obtained, so that the cross entropy of each sample is calculated, and then the weighted sum is performed by these weights, or another weighting method is used, that is, the sample features taken are weighted by the calculated weight sum, and then the loss calculation is performed for the weighted features.

[0063] In addition, after adding a weight layer to the facial expression recognition model (i.e., adding one weight branch), the branch learns the certainty of each sample, and then uses Attention Weighted Loss to introduce the learned uncertainty into the Attention Weighted Loss, thereby suppressing the impact of uncertainty labeling on the facial expression recognition training process.

[0064] All the above-mentioned optional technical solutions may be combined in any combination to form optional embodiments of the present disclosure, and will not be described one by one here.

[0065] Below are apparatus embodiments of the present disclosure for carrying out the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0066] FIG. 9 is a schematic diagram of a facial expression recognition device provided in an embodiment of the present disclosure. As shown in FIG. 9, the facial expression recognition device includes: an image acquisition module 901 for acquiring a recognition target image; a first feature acquisition module 902 for acquiring an image feature map of the target image based on the target image; a second feature acquisition module 903 for identifying global and local feature information based on the image feature map; and an expression type identification module 904 for identifying an expression type of the recognition target image based on the global feature information and the local feature information.

[0067] In some embodiments, the method is applied to a facial expression recognition model, the facial expression recognition model includes a first neural network model, and the first feature acquisition module 902 specifically includes: inputting a recognition target image into a first neural network model to obtain an image feature map of the recognition target image; Here, the first neural network model includes a number of first convolution modules, the number of first convolution modules are connected in sequence, and each of the first convolution modules includes a filter, a batch normalization layer, an MP model, and an activation function.

[0068] In some embodiments, the method is applied to a facial expression recognition model, the facial expression recognition model includes a global model and a local model, and the second feature acquisition module 903 specifically includes: Input the image feature map into the global model to obtain global feature information, The image feature map is input to the local model to obtain local feature information.

[0069] In some embodiments, the global model includes a first convolution layer, an H-Sigmoid activation function layer, a channel attention module, a spatial attention module, and a second convolution layer, and the second feature acquisition module 903 specifically includes: Input the image feature map into the first convolutional layer to obtain the first feature map; The first feature map is input to the H-Sigmoid activation function layer to obtain the second feature map. Input the second feature map into a channel attention module to obtain a channel attention map; Input the channel attention map into the spatial attention module to obtain the spatial attention map; The spatial attention map is input to the second convolutional layer to obtain global feature information.

[0070] In some embodiments, the local model includes N local feature extraction convolution layers, an attention module, where N is a positive integer greater than 1, and the second feature acquisition module 903 specifically includes: Generate N local image blocks based on the image feature map; Input each local image block to each local feature extraction convolution layer respectively to obtain N local feature maps; The N local feature maps are input to the attention module to obtain local feature information.

[0071] In some embodiments, the attention module includes a pooling layer, a second convolution module, N third convolution layers, and a normalization layer, and the second feature acquisition module 903 specifically includes: Fuse the N local feature maps to obtain a fused feature map; Input the fused feature map into the pooling layer to obtain the pooled feature map; The pooled feature map is input to the second convolution module to obtain the processed local feature map; The processed local feature map is input to each third convolutional layer to obtain N sub-local feature maps; For each sub-local feature map, input the sub-local feature map into a normalization layer to obtain a weight value corresponding to the sub-local feature map; and obtain a local feature map corresponding to the sub-local feature map based on the weight value corresponding to the sub-local feature map and the local feature map; It is to fuse N local feature maps to obtain local feature information.

[0072] In some embodiments, the facial expression type identification module 904 includes: The global feature information is input into the first global average pooling layer to obtain global pooling feature information; The local feature information is input into the second global average pooling layer to obtain local pooling feature information; The global pooling feature information and the local pooling feature information are input to the fully connected layer to obtain the facial expression type of the recognition target image.

[0073] In some embodiments, the facial expression recognition model further comprises a weight layer, the weight layer comprising two fully connected layers and one Sigmoid activation layer, and the apparatus further comprises a result determination module; inputting the global pooling feature information and the local pooling feature information into a weight layer to obtain a true weight value of an expression type of a recognition target image, the true weight value of the expression type of the recognition target image being for representing a probability that the expression type is a corresponding true expression type of the recognition target image; The facial expression type detection result corresponding to the recognition target image is specified based on the true weight value of the facial expression type of the recognition target image and the facial expression type of the recognition target image.

[0074] Beneficial effects of the embodiment of the present disclosure compared to the prior art: The facial expression recognition device provided in the embodiment of the present disclosure includes an image acquisition module 901 for acquiring a recognition target image, a first feature acquisition module 902 for acquiring an image feature map of the recognition target image based on the recognition target image, a second feature acquisition module 903 for identifying global feature information and local feature information based on the image feature map, and an facial expression type identification module 904 for identifying the facial expression type of the recognition target image based on the global feature information and the local feature information. Because the global feature information of the recognition target image reflects the overall face information, and the local feature information of the recognition target image reflects the detailed information of each region of the face, the effective combination of the local feature information and the global feature information can restore more detailed information of the image of facial expressions, and thus the facial expression type of the recognition target image identified based on the global feature information and the local feature information is more accurate, that is, the accuracy of the recognition result of the facial expression type corresponding to the detection target image can be improved, and the user experience can be further improved.

[0075] It should be understood that the size of the numbers of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and does not arbitrarily limit the implementation process of the embodiments disclosed herein.

[0076] Fig. 10 is a schematic diagram of a computer device 10 provided in an embodiment of the present disclosure. As shown in Fig. 10, the computer device 10 of the embodiment includes a processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable by the processor 1001. When the processor 1001 executes the computer program 1003, it realizes the steps in each of the above method embodiments. Or, when the processor 1001 executes the computer program 1003, it realizes each module / function of each module in each of the above device embodiments.

[0077] For illustrative purposes, the computer program 1003 may be divided into one or more modules, and one or more modules may be stored in the memory 1002 and executed by the processor 1001 to accomplish the present disclosure. One or more modules may be a series of computer program command sections capable of performing a specific function, and the command sections are for explaining the process of the computer program 1003 being executed on the computer device 10.

[0078] The computing device 10 may be a computing device such as a desktop computer, a laptop, a palmtop computer, a cloud server, etc. The computing device 10 may include, but is not limited to, a processor 1001 and a memory 1002. As will be appreciated by those skilled in the art, FIG. 10 is merely an example of the computing device 10, and is not intended to limit the computing device 10, which may include more or fewer components than shown, or may combine certain components or different components, for example, the computing device may include input / output devices, network access devices, buses, etc.

[0079] The processor 1001 may be a central processing unit (CPU) or other general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general purpose processor may be a microprocessor, any common processor, etc.

[0080] The memory 1002 may be an internal storage module of the computer device 10, such as a hard disk or memory of the computer device 10. The memory 1002 may be an external storage device of the computer device 10, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., that is installed in the computer device 10. Furthermore, the memory 1002 may comprise an internal storage module of the computer device 10 or an external storage device. The memory 1002 is for storing computer programs and other programs and data required by the computer device. The memory 1002 may also be used to temporarily store data that has been output or is to be output.

[0081] For convenience and simplicity of explanation, those skilled in the art can clearly understand that, although only the above-mentioned functional modules and division of modules are given as examples, in actual application, the above-mentioned functions can be assigned to different functional modules and completed as necessary, that is, by dividing the internal structure of the device into different functional modules or modules, all or part of the above-mentioned functions can be completed. Each functional module and module in the embodiment may be integrated into one processing module, each module may exist physically alone, or two or more modules may be integrated into one module, and the integrated module may be realized in the form of hardware or in the form of a software functional module. Note that the specific names of each functional module and module are provided for easy distinction from each other, and do not limit the scope of protection of the present disclosure. For the specific operation process of the modules and modules in the above system, reference may be made to the corresponding process in the embodiment of the above-mentioned method, and the description will be omitted here.

[0082] In the above embodiments, the description of each embodiment has its own emphasis, and for the details or parts not described in an embodiment, reference may be made to the relevant descriptions of other embodiments.

[0083] Those skilled in the art can recognize that the modules and algorithm steps of each example described in the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software is determined by the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to realize the described functions for each specific application, but such realization should not be considered to go beyond the scope of this disclosure.

[0084] In the embodiments provided in this disclosure, it should be understood that the disclosed device / computer device and method can be realized in other ways. For example, the above-described device / computer device embodiments are merely exemplary, and the division of the modules or modules is merely a logical functional division, and other division methods may be applied in actual implementation, for example, multiple modules or components may be combined or integrated into other systems, or some features may be ignored or not implemented. In addition, the couplings or direct couplings or communication connections between the displayed or discussed interfaces may be indirect couplings or communication connections through some interfaces, devices or modules, and may be electrical, mechanical, or other forms.

[0085] Moreover, modules described as separate components may or may not be physically separate, and components described as modules may or may not be physical modules, i.e., located in one location or distributed across multiple network modules, some or all of which may be selected according to actual needs to achieve the objectives of the solutions of the present embodiment.

[0086] In addition, each functional module in each embodiment of the present disclosure may be integrated into one processing module, each module may exist physically alone, or two or more modules may be integrated into one module. The integrated module may be realized in the form of hardware or in the form of a software functional module.

[0087] The integrated module / module may be realized in the form of a software function module and stored in a computer-readable storage medium when sold or used as an independent product. With this understanding, the present disclosure can realize all or part of the processes in the method of the above embodiments by commanding relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above method embodiments can be realized. The computer program may include a computer program code, and the computer program code may be in source code form, object code form, executable file, or some intermediate form, etc. The computer-readable storage medium may include any entity or device capable of carrying a computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electric carrier wave signal, an electric communication signal, and a software distribution medium, etc. Furthermore, the content contained on a computer-readable storage medium may be increased or decreased as required by the legislation and patent practice of a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, a computer-readable storage medium does not include electrical carrier signals and telecommunications signals.

[0088] The above-described examples are merely for illustrating the technical solutions of the present disclosure, and are not intended to limit the same. Although the present disclosure has been described in detail with reference to the above-described examples, those skilled in the art may still amend the technical solutions described in the above-described examples, or equivalently replace some of the technical features thereof, but it should be understood that such amendments or replacements do not deviate from the substance of the corresponding technical solutions from the idea and scope of the technical solutions of the embodiments of the present disclosure, and all of them should be included in the protection scope of the present disclosure.

Claims

1. 1. A method for facial expression recognition, comprising: acquiring a recognition target image; obtaining an image feature map of the recognition target image based on the recognition target image; determining the global and local feature information based on the image feature map; and identifying a facial expression type of the recognition target image based on the global feature information and the local feature information. A facial expression recognition method.

2. The method is applied to a facial expression recognition model, the facial expression recognition model comprising a first neural network model, and the step of obtaining an image feature map of the target image based on the target image includes: inputting the recognition target image into a first neural network model to obtain an image feature map of the recognition target image; Wherein, the first neural network model comprises a number of first convolution modules, the number of first convolution modules are connected in series, and each of the first convolution modules comprises a filter, a batch normalization layer, an MP model, and an activation function.

2. The method of claim 1 .

3. The method is applied to a facial expression recognition model, the facial expression recognition model comprising a global model and a local model, and the step of identifying global feature information and local feature information based on the image feature map comprises: inputting the image feature map into a global model to obtain global feature information; inputting the image feature map into a local model to obtain local feature information.

2. The method of claim 1 .

4. The global model includes a first convolution layer, an H-Sigmoid activation function layer, a channel attention module, a spatial attention module, and a second convolution layer. The step of inputting the image feature map into the global model and obtaining global feature information includes: inputting the image feature map into the first convolutional layer to obtain a first feature map; inputting the first feature map into an H-Sigmoid activation function layer to obtain a second feature map; inputting the second feature map into the channel attention module to obtain a channel attention map; inputting the channel attention map into the spatial attention module to obtain a spatial attention map; inputting the spatial attention map into the second convolutional layer to obtain the global feature information.

4. The method according to claim 3 .

5. The local model includes N local feature extraction convolution layers and an attention module, where N is a positive integer greater than 1, and the step of inputting the image feature map into the local model and obtaining local feature information includes: generating N local image blocks based on the image feature map; Inputting each local image block into each local feature extraction convolution layer respectively to obtain N local feature maps; inputting the N local feature maps into the attention module to obtain local feature information.

4. The method according to claim 3 .

6. The attention module includes a pooling layer, a second convolution module, N third convolution layers, and a normalization layer, and the step of inputting the N local feature maps to the attention module and obtaining local feature information includes: fusing the N local feature maps to obtain a fused feature map; inputting the fused feature map into the pooling layer to obtain a pooled feature map; inputting the pooled feature map into the second convolution module to obtain a processed local feature map; inputting the processed local feature map into each third convolutional layer respectively to obtain N sub-local feature maps; For each sub-local feature map, input the sub-local feature map into a normalization layer to obtain a weight value corresponding to the sub-local feature map, and obtain a local feature map corresponding to the sub-local feature map based on the weight value corresponding to the sub-local feature map and the local feature map; Fusing the N local feature maps to obtain local feature information.

6. The method of claim 5 .

7. The step of identifying a facial expression type of the recognition target image based on the global feature information and the local feature information includes: inputting the global feature information into a first global average pooling layer to obtain global pooling feature information; inputting the local feature information into a second global average pooling layer to obtain local pooling feature information; The global pooling feature information and the local pooling feature information are input to a fully connected layer to obtain a facial expression type of the recognition target image.

6. The method of claim 5 .

8. The facial expression recognition model further comprises a weight layer, the weight layer including two fully connected layers and one sigmoid activation layer, and the method further comprises: a step of inputting the global pooling feature information and the local pooling feature information into the weight layer to obtain a true weight value of the facial expression type of the recognition target image, the true weight value of the facial expression type of the recognition target image being for representing a probability that the facial expression type is a corresponding true facial expression type of the recognition target image; and identifying a facial expression type detection result corresponding to the recognition target image based on a truth weight value of the facial expression type of the recognition target image and the facial expression type of the recognition target image.

8. The method of claim 7.

9. A facial expression recognition device, comprising: an image acquisition module for acquiring a recognition target image; a first feature acquisition module for acquiring an image feature map of the recognition target image based on the recognition target image; a second feature acquisition module for identifying global and local feature information based on the image feature map; and a facial expression type identification module for identifying a facial expression type of the recognition target image based on the global feature information and the local feature information. A facial expression recognition device characterized by:

10. 1. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, comprising: The processor, when executing the computer program, performs the steps of the method of claim 1.

1. A computer device comprising:

11. A computer-readable storage medium on which a computer program is stored, The computer program, when executed by a processor, performs the steps of the method of claim 1. A computer-readable storage medium comprising:

Images