Expression recognition method and device, equipment, storage medium and computer program product
By combining the backbone network with the expression prediction module in the facial expression recognition model, and performing dimensionality reduction and linear attention processing in the feature extraction unit, the problems of complex calculations and high resource consumption in the existing technology are solved, and the low cost is achieved Lightweight deployment and efficient identification on hardware.
Patent Information
- Application Number
- CN202411063676.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-12-06
AI Technical Summary
The existing facial expression recognition model has complex calculations and cannot be deployed on low-cost hardware. The traditional visual-based facial expression recognition method consumes a lot of resources and has high calculation volume.
A network model combining a backbone network and expression prediction module is used to extract features through multiple cascading feature extraction units, and dimensionality reduction and linear attention processing are performed before feature extraction to reduce the calculation amount.
While not losing expression recognition accuracy, the calculation amount is significantly reduced, allowing the expression recognition model to be deployed on low-cost hardware and improving the model's lightweight deployment capabilities.
Smart Images

Figure CN119091484A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to image processing technology, and is related to but not limited to expression recognition methods and devices, equipment, storage media, and computer program products. Background Art
[0002] Facial expression recognition (FER) is crucial to understanding human emotions. Although the accuracy of current facial expression recognition models is constantly improving, the amount of computation required by the models is also increasing, making them difficult to deploy on low-cost hardware. Traditional vision-based facial expression recognition methods are computationally complex and resource-intensive due to operations such as preprocessing, feature extraction, and multi-stage classification. Summary of the invention
[0003] The expression recognition method and apparatus, device, storage medium, and computer program product provided in the present application can reduce the computational complexity of expression recognition.
[0004] In a first aspect, an embodiment of the present application provides an expression recognition method, comprising: using a first network model to perform expression recognition on a first image to obtain an expression recognition result; the first network model comprises a backbone network and an expression prediction module; wherein the backbone network is used to perform feature extraction on the first image to obtain a first feature map; the expression prediction module is used to determine the expression recognition result based on the first feature map; the backbone network comprises a plurality of cascaded feature extraction units, at least one feature extraction unit of the plurality of cascaded feature extraction units is used to: perform dimensionality reduction processing on the input second feature map to obtain a third feature map; and perform linear attention processing on the third feature map and output it to the next feature extraction unit or the expression prediction module; wherein the output of the last feature extraction unit of the backbone network is the first feature map.
[0005] In a second aspect, an embodiment of the present application provides an expression recognition device, a recognition module, configured to use a first network model to perform expression recognition on a first image to obtain an expression recognition result; the first network model includes a backbone network and an expression prediction module; wherein the backbone network is used to extract features from the first image to obtain a first feature map; the expression prediction module is used to determine the expression recognition result based on the first feature map; the backbone network includes a plurality of cascaded feature extraction units, at least one feature extraction unit of the plurality of cascaded feature extraction units is used to: perform dimensionality reduction processing on the input second feature map to obtain a third feature map; and perform linear attention processing on the third feature map and output it to the next feature extraction unit or the expression prediction module; wherein the output of the last feature extraction unit of the backbone network is the first feature map.
[0006] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.
[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in the embodiment of the present application.
[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the method provided in the embodiment of the present application.
[0009] In the embodiment of the present application, the first network model is used to perform expression recognition on the first image to obtain an expression recognition result; the backbone network for feature extraction in the first network model includes multiple cascaded feature extraction units; wherein the feature extraction unit is used to perform dimensionality reduction processing on the input feature map before performing linear attention processing on the input feature map. In this way, it is beneficial to reduce the computational complexity of expression recognition without losing the accuracy of expression recognition.
[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0012] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0013] Figure 1 A schematic diagram of the structure of a first network model provided in an embodiment of the present application;
[0014] Figure 2 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 1 ;
[0015] Figure 3 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 2 ;
[0016] Figure 4 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 3 ;
[0017] Figure 5 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 4 ;
[0018] Figure 6 A schematic diagram of the structure of a Mamba algorithm provided in an embodiment of the present application;
[0019] Figure 7 A schematic diagram of the structure of a Light-Mamba algorithm provided in an embodiment of the present application;
[0020] Figure 8 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 5 ;
[0021] Fig. 9 A schematic diagram of the structure of an expression recognition device provided in an embodiment of the present application;
[0022] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.
[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0025] In the following description, reference is made to “some embodiments”, “this embodiment”, “embodiments of the present application” and examples, etc., which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0026] The descriptions of “first, second, third” etc. that appear in the embodiments of the present application are only for illustration and distinction of the description objects. There is no distinction of order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.
[0027] The embodiment of the present application provides an expression recognition method, the method comprising: using a first network model to perform expression recognition on a first image to obtain an expression recognition result; the first network model comprises a backbone network and an expression prediction module; wherein the backbone network is used to perform feature extraction on the first image to obtain a first feature map; the expression prediction module is used to determine the expression recognition result according to the first feature map;
[0028] The backbone network includes multiple cascaded feature extraction units, and at least one of the multiple cascaded feature extraction units is used to: perform dimensionality reduction processing on the input second feature map to obtain a third feature map; and perform linear attention processing on the third feature map and output it to the next feature extraction unit or the expression prediction module; wherein the output of the last feature extraction unit of the backbone network is the first feature map.
[0029] It can be understood that in the embodiment of the present application, the first network model is used to perform expression recognition on the first image to obtain the expression recognition result; the backbone network for feature extraction in the first network model includes multiple cascaded feature extraction units; wherein the feature extraction unit is used to perform dimensionality reduction processing on the input feature map before performing linear attention processing on the input feature map. In this way, it is beneficial to reduce the computational complexity of expression recognition without losing the accuracy of expression recognition.
[0030] It should be understood that in the embodiments of the present application, there is no limitation on the first network model, and the first network model can recognize expressions. The first network model can be a machine learning model or a deep learning model. There is no limitation on the backbone network and the expression prediction module. Exemplarily, in some embodiments, the backbone network can be a network in the residual network (Residual Network, ResNet) series, such as: ResNet 18, ResNet 24, ResNet 50, ResNet 101, ResNet152; the first network model can also be a network in the mobile network (MobileNet) series; for example: MobileNet V1, MobileNet V2, MobileNet V3, MobileNetV3-Large, MobileNet V3-Small.
[0031] For example, as a possible implementation, Figure 1 A schematic diagram of the structure of a first network model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the input of the first network model is the first image 101, and the output is the expression recognition result 104; the structure of the first network model includes: a backbone network 102, an expression prediction module 103; wherein, the backbone network 102 includes at least one feature extraction unit 1021, and the feature extraction unit 1021 includes a dimensionality reduction processing unit 211 and a linear attention processing unit 212; the dimensionality reduction processing unit 211 is used to perform dimensionality reduction processing on the second feature map, and the linear attention processing unit 212 is used to perform linear attention processing on the third feature map.
[0032] It should be understood that in the embodiment of the present application, the first image 101 can be directly input into the first network model, or can be indirectly input into the first network model. The first network model can also include a preprocessing module ( Figure 1 ). If the first network model includes a preprocessing module, the first image 101 is directly input into the first network model, that is, the first image 101 is input into the backbone network 102 after being preprocessed; if the first network model does not include a preprocessing module, the first image 101 is indirectly input into the first network model, that is, the first image 101 is input into the backbone network 102 in the first network model after being preprocessed; if the first network model does not include a preprocessing module, the first image 101 is directly input into the first network model, that is, the first image 101 is directly input into the backbone network 102 in the first network model, and at this time, the first image 101 refers to the preprocessed image.
[0033] In the embodiment of the present application, there is no limitation on the number of feature extraction units 1021 in the backbone network. The more feature extraction units 1021 there are, the higher the accuracy of expression recognition may be, and the higher the performance requirements of the electronic device may be.
[0034] In the embodiments of the present application, the dimensionality reduction processing of the input second feature map to obtain the third feature map can be implemented by the following embodiments 1, 2, 3 and 4.
[0035] In Example 1, the dimensionality reduction processing is performed on the input second feature map to obtain the third feature map, including: performing a first convolution (Convolution, Conv) processing on the second feature map to obtain the third feature map; wherein the number of convolution kernels used in the first convolution processing is less than the number of channels of the second feature map.
[0036] It can be understood that in the embodiment of the present application, the first convolution process is performed on the second feature map to obtain the third feature map; wherein the number of convolution kernels used in the first convolution process is less than the number of channels of the second feature map. In this way, the number of channels of the output third feature map is less than the number of channels of the second feature map, thereby achieving dimensionality reduction of the number of channels of the second feature map, which is beneficial to reduce the amount of calculation of linear attention processing on the third feature map.
[0037] It should be understood that in the above embodiment, the dimension reduction processing unit 211 includes a first convolution processing unit.
[0038] In the embodiment of the present application, the first convolution processing is not limited, and the number of convolution kernels used in the first convolution processing is less than the number of channels of the second feature map. Exemplarily, as a possible implementation, the number of channels of the second feature map is C, and the number of convolution kernels used in the first convolution processing is less than C; for example: the number of convolution kernels used in the first convolution processing is: wait.
[0039] In Example 2, the dimensionality reduction processing of the input second feature map to obtain the third feature map includes: performing a second convolution processing, a normalization processing and an activation processing on the second feature map in sequence to obtain a fourth feature map; performing a first convolution processing on the fourth feature map to obtain the third feature map; wherein the number of convolution kernels used in the second convolution processing is less than the number of channels of the second feature map, and the number of convolution kernels used in the first convolution processing is less than the number of channels of the fourth feature map.
[0040] It can be understood that in the embodiment of the present application, before the first convolution processing is performed on the second feature map, the second feature map is first subjected to the second convolution processing, the normalization processing and the activation processing in sequence; wherein, the number of convolution kernels used in the second convolution processing is less than the number of channels of the second feature map, and the number of convolution kernels used in the first convolution processing is less than the number of channels of the fourth feature map. In this way, compared with only performing the first convolution processing on the second feature map, it is beneficial to further reduce the amount of calculation of the linear attention processing on the third feature map, thereby reducing the amount of calculation of expression recognition.
[0041] It should be understood that in the embodiment of the present application, the second convolution processing is not limited; the number of convolution kernels used in the second convolution processing is less than the number of channels of the second feature map. Exemplarily, as a possible implementation, the number of channels of the second feature map is C, and the number of convolution kernels used in the second convolution processing is less than C; for example: the number of convolution kernels used in the second convolution processing is: Correspondingly, if the number of channels of the fourth feature map is The number of convolution kernels used in the first convolution process is less than For example, the number of convolution kernels used in the first convolution process is:
[0042] In the embodiments of the present application, normalization processing and activation processing are not limited. The normalization processing may be batch normalization (Batch Normalization, BN), instance normalization (Instance Normalization, IN), group normalization (Group Normalization, GN), weight normalization (Weight Normalization, WB); batch renormalization (Batch Renormalization, BR), etc. The activation processing may be S-shaped growth curve (Sigmoid) processing, corrected linear unit (ReLU), leaky corrected linear unit (Leaky ReLU), parameterized corrected linear unit (ParametricReLU), hyperbolic tangent (tanh) function, normalized exponential linear unit (Softmax), scaled exponential linear unit (Swish) with compression, Maxout, Softplus, Mish, Gaussian error linear unit (GELU), etc.
[0043] In Example 3, the dimensionality reduction processing of the input second feature map to obtain the third feature map includes: performing a second convolution processing on the second feature map to obtain a fourth feature map; performing a first convolution processing, a normalization processing and an activation processing on the fourth feature map in sequence to obtain the third feature map; wherein the number of convolution kernels used in the second convolution processing is less than the number of channels of the second feature map, and the number of convolution kernels used in the first convolution processing is less than the number of channels of the fourth feature map.
[0044] In Example 4, the dimensionality reduction processing of the input second feature map to obtain the third feature map includes: performing a first convolution processing on the second feature map to obtain a fourth feature map; performing a second convolution processing, a normalization processing and an activation processing on the fourth feature map in sequence to obtain the third feature map; wherein the number of convolution kernels used in the first convolution processing is less than the number of channels of the second feature map, and the number of convolution kernels used in the second convolution processing is less than the number of channels of the fourth feature map.
[0045] In an embodiment of the present application, the linear attention processing of the third feature map and then outputting it to the next feature extraction unit 1021 or the expression prediction module 103 can be achieved through the following embodiments 5, 6, 7 and 8.
[0046] Further, in Example 5, the linear attention processing of the third feature map is output to the next feature extraction unit 1021 or the expression prediction module 103, including: performing linear attention feature extraction on the third feature map to obtain a fifth feature map; splicing the fifth feature map and the second feature map to obtain a sixth feature map; determining a seventh feature map based on the sixth feature map, and outputting the seventh feature map to the next feature extraction unit 1021 or the expression prediction module 103.
[0047] It can be understood that in the embodiment of the present application, the third feature map after the dimensionality reduction process is subjected to linear attention feature extraction to obtain the fifth feature map; then the fifth feature map and the second feature map are spliced to obtain the sixth feature map; then the seventh feature map is determined according to the sixth feature map, and is output to the next feature extraction unit 1021 or the expression prediction unit 103. In this way, it is beneficial to reduce the amount of calculation of linear attention feature extraction while splicing with the input second feature map to solve the problem of insufficient long-distance dependence of the feature extraction unit 1021, thereby alleviating the gradient vanishing problem caused by increasing the depth in the deep neural network, and thus improving the accuracy of expression recognition.
[0048] It should be understood that in the embodiment of the present application, when the seventh feature map is output to the next feature extraction unit 1021, the steps to be executed for the seventh feature map are the same as those for the second feature map; the steps to be executed for the seventh feature map are the same as those for the input feature maps of all feature extraction units 1021 at the intermediate level.
[0049] In Example 5, the third characteristic map may be obtained based on any one of Example 1, Example 2, Example 3 and Example 4.
[0050] In Example 6, the linear attention processing is performed on the third feature map and then outputted to the next feature extraction unit 1021 or the expression prediction module 103, including: performing linear attention feature extraction on the third feature map to obtain a fifth feature map; performing a second convolution process, a normalization process and an activation process on the second feature map in sequence to obtain a fourth feature map; wherein the number of convolution kernels used in the second convolution process is less than the number of channels of the second feature map; concatenating the fourth feature map and the fifth feature map to obtain a sixth feature map; determining a seventh feature map based on the sixth feature map, and outputting the seventh feature map to the next feature extraction unit 1021 or the expression prediction module 103.
[0051] It can be understood that in the embodiment of the present application, the fifth feature map is not the second feature map, but the fourth feature map that further reduces the dimension of the second feature map. This is beneficial to further reduce the dimension of the number of channels of the sixth feature map, thereby further reducing the computational complexity of the backbone network, that is, further reducing the computational complexity of expression recognition.
[0052] It should be understood that in Example 6, the third characteristic graph can be obtained based on any one of Example 1, Example 2, Example 3 and Example 4.
[0053] In some embodiments, when the third feature map is obtained based on Example 2, the linear attention processing of the third feature map is output to the next feature extraction unit 1021 or the expression prediction module 103, including: performing linear attention feature extraction on the third feature map to obtain a fifth feature map; splicing the fifth feature map and the fourth feature map to obtain a sixth feature map; determining a seventh feature map based on the sixth feature map, and outputting the seventh feature map to the next feature extraction unit 1021 or the expression prediction module 103.
[0054] It can be understood that in the embodiment of the present application, based on the third feature map obtained by sequentially performing the second convolution processing, normalization processing, activation processing and the first convolution processing on the second feature map, linear attention feature extraction is performed to obtain the fifth feature map; the fifth feature map is spliced with the fourth feature map that has undergone the second convolution processing, normalization processing and activation processing. In this way, it is beneficial to further reduce the computational amount of the linear attention processing on the third feature map while further reducing the dimension of the number of channels of the sixth feature map, thereby further reducing the computational amount of the backbone network, that is, further reducing the computational amount of expression recognition.
[0055] In Example 7, the linear attention processing of the third feature map is output to the next feature extraction unit 1021 or the expression prediction module 103, including: performing linear attention feature extraction on the third feature map to obtain a fifth feature map; performing a second convolution processing on the second feature map to obtain a fourth feature map; wherein the number of convolution kernels used in the second convolution processing is less than the number of channels of the second feature map; concatenating the fourth feature map and the fifth feature map to obtain a sixth feature map; determining a seventh feature map based on the sixth feature map, and outputting the seventh feature map to the next feature extraction unit 1021 or the expression prediction module 103.
[0056] In Example 8, the linear attention processing of the third feature map is output to the next feature extraction unit 1021 or the expression prediction module 103, including: performing linear attention feature extraction on the third feature map to obtain a fifth feature map; performing a first convolution processing on the second feature map to obtain a fourth feature map; wherein the number of convolution kernels used in the second convolution processing is less than the number of channels of the second feature map; concatenating the fourth feature map and the fifth feature map to obtain a sixth feature map; determining a seventh feature map based on the sixth feature map, and outputting the seventh feature map to the next feature extraction unit 1021 or the expression prediction module 103.
[0057] It should be understood that in Example 7 or Example 8, the third characteristic graph can be obtained based on any one of Example 1, Example 2, Example 3 and Example 4.
[0058] In some embodiments, the method further includes: determining the seventh characteristic map based on the sixth characteristic map, which can be implemented by the following embodiments 9 and 10.
[0059] In Example 9, determining the seventh feature map based on the sixth feature map includes: performing feature rearrangement processing, third convolution processing, normalization processing and activation processing on the sixth feature map in sequence to obtain the seventh feature map; wherein, when the seventh feature map is output to the next feature extraction unit 1021, the number of convolution kernels of the third convolution processing is equal to the number of channels of the sixth feature map; when the seventh feature map is output to the expression prediction module 103, the number of convolution kernels of the third convolution processing is equal to the number of channels of the sixth feature map.
[0060] It can be understood that in the embodiment of the present application, the sixth feature map is subjected to feature rearrangement processing, the third convolution processing, normalization processing and activation processing. In this way, the sixth feature map is subjected to feature rearrangement processing, and then the feature map subjected to feature rearrangement processing is subjected to the third convolution processing, normalization processing and activation processing, that is, feature extraction is performed; this is beneficial to improving the generalization ability of the first network model, improving the learning ability of the first network model for features, and thus improving the effect of expression recognition.
[0061] It should be noted that in the embodiment of the present application, the third convolution processing is not limited. When the seventh feature map is output to the next feature extraction unit 1021, the number of convolution kernels of the third convolution processing is equal to the number of channels of the sixth feature map. When the seventh feature map is output to the expression prediction module 103, if the expression prediction module 103 includes convolution processing, the convolution kernel of the third convolution processing is determined based on the number of channels of the sixth feature map and the number of input channels of the first convolution processing in the expression prediction module 103 inputted with the seventh feature map; if the expression prediction module 103 does not include convolution processing, the number of convolution kernels of the third convolution processing is equal to the number of channels of the sixth feature map.
[0062] It should be understood that the embodiment 9 can be combined with any one of embodiments 5 to 8.
[0063] Exemplarily, as a possible implementation, Example 9 is combined with Example 5; wherein the third characteristic graph in Example 5 is obtained based on Example 1. Figure 2 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 1 ,like Figure 2 As shown, the feature extraction unit 1021 includes: Conv1 201, linear attention feature extraction 202, splicing 203, Shuffle 204, Conv3 205, BN 206 and activation processing 207.
[0064] Among them, based on Figure 2 The feature extraction unit 1021 shown in the figure processes the second feature map, including: performing a first convolution process 201 (i.e., Conv1 201) on the second feature map to obtain the third feature map; performing linear attention feature extraction 202 on the third feature map to obtain a fifth feature map; concatenating 203 the fifth feature map and the second feature map to obtain a sixth feature map; performing feature rearrangement process 204 (i.e., Shuffle 204), third convolution process 205 (i.e., Conv3 205), normalization process 206 (i.e., BN 206) and activation process 207 on the sixth feature map in sequence to obtain the seventh feature map.
[0065] As another possible implementation, Example 9 is combined with Example 6; wherein the third characteristic graph in Example 6 is obtained based on Example 1. Figure 3 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 2 ,like Figure 3As shown, the feature extraction unit 1021 includes: Conv1 201, linear attention feature extraction 202, splicing 203, Shuffle 204, Conv3 205, BN 206, activation processing 207, Conv2 301, BN 302 and activation processing 303.
[0066] Among them, based on Figure 3 The feature extraction unit 1021 shown in the figure processes the second feature map, including: performing a first convolution process 201 (i.e., Conv1 201) on the second feature map to obtain the third feature map; performing a linear attention feature extraction 202 on the third feature map to obtain a fifth feature map; performing a second convolution process 301 (i.e., Conv2 301), a normalization process 302 (BN 302), and an activation process 303 on the second feature map in sequence to obtain a fourth feature map; concatenating 203 the fourth feature map and the fifth feature map to obtain a sixth feature map; performing a feature rearrangement process 204 (i.e., Shuffle 204), a third convolution process 205 (i.e., Conv3 205), a normalization process 206 (i.e., BN 206), and an activation process 207 on the sixth feature map in sequence to obtain the seventh feature map.
[0067] As another possible implementation, Example 9 is combined with Example 6; wherein the third characteristic graph in Example 6 is obtained based on Example 2. Figure 4 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 3 ,like Figure 4 As shown, the feature extraction unit 1021 includes: Conv1 201, linear attention feature extraction 202, splicing 203, Shuffle 204, Conv3 205, BN 206, activation processing 207, Conv2 301, BN 302 and activation processing 303.
[0068] Among them, based on Figure 4The feature extraction unit 1021 shown in the figure processes the second feature map, including: performing a second convolution process 301 (i.e., Conv2 301), a normalization process 302 (BN 302), and an activation process 303 on the second feature map in sequence to obtain a fourth feature map; performing a first convolution process 201 (i.e., Conv1 201) on the fourth feature map to obtain the third feature map; performing linear attention feature extraction 202 on the third feature map to obtain a fifth feature map; performing a second convolution process 301 (i.e., Conv2 301), a normalization process 302 (BN 302), and an activation process 303 on the second feature map in sequence to obtain a fourth feature map; concatenating the fourth feature map and the fifth feature map 203 to obtain a sixth feature map; performing a feature rearrangement process 204 (i.e., Shuffle 204), a third convolution process 205 (i.e., Conv3 205), a normalization process 206 (i.e., BN 207) on the sixth feature map in sequence. 206) and activation processing 207 to obtain the seventh feature map.
[0069] In Example 10, determining the seventh feature map based on the sixth feature map includes: performing feature rearrangement processing, third convolution processing, normalization processing and activation processing on the sixth feature map in sequence to obtain an eighth feature map; performing feature fusion on the eighth feature map and the second feature map to obtain the seventh feature map; wherein, when the seventh feature map is output to the next feature extraction unit 1021, the number of convolution kernels of the third convolution processing is equal to the number of channels of the sixth feature map; when the seventh feature map is output to the expression prediction module 103, the number of convolution kernels of the third convolution processing is equal to the number of channels of the sixth feature map.
[0070] It can be understood that in the embodiment of the present application, the sixth feature map is subjected to feature rearrangement processing, the third convolution processing, normalization processing and activation processing to obtain the eighth feature map; and the eighth feature map and the second feature map are subjected to feature fusion to obtain the seventh feature map. In this way, compared with the sixth feature map subjected to feature rearrangement processing, the third convolution processing, normalization processing and activation processing to obtain the seventh feature map, it is beneficial to further solve the problem of insufficient long-distance dependence of the feature extraction unit 1021, thereby alleviating the gradient vanishing problem caused by increasing the depth in the deep neural network, and thus improving the accuracy of expression recognition.
[0071] Exemplarily, as a possible implementation, Example 10 is combined with Example 6; wherein the third characteristic graph in Example 6 is obtained based on Example 2. Figure 5 A structural diagram of a feature extraction unit provided in an embodiment of the present application Figure 4 ,like Figure 5As shown, the feature extraction unit 1021 includes: Conv1 201, linear attention feature extraction 202, splicing 203, Shuffle 204, Conv3 205, BN 206, activation processing 207, Conv2 301, BN 302, activation processing 303 and feature fusion 501.
[0072] Among them, based on Figure 5 The feature extraction unit 1021 shown in the figure processes the second feature map, including: performing a second convolution process 301 (i.e., Conv2 301), a normalization process 302 (BN 302), and an activation process 303 on the second feature map in sequence to obtain a fourth feature map; performing a first convolution process 201 (i.e., Conv1 201) on the fourth feature map to obtain the third feature map; performing linear attention feature extraction 202 on the third feature map to obtain a fifth feature map; performing a second convolution process 301 (i.e., Conv2 301), a normalization process 302 (BN 302), and an activation process 303 on the second feature map in sequence to obtain a fourth feature map; concatenating the fourth feature map and the fifth feature map 203 to obtain a sixth feature map; performing a feature rearrangement process 204 (i.e., Shuffle 204), a third convolution process 205 (i.e., Conv3 205), a normalization process 206 (i.e., BN 207) on the sixth feature map in sequence. 206) and activation processing 207 to obtain the eighth feature map; feature fusion 501 is performed on the eighth feature map and the second feature map to obtain the seventh feature map.
[0073] In some embodiments, the first network model is obtained by training through N training processes; wherein, the i-th training process includes: determining a first number of defective samples based on the i value, N and a first threshold; wherein, i is greater than or equal to 1 and less than or equal to N, and N is greater than 1; the defective samples refer to samples that do not meet quality requirements; the defective samples include at least one of the following: samples with blurred faces, samples with occluded faces, samples with uneven lighting, and samples without labels; obtaining a first number of the defective samples; obtaining a second number of normal samples; wherein the normal samples refer to samples that meet quality requirements; and training the first network model based on the defective samples and the normal samples.
[0074] It can be understood that in the embodiment of the present application, the samples used to train the first network model include not only normal samples, but also at least one of samples with blurred faces, samples with occluded faces, samples with uneven illumination, and samples without labels. This is beneficial to improve the expression recognition accuracy of defective images while maintaining the expression recognition accuracy of normal images.
[0075] It should be understood that in the embodiments of the present application, the specific values of N and the first threshold are not limited. The ratio of N to the first threshold may be 3 / 5, 3 / 4, 4 / 5, 5 / 6, etc. In some embodiments, N is 100 and the first threshold is 60.
[0076] In the embodiment of the present application, the first number and the second number are not limited. The more sample data involved in the training, the better the training effect, the longer the training time, and the greater the performance overhead of the electronic device used.
[0077] In some embodiments, determining the first number of defective samples based on the i value, N and a first threshold includes: determining a first ratio between the i value and the first threshold when the i value is less than or equal to the first threshold; and determining a first difference between the first ratio and a first value; determining a first product between the first difference and the number of defective samples; determining the first number of defective samples to be the first product; and determining the first number of defective samples to be 0 when the i value is greater than the first threshold and less than or equal to N.
[0078] It can be understood that in the embodiment of the present application, when the number of training times is less than or equal to the first threshold, the data involved in the training includes defective samples and normal samples; when the number of training times is greater than the first threshold, the samples involved in the training include normal samples. This is because the inventors have found through experiments that if defective samples are continuously added during the training process, the training effect will be limited, and may even cause the degradation of the recognition accuracy of the first network model in the later stages of training; therefore, in the training process of the first network model, defective samples are used in the training in the early and middle stages of the training, and normal samples are used in the training in the later stages of the training; this is beneficial to further improve the recognition accuracy of the first network model.
[0079] It should be understood that in an embodiment of the present application, a training strategy is proposed, and the training strategy is: in the early or early middle stage of training, the first network model is trained using normal samples and defective samples, and in the middle or late stage of training, only normal samples are used to train the first network model. The embodiment of the present application does not limit the specific implementation method of the training strategy provided by the present application. The above embodiment is only one possible way to implement the training strategy provided by the embodiment of the present application, and the training strategy provided by the embodiment of the present application can also be implemented through other implementation methods.
[0080] For example, as a possible implementation, after the number of training rounds reaches e t (i.e., an example of the first threshold), D′ (i.e., an example of a defective sample) is used for training, and D′ is gradually reduced for training.t In the case of , stop adding D′, and use D (i.e., an example of normal samples) for training for a few rounds. This can not only improve the recognition ability of the facial expression recognition model (i.e., an example of the first network model) for defective samples in the early and middle stages of model training, but also use normal samples in the later rounds to maintain the model's recognition ability for normal samples.
[0081] In some embodiments, the samples for training the first network model are determined by the following formula (1) and formula (2):
[0082]
[0083]
[0084] Among them, D all represents the training set after being adjusted by the training strategy fn, D represents the normal sample, and D′ represents the defective sample or the sample with a small number in the training set; e all represents the total number of training rounds (i.e., an example of N); e t (i.e., an example of the first threshold) indicates the number of rounds at which D′ should be stopped from participating in training. It is recommended that e t =e all ×3 / 4 or e t =e all ×4 / 5. (1-β) (i.e., an example of the first difference) represents the proportion of the data set D′ participating in the training. In the early stage of training, a large number of D′ participates in the training, thereby increasing the ability of the first network model to learn the data features in D′. Then, the participation of D′ in the training is gradually reduced, so that the model learns various data distribution features in the normal sample D.
[0085] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.
[0086] The embodiment of the present application provides an expression recognition method, which is a lightweight facial expression recognition optimization method based on deep learning, and the method can recognize image data obtained using a camera. The method meets the requirements of low-cost device deployment by designing an end-to-end small model.
[0087] The embodiment of the present application provides a lightweight Mamba (Light-Mamba) algorithm, which is obtained by optimizing the Mamba algorithm, and then combining the Light-Mamba algorithm with a convolutional neural network (CNN) (i.e., an example of a feature extraction unit) to fully utilize the advantages of convolution and Mamba algorithms. In this way, it is beneficial to improve the effect of expression recognition while reducing the amount of calculation. In addition, the embodiment of the present application also provides a training strategy, which trains the model (i.e., an example of the first network model) by adjusting the training data set, thereby obtaining a model that is friendly to the use environment and easy to deploy.
[0088] Among them, the Mamba algorithm is a recently emerging selective structural state space model that performs well in long sequence modeling. The Mamba algorithm can alleviate the modeling constraints of CNN's long-distance dependencies and has performance similar to Transformers, but its computational complexity is much lower than that of Transformers. In recent papers, the Mamba algorithm has been applied in different fields and has different variants.
[0089] Specifically, in some embodiments, in order to ensure the accuracy and speed of the model, the Mamba structure is lightly optimized to obtain the Light-Mamba algorithm. Figure 6 A schematic diagram of the structure of a Mamba algorithm provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the Mamba algorithm includes: normalization 601 (ie, Norm 601 ), state space model block 602 (StateSpace Model Block, SSM Block) (ie, SSM Block 602 ) and feature fusion 603 .
[0090] Figure 7 A schematic diagram of the structure of a Light-Mamba algorithm provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the Light-Mamba algorithm includes: convolution processing 701 (i.e. Conv 701, which is also an example of the first convolution processing), normalization 601 (i.e. Norm 601), state space model block 602 (State Space Model Block, SSMBlock) (i.e. SSM Block 602), splicing 702 (i.e. Concat702) and feature rearrangement 703 (i.e. Shuffle 703).
[0091] The Light-Mamba algorithm provided in the embodiment of the present application is a lightweight optimization method for the Mamba algorithm. The Light-Mamba algorithm can maintain the advantages of the Mamba algorithm in terms of facial expression recognition accuracy while reducing its computational complexity.
[0092] Assuming that the number of channels of the input feature map is C, a convolution layer is added before the Mamba algorithm (i.e., an example of the first convolution process), the input feature channel is changed to C / 2, and then operations such as Norm 601 and SSM Block 602 are performed. In this way, in the process of performing Norm 601 and SSM Block 602, since the number of channels of the input feature map is halved, the amount of calculation is also reduced accordingly. In addition, compared with the feature fusion by addition in the original Mamba algorithm, the Light-Mamba algorithm optimizes it to concatenate 702 (i.e., Concat 702) the output of SSM Block 602 with the output of the added Conv 701, and then perform feature rearrangement processing 703 (i.e., Shuffle 703) on the concatenated feature map, so that the number of channels of the feature map output by the Light-Mamba algorithm is consistent with the number of channels of the input feature map.
[0093] The Mamba algorithm can be expressed by the following formula (3):
[0094] Fout=fn(SSM(Norm(Fin_c))) (3)
[0095] In formula (3), Norm represents Norm 601, SSM represents Shuffle 703, and F in_c represents a feature map with C channels, and fn represents feature fusion processing, that is, Figure 6 The portion in which the output of SSM Block 602 is added to the input is executed.
[0096] The Light-Mamba algorithm can be expressed by the following formula (4) and formula (5):
[0097] Fo'ut=fn'(SSM(Norm(Fin_c / 2 ))) (4)
[0098] F in_c / 2 =Conv(F in_c ) (5)
[0099] Where Conv means Conv 701; F in_c / 2 yes Figure 7 Through Conv 701 to F in_c After convolution, Fin_c / 2 The number of channels is F in_c half of the number of channels; fn' means Figure 7 Concat 702 and Shuffle 703 in the embodiment are feature fusion parts; in some embodiments, fn' may also represent a feature fusion operation of a pair of Concat 702 and the fourth convolution process.
[0100] Compared with formula (3), the number of channels of the feature map operated by Norm 601 and SSM Block 602 in formula (4) is halved, so the amount of calculation and the amount of parameters of Norm 601 and SSM Block 602 in formula (4) are also relatively reduced. After feature fusion, i.e., Concat 702 and Shuffle 703, the number of channels of the output feature map is consistent with the number of channels of the output feature map in formula (3).
[0101] The embodiment of the present application proposes a Light-Mamba algorithm by optimizing the structure of the Mamba algorithm. The Light-Mamba algorithm can not only be used alone in the deep learning model structure, but also can be combined with CNN to improve the model feature extraction capability by combining the advantages of both.
[0102] It should be understood that in the embodiments of the present application, the Light-Mamba algorithm can be applied not only to the expression recognition model, but also to the deep learning model in other fields. The Light-Mamba algorithm provides a lightweight optimization method for the deep learning model, which is easy to deploy in low-cost devices.
[0103] CNN extracts features by continuously sliding windows on the input image. The cascaded CNN can gradually learn high-level features from the original image. CNN has sufficient translation invariance and locality. The SSM Block 602 in Light-Mamba can establish long-distance dependencies. Combining the local feature extraction capability of CNN with the feature of Light-Mamba that captures long-distance dependencies, it has excellent results in the lightweight deployment of facial expression recognition models. The backbone network of the lightweight model for facial expression recognition (i.e., an example of the first network model) can choose the currently popular small model backbone network, such as the mobile series or resnet.
[0104] Figure 8 A schematic diagram of a feature extraction unit provided in an embodiment of the present application Figure 5 ,like Figure 8As shown, the feature extraction unit 1021 is designed by combining Light-Mamba and CNN units, specifically including: CNN unit 801, Light-Mamba 802 (structural diagram as shown in Figure 7 As shown), splicing 803 (i.e., cat 803; wherein, Concat, cat), CNN unit 804 and feature fusion 805; wherein, CNN unit 801 includes: Conv2, BN and act; CNN unit 804 includes: Conv3, BN and act; the Conv2 represents the second convolution processing, and Conv3 represents the third convolution processing.
[0105] After the input feature map passes through CNN unit 801, the number of feature channels changes from C to C / 2, and then is sent to Light-Mamba802. This branch first extracts features through the first convolution process, while changing the number of feature channels to C / 4, and then obtains the first output result through other operations in the Light-Mamba algorithm; another branch directly uses CNN unit 801 for feature extraction; then it is spliced with the first output result of the Light-Mamba model 803, and finally input into CNN unit 804 to obtain the second output result. In some embodiments, the number of channels of CNN unit 804 can be changed as needed. After that, the second output result is feature fused 805 with the input feature map of feature extraction unit 1021. It can be seen that feature extraction unit 1021 is a residual structure, so it can increase the effect of feature fusion.
[0106] In some embodiments, one of the operations of feature fusion 805 of the second output result and the input feature map of the feature extraction unit 1021 can be deleted as needed. In the embodiment of the present application, Conv can be replaced with a variant convolution such as depthwise convolution (DWconv).
[0107] Figure 8 The feature extraction unit 1021 shown can be applied in the backbone network of the facial expression recognition model to replace the simple convolution module. The feature extraction unit 1021 provided in the embodiment of the present application combines the Light-Mamba algorithm and the CNN unit; specifically: the Mamba algorithm is firstly structured to be lightweight and optimized to reduce its computational complexity, and then combined with the CNN unit to fully compensate for the lack of long-distance dependence of the CNN.
[0108] The training strategy of the expression recognition model (i.e., an example of the first network model) provided in the embodiment of the present application is mainly a processing strategy for difficult samples (i.e., an example of defective samples) and samples with a small number of labels during model training. Specifically, since the model has a slightly poor recognition effect on difficult samples or samples with few labels, such samples are added during training. Experimental verification has found that if data difficult samples or samples with few labels are continuously added during the training process, the training effect is limited, and may even cause degradation of the model effect in the later stages of training.
[0109] Therefore, the training strategy of the embodiment of the present application is: when the number of training rounds reaches e t (i.e., an example of the first threshold), use D′ (i.e., an example of a defective sample) to participate in the training, and gradually reduce D′ to participate in the training. When the number of training times is greater than e t In the case of , stop adding D′, and use D (i.e., an example of normal samples) for training for a few more rounds. This can not only improve the recognition ability of the facial expression recognition model (i.e., an example of the first network model) for defective samples in the early and middle stages of model training, but also use normal samples in the later rounds to maintain the model's recognition ability for normal samples.
[0110] In some embodiments, the samples for training the first network model are determined by the following formula (6) and formula (7):
[0111]
[0112]
[0113] Among them, D all represents the training set after being adjusted by the training strategy fn, D represents the normal sample, and D′ represents the defective sample or the sample with a small number in the training set; e all represents the total number of training rounds (i.e., an example of N); e t (i.e., an example of the first threshold) indicates the number of rounds at which D′ should be stopped from participating in training. It is recommended that e t =e all ×3 / 4 or e t =e all ×4 / 5. (1-β) (i.e., an example of the first difference) represents the proportion of the data set D′ participating in the training. In the early stage of training, a large number of D′ participates in the training, thereby increasing the ability of the first network model to learn the data features in D′. Then, the participation of D′ in the training is gradually reduced, so that the model learns various data distribution features in the normal sample D.
[0114] The expression recognition method provided in the embodiment of the present application provides a lightweight structural optimization of the currently popular Mamba algorithm to obtain the Light-Mamba algorithm. The obtained Light-Mamba algorithm is then combined with CNN and applied to the expression recognition model (i.e., a type of the first model). Among them, the algorithm combining the Light-Mamba algorithm with CNN can not only be applied to the expression recognition model, but also to the deep learning model structure in other fields. The Light-Mamba algorithm provides a lightweight optimization method for the deep learning model, which is easy to deploy in low-cost equipment. Finally, the embodiment of the present application proposes a training strategy for the expression recognition model, which further improves the model recognition effect.
[0115] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.
[0116] Based on the foregoing embodiments, an embodiment of the present application provides a display control device, which includes the modules included and the units included in the modules, and can be implemented by a processor or an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be an AI acceleration engine (such as NPU, etc.), GPU, central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.
[0117] Fig. 9 A schematic diagram of the structure of the expression recognition device provided in the embodiment of the present application is shown in FIG. Fig. 9 As shown, the device 90 includes an identification module 901, wherein:
[0118] The recognition module 901 is configured to perform expression recognition on the first image using a first network model to obtain an expression recognition result; the first network model includes a backbone network and an expression prediction module; wherein the backbone network is used to perform feature extraction on the first image to obtain a first feature map; the expression prediction module is used to determine the expression recognition result according to the first feature map;
[0119] The backbone network includes a plurality of cascaded feature extraction units, at least one of which is used to: perform dimensionality reduction processing on the input second feature map to obtain a third feature map; and perform linear attention processing on the third feature map and then output it to the next feature extraction unit or the expression prediction module; wherein the output of the last feature extraction unit of the backbone network is the first feature map
[0120] In some embodiments, the dimensionality reduction processing of the input second feature map to obtain the third feature map includes: performing a first convolution processing on the second feature map to obtain the third feature map; wherein the number of convolution kernels used in the first convolution processing is less than the number of channels of the second feature map.
[0121] In some embodiments, the dimensionality reduction processing of the input second feature map to obtain the third feature map includes: performing a second convolution processing, a normalization processing, and an activation processing on the second feature map in sequence to obtain a fourth feature map; performing a first convolution processing on the fourth feature map to obtain the third feature map; wherein the number of convolution kernels used in the second convolution processing is less than the number of channels of the second feature map, and the number of convolution kernels used in the first convolution processing is less than the number of channels of the fourth feature map.
[0122] Further, in some embodiments, the linear attention processing of the third feature map is output to the next feature extraction unit or the expression prediction module, including performing linear attention feature extraction on the third feature map to obtain a fifth feature map; splicing the fifth feature map and the fourth feature map to obtain a sixth feature map; determining a seventh feature map based on the sixth feature map, and outputting the seventh feature map to the next feature extraction unit or the expression prediction module.
[0123] In some embodiments, the linear attention processing of the third feature map and then outputting it to the next feature extraction unit or the expression prediction module includes: performing linear attention feature extraction on the third feature map to obtain a fifth feature map; splicing the fifth feature map and the second feature map to obtain a sixth feature map; determining a seventh feature map based on the sixth feature map, and outputting the seventh feature map to the next feature extraction unit or the expression prediction module.
[0124] In some embodiments, the performing of linear attention processing on the third feature map and then outputting it to the next feature extraction unit or the expression prediction module includes: performing linear attention feature extraction on the third feature map to obtain a fifth feature map; performing second convolution processing, normalization processing, and activation processing on the second feature map in sequence to obtain a fourth feature map; wherein the number of convolution kernels used in the second convolution processing is less than the number of channels of the second feature map; concatenating the fourth feature map and the fifth feature map to obtain a sixth feature map; determining a seventh feature map based on the sixth feature map, and outputting the seventh feature map to the next feature extraction unit or the expression prediction module. .
[0125] Further, in some embodiments, determining the seventh feature map based on the sixth feature map includes: performing feature rearrangement processing, third convolution processing, normalization processing, and activation processing on the sixth feature map in sequence to obtain the seventh feature map; or, performing feature rearrangement processing, third convolution processing, normalization processing, and activation processing on the sixth feature map in sequence to obtain an eighth feature map; performing feature fusion on the eighth feature map and the second feature map to obtain the seventh feature map; wherein the number of convolution kernels of the third convolution processing is equal to the number of channels of the sixth feature map.
[0126] In some embodiments, the first network model is obtained by training through N training processes; wherein, the i-th training process includes: determining a first number of defective samples based on the i value, N and a first threshold; wherein, i is greater than or equal to 1 and less than or equal to N, and N is greater than 1; the defective samples refer to samples that do not meet quality requirements; the defective samples include at least one of the following: samples with blurred faces, samples with occluded faces, samples with uneven lighting, and samples without labels; obtaining a first number of the defective samples; obtaining a second number of normal samples; wherein the normal samples refer to samples that meet quality requirements; and training the first network model based on the defective samples and the normal samples.
[0127] In some embodiments, determining the first number of defective samples based on the i value, N and a first threshold includes: determining a first ratio between the i value and the first threshold when the i value is less than or equal to the first threshold; and determining a first difference between the first ratio and a first value; determining a first product between the first difference and the number of defective samples; determining the first number of defective samples to be the first product; and determining the first number of defective samples to be 0 when the i value is greater than the first threshold and less than or equal to N.
[0128] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0129] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.
[0130] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device or processor to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.
[0131] An embodiment of the present application provides an electronic device, Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.10 As shown, the electronic device 100 includes a memory 1001 and a processor 1002, wherein the memory 1001 stores a computer program that can be run on the processor 1002, and the processor 1002 implements the steps in the method provided in the above embodiment when executing the program.
[0132] It should be noted that the memory 1001 is configured to store instructions and applications executable by the processor 1002, and can also cache data to be processed or processed by the processor 1002 and various modules in the electronic device 100 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).
[0133] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.
[0134] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of the method provided in the above embodiment.
[0135] It should be noted here that the description of the above device, storage medium and program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device, storage medium and program product embodiments of this application, please refer to the description of the method embodiment of this application for understanding.
[0136] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.
[0137] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.
[0138] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0139] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.
[0140] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0141] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0142] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.
[0143] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device or processor to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0144] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0145] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0146] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0147] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for facial expression recognition, characterized in that: The method comprises: Using a first network model to perform expression recognition on a first image to obtain an expression recognition result; the first network model includes a backbone network and an expression prediction module; wherein the backbone network is used to perform feature extraction on the first image to obtain a first feature map; the expression prediction module is used to determine the expression recognition result according to the first feature map; The backbone network includes multiple cascaded feature extraction units, and at least one of the multiple cascaded feature extraction units is used to: perform dimensionality reduction processing on the input second feature map to obtain a third feature map; and perform linear attention processing on the third feature map and output it to the next feature extraction unit or the expression prediction module; wherein the output of the last feature extraction unit of the backbone network is the first feature map.
2. The method according to claim 1, characterized in that The step of performing dimensionality reduction processing on the input second feature map to obtain a third feature map includes: Perform a first convolution process on the second feature map to obtain the third feature map; wherein the number of convolution kernels used in the first convolution process is less than the number of channels of the second feature map.
3. The method according to claim 1, characterized in that: The step of performing dimensionality reduction processing on the input second feature map to obtain a third feature map includes: performing a second convolution process, a normalization process, and an activation process on the second feature map in sequence to obtain a fourth feature map; Performing a first convolution process on the fourth feature map to obtain the third feature map; wherein the number of convolution kernels used in the second convolution process is less than the number of channels of the second feature map, and the number of convolution kernels used in the first convolution process is less than the number of channels of the fourth feature map.
4. The method according to claim 3, characterized in that The linear attention processing of the third feature map is output to the next feature extraction unit or the expression prediction module, including Performing linear attention feature extraction on the third feature map to obtain a fifth feature map; Concatenating the fifth feature map and the fourth feature map to obtain a sixth feature map; According to the sixth feature map, a seventh feature map is determined, and the seventh feature map is output to the next feature extraction unit or the expression prediction module.
5. The method according to claim 2, characterized in that: The performing linear attention processing on the third feature map and outputting it to the next feature extraction unit or the expression prediction module comprises: Performing linear attention feature extraction on the third feature map to obtain a fifth feature map; Concatenating the fifth feature map and the second feature map to obtain a sixth feature map; According to the sixth feature map, a seventh feature map is determined, and the seventh feature map is output to the next feature extraction unit or the expression prediction module.
6. The method according to claim 2, characterized in that The performing linear attention processing on the third feature map and outputting it to the next feature extraction unit or the expression prediction module comprises: Performing linear attention feature extraction on the third feature map to obtain a fifth feature map; performing a second convolution process, a normalization process, and an activation process on the second feature map in sequence to obtain a fourth feature map; wherein the number of convolution kernels used in the second convolution process is less than the number of channels of the second feature map; Concatenating the fourth feature map and the fifth feature map to obtain a sixth feature map; According to the sixth feature map, a seventh feature map is determined, and the seventh feature map is output to the next feature extraction unit or the expression prediction module.
7. The method according to any one of claims 4 to 6, characterized in that: Determining the seventh characteristic map according to the sixth characteristic map includes: performing feature rearrangement processing, third convolution processing, normalization processing, and activation processing on the sixth feature map in sequence to obtain the seventh feature map; or, performing a feature rearrangement process, a third convolution process, a normalization process, and an activation process on the sixth feature map in sequence to obtain an eighth feature map; Performing feature fusion on the eighth feature map and the second feature map to obtain the seventh feature map; Among them, the number of convolution kernels of the third convolution processing is equal to the number of channels of the sixth feature map.
8. The method according to any one of claims 1 to 6, characterized in that: The first network model is obtained by training through N training processes; wherein the i-th training process includes: Determine a first number of defective samples according to the i value, N and a first threshold value; wherein i is greater than or equal to 1 and less than or equal to N, and N is greater than 1; the defective samples refer to samples that do not meet quality requirements; the defective samples include at least one of the following: samples with blurred faces, samples with blocked faces, samples with uneven illumination, and samples without labels; Acquire a first number of defect samples; Acquire a second number of normal samples; wherein the normal samples refer to samples that meet the quality requirements; The first network model is trained according to the defective samples and the normal samples.
9. The method according to claim 8, characterized in that The step of determining a first number of defective samples according to the i value, N and a first threshold value includes: In the case where the i value is less than or equal to the first threshold, determining a first ratio between the i value and the first threshold; and determining a first difference between the first ratio and a first value; determining a first product between the first difference and the number of defective samples; and determining the first number of defective samples to be the first product; When the i value is greater than the first threshold and less than or equal to N, the first number of defective samples is determined to be 0.
10. An expression recognition device, characterized in that: The device comprises: The recognition module is configured to use a first network model to perform expression recognition on a first image to obtain an expression recognition result; the first network model includes a backbone network and an expression prediction module; wherein the backbone network is used to perform feature extraction on the first image to obtain a first feature map; the expression prediction module is used to determine the expression recognition result according to the first feature map; The backbone network includes multiple cascaded feature extraction units, and at least one of the multiple cascaded feature extraction units is used to: perform dimensionality reduction processing on the input second feature map to obtain a third feature map; and perform linear attention processing on the third feature map and output it to the next feature extraction unit or the expression prediction module; wherein the output of the last feature extraction unit of the backbone network is the first feature map.
11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Lightweight mask face recognition method and system, storage medium and equipment
CN112560828A
Face key point detection method based on attention guidance lightweight network
CN115966004A
Feature-enhanced lightweight network FGNet facial expression recognition method
CN116311414A
Anomaly detection using dynamic density estimation
CN116324877A
Lightweight model-based driver expression recognition method
CN118072292A
