Expression recognition model training method, expression recognition method, system and equipment

Abstract

The invention discloses a training method of an expression recognition model, and an expression recognition method, system and device, and the training method of the expression recognition model comprises the steps: obtaining a training face image, inputting the training face image into an expression recognition model, and carrying out the separation type deformation attention convolution processing of the training face image, and obtaining a prediction result, and calculating the prediction result according to the loss function to obtain a loss value, and updating parameters of the expression recognition model according to the loss value to obtain a trained expression recognition model. The embodiment of the invention can improve the accuracy of expression recognition and the utilization efficiency of computing resources, and can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a training method for an expression recognition model, an expression recognition method, a system and a device. Background Art

[0002] Facial expressions are an important part of non-verbal communication. They can convey rich emotional and status information through facial muscle movements. In related technologies, when the face is presented at a specific angle or is blocked by other objects, the accuracy of expression recognition will drop significantly. There are facial expression recognition algorithms based on large neural networks, which improve the accuracy of expression recognition by introducing global or local attention mechanisms and increasing network depth. However, in actual applications, it is found that introducing global or local attention mechanisms and increasing network depth will reduce the utilization efficiency of computing resources and affect the efficiency of expression recognition.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a training method for an expression recognition model, an expression recognition method, a system and a device, which can improve the expression recognition rate and the efficiency of computing resource utilization.

[0005] To achieve the above object, an embodiment of the present application provides a method for training an expression recognition model, the method comprising:

[0006] Get training face images;

[0007] Inputting the training face image into the expression recognition model to perform separate deformation attention convolution processing to obtain a prediction result;

[0008] Calculate the prediction result according to the loss function to obtain a loss value;

[0009] The parameters of the facial expression recognition model are updated according to the loss value to obtain a trained facial expression recognition model.

[0010] In some embodiments, the step of inputting the training face image into the expression recognition model for separate deformation attention convolution processing to obtain a prediction result comprises the following steps:

[0011] Performing feature extraction processing on the training face image to obtain an input feature map;

[0012] Performing layer-deformable convolution processing on the input feature map to obtain an output feature map;

[0013] Performing channel layer attention point convolution processing on the output feature map to obtain a target feature map;

[0014] The target feature map is fully connected and activated to obtain the prediction result.

[0015] In some embodiments, performing layer deformable convolution processing on the input feature map to obtain an output feature map comprises the following steps:

[0016] Assigning a separate deformable convolution kernel to each channel of the input feature map;

[0017] The position offset variable and the deformed convolution kernel are combined to perform convolution calculation processing on any point on the channel to obtain the output feature map.

[0018] In some embodiments, performing channel layer attention point convolution processing on the output feature map to obtain a target feature map includes the following steps:

[0019] Performing average pooling processing on the output feature map to obtain global attention information;

[0020] Performing nonlinear convolution processing on the global attention information to obtain a first convolution result;

[0021] Convolution processing is performed on each pixel of the output feature map in combination with the channel offset variable and the first convolution result to obtain the target feature map.

[0022] In some embodiments, the calculation and processing of the prediction result according to the loss function to obtain the loss value includes the following steps:

[0023] Performing classification loss calculation processing on the prediction result to obtain a first loss result;

[0024] Performing a central loss calculation process on the prediction result to obtain a second loss result;

[0025] The first loss result and the second loss result are calculated and processed in combination with the hyperparameters to obtain the loss value.

[0026] To achieve the above object, another aspect of the embodiment of the present application provides an expression recognition method for performing expression recognition using an expression recognition model trained by the training method described above, the expression recognition method comprising the following steps:

[0027] Obtain a face image to be recognized;

[0028] The face image to be recognized is input into the expression recognition model for expression recognition processing to obtain an expression recognition result.

[0029] To achieve the above object, another aspect of the embodiment of the present application provides a training system for an expression recognition model, the training system comprising:

[0030] An acquisition module is used to acquire training face images;

[0031] A prediction module, used for inputting the training face image into the expression recognition model to perform a separated deformation attention convolution process to obtain a prediction result;

[0032] A loss module, used to calculate and process the prediction result according to a loss function to obtain a loss value;

[0033] An updating module is used to update the parameters of the expression recognition model according to the loss value to obtain a trained expression recognition model.

[0034] To achieve the above object, another aspect of the embodiment of the present application provides an expression recognition system, the expression recognition system comprising:

[0035] A capture module, used to obtain a face image to be identified;

[0036] The recognition module is used to input the face image to be recognized into the expression recognition model for expression recognition processing to obtain an expression recognition result.

[0037] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0038] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0039] The embodiments of the present application include at least the following beneficial effects: the present application provides a training method for an expression recognition model, an expression recognition method, a system and a device, which obtains a prediction result by inputting a training face image into an expression recognition model for separable deformable attention convolution processing, and can introduce global attention information and a learnable offset through the separable deformable attention convolution, which not only retains the computational efficiency of the separable convolution, but also enables the convolution to adaptively find the comprehensive image features that determine the expression classification, reduces the problems of irregular facial angles and facial occlusions in the image, and improves the efficiency of expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a method for training an expression recognition model provided in an embodiment of the present application;

[0041] Figure 2 It is a schematic diagram of a general convolution kernel calculation provided in an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of performing a convolution operation on a feature map provided in an embodiment of the present application;

[0043] Figure 4 This is a flow chart of a separated deformable attention convolution provided by an embodiment of the present application;

[0044] Figure 5 This is a flowchart of performing a layer deformation convolution operation on any channel in a separated deformation attention convolution provided by an embodiment of the present application;

[0045] Figure 6 It is a flowchart of an operation of performing a channel layer attention point convolution on any pixel point in a separated deformable attention convolution provided by an embodiment of the present application;

[0046] Figure 7 is a flow chart of an expression recognition model application provided by an embodiment of the present application;

[0047] Figure 8 It is a structural schematic diagram of a training system for an expression recognition model provided in an embodiment of the present application;

[0048] Fig. 9 is a structural diagram of an expression recognition system provided in an embodiment of the present application;

[0049] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0051] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0052] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0055] 1) Facial Expression Recognition (FER): It is a machine learning problem that deals with identifying human facial expressions. It uses a series of algorithms to identify and determine the emotions expressed by the input face image, including happiness, sadness, anger, surprise, fear and disgust. It has a wide range of applications in education, health care and psychological counseling.

[0056] 2) Deformable Convolution: It is a special convolution operation in the field of deep learning. It introduces a learnable offset without adding additional supervised learning, enhances the adaptability to spatial sampling, and can effectively cope with dense spatial transformations in complex visual tasks. It significantly improves the performance of the model when processing irregularly shaped targets, bringing important breakthroughs in the field of deep learning.

[0057] 3) MobileNet model: The MobileNet model is a lightweight deep neural network proposed by Google for embedded devices such as mobile phones. The core idea used is Depthwise Separable Convolution. Compared with traditional convolutional neural networks, it greatly reduces model parameters and computational complexity while slightly reducing accuracy.

[0058] Facial expressions, as an important part of non-verbal communication, convey rich emotions and status information through facial muscle movements. Facial expression analysis (FER) plays a vital role in many fields such as human-computer interaction, intelligent control, security monitoring, medical diagnosis and communication. For example, FER can be used in the education field to evaluate teaching quality, and in the medical field to analyze the psychological state of patients. Therefore, in-depth research and application of facial expression analysis is of great significance.

[0059] However, facial expression analysis faces a series of challenges. First, the inter-class similarities and intra-class differences of human facial expressions, that is, the subtle differences between different expressions and the significant changes in the same expression among different individuals, make accurate recognition extremely difficult. Second, facial angle and occlusion issues also have a significant impact on the recognition effect of facial expression analysis. When the face is presented at a specific angle or is occluded by other objects, the accuracy of expression recognition will drop significantly. Therefore, in-depth research and response to these challenges are of great significance to improving the accuracy and reliability of facial expression analysis.

[0060] In related technologies, existing methods are gradually leaning towards large neural networks. Most algorithm models are based on the CNN network architecture, and improve the accuracy of expression recognition by introducing global or local attention mechanisms and increasing network depth. However, it is worth noting that expression recognition is usually the first stage in the application, and its accuracy and real-time performance are crucial for subsequent calculations and analysis. Therefore, while pursuing a high recognition rate, it is also necessary to fully consider the utilization efficiency of computing resources and reserve sufficient resources for subsequent calculations and analysis.

[0061] For example, some researchers have proposed a method of using a staged MobileNetV1 as a backbone feature extractor and combining it with a Transformer architecture to recognize facial expressions. This method aims to save computing resources and improve the practicality of facial expression analysis technology by optimizing the network structure. However, while reducing the depth of the network model, the model does not fully utilize the facial expression information and does not alleviate the problem of facial occlusion in the image. Therefore, research on expression recognition needs to continue to explore in-depth optimization methods for facial expression analysis technology to improve its accuracy and reliability while taking into account the efficiency of computing resources.

[0062] In view of this, the embodiments of the present application provide a training method, an expression recognition method, a system and a device for an expression recognition model, which obtains a training face image, inputs the training face image into the expression recognition model for separated deformation attention convolution processing to obtain a prediction result, calculates the prediction result according to the loss function to obtain a loss value, and updates the parameters of the expression recognition model according to the loss value to obtain a trained expression recognition model. The embodiments of the present application effectively deal with the occlusion problem by efficiently integrating comprehensive facial information, while optimizing the allocation of computing resources, ensuring that facial expressions can be accurately captured and analyzed in complex environments, and improving the facial expression recognition rate.

[0063] The training method and expression recognition method of the expression recognition model provided in the embodiment of the present application relate to the field of computer vision technology. The training method and expression recognition method of the expression recognition model provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application of the training method and expression recognition method of the expression recognition model, but is not limited to the above forms.

[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0065] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0066] Figure 1 is an optional flowchart of a method for training an expression recognition model provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0067] Step S101, obtaining a training face image;

[0068] Step S102, inputting the training face image into the expression recognition model to perform a separate deformation attention convolution process to obtain a prediction result;

[0069] Step S103, calculating and processing the prediction result according to the loss function to obtain a loss value;

[0070] Step S104, updating the parameters of the expression recognition model according to the loss value to obtain a trained expression recognition model.

[0071] Steps S101 to S104 shown in the embodiment of the present application, by acquiring training face images, which include face images with expressions, are used to train the expression recognition model. The training face images are then input into the expression recognition model for separate deformable attention convolution processing to obtain prediction results, wherein the expression recognition model uses MobileNetV1 as the model architecture, and the separate convolution operation therein is replaced with the separate deformable attention convolution of the embodiment of the present application, and the convolution kernel size is not changed. It should be noted that the separate deformable attention convolution in the embodiment of the present application is to divide the conventional convolution into two steps, specifically divided into layer deformable convolution and channel layer attention point convolution. In the layer convolution stage, the embodiment of the present application introduces learnable convolution position offset variables, which can adaptively adjust the position of the convolution kernel so that it focuses on those feature points that are crucial to expression classification. This dynamic adjustment mechanism greatly enhances the model's ability to capture subtle changes in expression, and can accurately capture decisive features even when the face is partially occluded; in the channel convolution stage, the embodiment of the present application uses a learnable position offset variable and global attention information, performs a point multiplication operation on the global attention information after nonlinear transformation and the offset convolution kernel, and uses global information to finely control the offset degree of the convolution position. This process realizes the deep fusion of global and channel layer information, enabling the expression recognition model to more accurately locate and extract the most decisive features for expression classification. Then, the prediction results are calculated and processed according to the loss function to obtain the loss value, and the parameters of the expression recognition model can be updated through the loss value, so as to continuously train and obtain a model with the best training results, and obtain a trained expression recognition model. The embodiment of the present application designs and uses a lightweight attention convolution adaptive expression recognition algorithm for the model, introduces a separated deformation attention convolution, which can adapt to the position and shape of facial expressions while reducing the amount of calculation, and introduces an attention mechanism to guide the channel layer convolution, effectively alleviating the problem of facial occlusion in the expression recognition process, and comprehensively using the overall facial information. Specifically, the structure of the expression recognition model of the embodiment of the present application is shown in Table 1, which is a model training structure table.

[0072]

[0073] Table 1

[0074] The embodiment of the present application can integrate multi-source facial information by reasonably designing and using separate deformable attention convolution in the model algorithm, thereby flexibly responding to occlusion situations, reducing the use of computing resources, and improving the accuracy of facial expression recognition.

[0075] In step S102 of some embodiments, the step of inputting the training face image into the expression recognition model for separate deformation attention convolution processing to obtain a prediction result includes the following steps:

[0076] Performing feature extraction processing on the training face image to obtain an input feature map;

[0077] Performing layer-deformable convolution processing on the input feature map to obtain an output feature map;

[0078] Performing channel layer attention point convolution processing on the output feature map to obtain a target feature map;

[0079] The target feature map is fully connected and activated to obtain the prediction result.

[0080] In an embodiment of the present application, the expression recognition model uses a separable deformable attention convolution. In order to better introduce the separable deformable attention convolution, the present application first introduces the traditional convolution operation, and then introduces the lightweight attention convolution. Traditional convolution is to slide the convolution kernel on the input image according to a set step size. Each time it slides to a new position, the convolution kernel will cover a local area on the input image. This local area is called the receptive field. In each local area, the convolution kernel is element-wise multiplied with the corresponding local area of ​​the input image, and all products are added to obtain a sum. This sum is the value of the corresponding position on the output feature map. In traditional convolution, the position of each convolution on the feature map is fixed, and the convolution kernel is generally a regular convolution kernel, such as 3×3×c, 5×5×c convolution kernel, where c is the number of channels of the feature map. Taking the 3×3×c convolution kernel as an example, the general convolution kernel calculation method is as follows Figure 2 As shown, Figure 2 is a schematic diagram of a general convolution kernel calculation provided in an embodiment of the present application. Figure 2 In the figure, 201 represents the input feature map, h represents the height of the input feature map, w represents the width of the input feature map, 202 represents the convolution kernel, c1 represents the number of convolution sums, c represents the number of channels of each convolution kernel, 203 represents the output feature map, h1 represents the height of the output feature map, and w1 represents the width of the output feature map. The number of channels of the input feature map is the same as the number of channels of the convolution kernel, that is, in the corresponding channel, the feature map and the corresponding position of the convolution kernel are multiplied and then added. In order to better demonstrate the convolution operation, the feature map t1 is convolved as shown in the figure. Figure 3 As shown, Figure 3 It is a schematic diagram of a convolution operation on a feature map provided in an embodiment of the present application, in which a 3×3 convolution kernel is used to perform operations at an offset of each point in the convolution kernel relative to the center point.

[0081] The embodiment of the present application introduces a separated deformable attention convolution, which decomposes a convolution operation into two steps. The first step is the layer deformable convolution, and the second step is the channel layer attention point convolution layer. Figure 4 As shown, Figure 4 This is a flowchart of a separable deformable attention convolution provided in an embodiment of the present application, taking a 3×3×c convolution kernel as an example. Figure 4 In the embodiment, the input feature map is first subjected to layer deformable convolution to obtain the output feature map, and then the output feature map is subjected to channel deformation point convolution to obtain the target feature map. Finally, the target feature map is fully connected and activated to obtain the prediction result. The embodiment of the present application introduces a learnable convolution position offset variable in the layer deformable convolution, so as to adaptively find feature points that can determine the expression classification for convolution operation; and in the channel layer attention point convolution, not only the convolution position offset variable is introduced, but also the global attention information of the layer convolution is introduced, and a nonlinear transformation is performed on the global attention information to introduce nonlinear information, so that the global attention information is point-multiplied with the offset convolution kernel, and the global information is used to guide the degree of convolution position offset to find features that can determine the expression classification, thereby improving the accuracy of expression recognition.

[0082] In some embodiments, performing layer deformable convolution processing on the input feature map to obtain an output feature map comprises the following steps:

[0083] Assigning a separate deformable convolution kernel to each channel of the input feature map;

[0084] The position offset variable and the deformed convolution kernel are combined to perform convolution calculation processing on any point on the channel to obtain the output feature map.

[0085] In the embodiment of the present application, the input feature map is subjected to layer-deformable convolution processing, specifically, a separate deformable convolution kernel is assigned to each channel feature of the input feature map for convolution, and the channels are not interacted with each other, that is, the input feature map is subjected to layer-deformable convolution operation with the convolution kernel of the corresponding channel to obtain the output feature map. The layer-deformable convolution operation is as follows: Figure 5 As shown, Figure 5 This is a flow chart of performing layer deformation convolution operation on any channel in a separated deformable attention convolution provided by the embodiment of the present application. In the calculation process of the layer deformable convolution process, the embodiment of the present application adds a position offset variable Δp to adapt to the image features. 0 , the formula for deforming the convolution is as follows:

[0086]

[0087] Among them, Δp is the learnable position offset variable, R is the relative position coordinate matrix, and pn Represents the offset of each point in the convolution kernel relative to the center point, and the distance p 0 The enumeration value of the relative position of the surrounding area that does not exceed 1 pixel is expressed by the following formula (taking the 3×3 convolution kernel as an example):

[0088] p n ∈R={(-1,-1),(-1,0),…,(0,0),..,(1,0),(1,)};

[0089] Among them, p 0 +p n +Δp is the position offset coordinate, x(p 0 +p n +Δp) represents the pixel value at that coordinate, w(p 0 +p n +Δp) represents the weight of the corresponding position of the convolution kernel, y(p 0 ) represents the feature map p 0 The output of the elements at the position after convolution, the Relu() function introduces more nonlinear information for a nonlinear transformation. The embodiment of the present application can introduce a learnable position offset variable by performing layer-deformable convolution on the input feature map. The position offset variable enables the expression recognition model to adaptively search for feature points that can determine the expression classification according to the facial expression information to perform convolution operations, so that the model can adaptively adjust the position of the convolution operation according to the subtle changes in facial expressions, and accurately focus on those feature points that play a decisive role in expression classification. The layer-deformable convolution of the embodiment of the present application can greatly enhance the model's ability to capture complex facial expressions and improve the accuracy of expression recognition.

[0090] In some embodiments, performing channel layer attention point convolution processing on the output feature map to obtain a target feature map includes the following steps:

[0091] Performing average pooling processing on the output feature map to obtain global attention information;

[0092] Performing nonlinear convolution processing on the global attention information to obtain a first convolution result;

[0093] Convolution processing is performed on each pixel of the output feature map in combination with the channel offset variable and the first convolution result to obtain the target feature map.

[0094] In the embodiment of the present application, the output feature map of the layer deformable convolution output is subjected to channel layer attention point convolution processing to obtain the target feature map. Specifically, the layer deformable convolution output is used as the input of the channel layer attention point convolution layer, such as Figure 6 As shown, Figure 6This is a flowchart of the operation of performing channel-level attention point convolution on any pixel in a separated deformable attention convolution provided by an embodiment of the present application. In the embodiment of the present application, any channel-level attention point convolution operation is performed, and any channel on the channel convolution is performed. The channel-level attention point convolution operation at the pixel level is as follows: For each feature map on a channel, the feature map is averaged and pooled to obtain global attention information. The global attention information can guide the learnable channel offset variable and increase the connectivity of the two convolutions at the same time. Then, c1 1×1 nonlinear convolution operations are performed and the pixel point p on the output feature map is averaged and pooled. i2 The channel offset ΔC is introduced, c1 is the number of channels of the feature map with output. The formula for each channel layer attention point convolution operation is as follows:

[0095]

[0096] Among them, c is the number of channels of the output feature map of the layer deformable convolution, i is the variable representing the i-th channel, ΔC is the channel offset variable, w(p i2 ) indicates the convolution kernel corresponding to p i2 The weight of the convolution point position, avg(T2) represents the average pooling of the output feature map T2, x(p i2 +ΔC) represents the pixel value under the channel offset coordinate. In the embodiment of the present application, the model will first extract and analyze the global attention information output by the deformable convolution layer, perform nonlinear convolution on it, introduce additional nonlinear variables, enhance the richness and depth of feature expression, and introduce a learnable channel offset variable ΔC, so that the convolution operation can also be adaptively adjusted between channels. Finally, the attention mechanism is adaptively combined with the channel layer through dot multiplication, and global information is used to guide the convolution to find features that can determine the expression classification, optimize the connectivity of the characteristic information between the two convolutions, and make the separated deformable attention convolution a complete convolution operation to the outside. The embodiment of the present application uses global attention to guide the convolution position offset through the channel layer attention convolution. This operation enables the convolution to adaptively find the comprehensive image features that determine the expression classification, reduce the problems of irregular facial angles and facial occlusions in the image, and improve the accuracy of expression recognition.

[0097] In some embodiments, the calculation and processing of the prediction result according to the loss function to obtain the loss value includes the following steps:

[0098] Performing classification loss calculation processing on the prediction result to obtain a first loss result;

[0099] Performing a central loss calculation process on the prediction result to obtain a second loss result;

[0100] The first loss result and the second loss result are calculated and processed in combination with the hyperparameters to obtain the loss value.

[0101] In the embodiments of the present application, for data with complex distributions such as human faces, it is usually hoped that the data is not only separable between classes in the feature space, but more importantly, compact within the class. For example, the difference between an angry face and a disgusted face may be minimal, while the difference between two different individuals in the same expression category may be very significant. This variability reduces the accuracy and reliability of expression classification. Therefore, the embodiments of the present application calculate and process the prediction results according to the loss function. In order to reduce the intra-class gap and expand the inter-class gap, the embodiments of the present application combine SoftmaxLoss and CenterLoss as the loss function. The formula of the loss function is as follows:

[0102] Loss=SoftmaxLoss+λCenterLoss;

[0103] In order to increase the distance loss, speed up the feature extraction capability, and balance the weights of the intra-class distance and the inter-class distance, the λ hyperparameter is introduced into the loss function. The SoftmaxLoss loss function is used to calculate the classification loss of the prediction results to obtain the first loss result. The expression of SoftmaxLoss is as follows:

[0104]

[0105] Among them, x i ∈R d , indicating that it belongs to y i The deep features of the i-th layer of the class, y i represents the prediction result, i represents the i-th row in the prediction result, that is, the i-th category; d represents the dimension; W j ∈R d , represents the weight W∈R in the last fully connected layer d×n The jth column of ; b is the bias; m is the batch size; and n is the number of categories.

[0106] Then the prediction result is processed by the CenterLoss loss function to calculate the center loss and obtain the second loss result. The expression of CenterLoss is as follows:

[0107]

[0108] Where m is the batch size; c yi ∈R d Indicates that the category is y i The characteristic center of c yiIt will be updated as the deep features change, so it cannot be used directly in CenterLoss. However, since the embodiment of this application uses batch training data, c yi The center of the category corresponding to each sample in the current batch. Batch refers to a group of data samples used to train the model in one iteration. The embodiment of the present application can better train the expression recognition model and improve the accuracy of expression recognition by combining softmaxLoss and CenterLoss as the loss function and using hyperparameters to balance the weights of intra-class distance and inter-class distance.

[0109] Another aspect of the embodiment of the present application provides an expression recognition method for performing expression recognition using an expression recognition model trained by the training method as described above, the expression recognition method comprising the following steps:

[0110] Obtain a face image to be recognized;

[0111] The face image to be recognized is input into the expression recognition model for expression recognition processing to obtain an expression recognition result.

[0112] In the embodiments of this application, please refer to Figure 7 , Figure 7 By inputting the training image into the expression recognition model for training, outputting the prediction result, and using the loss function to supervise the expression recognition model, the model with the best training result is obtained through continuous training as the trained expression recognition model. Among them, supervised training is a training method in machine learning, which uses a data set with a mark or label to train the model. In this method, each training sample contains an input and a corresponding output, which is called a mark or label. The task of the model is to learn the mapping relationship from input to output. Finally, the trained expression recognition model is used as the application model. By inputting the face image to be recognized, the face image to be recognized is input into the expression recognition model for expression recognition processing to obtain the expression recognition result.

[0113] Another aspect of the embodiment of the present application provides an expression recognition method for performing expression recognition using an expression recognition model trained by the training method as described above, the expression recognition method comprising the following steps:

[0114] Obtain a face image to be recognized;

[0115] The face image to be recognized is input into the expression recognition model for expression recognition processing to obtain an expression recognition result.

[0116] In an embodiment of the present application, the expression recognition method performs expression recognition through an expression recognition model trained by the training method described above, specifically by obtaining a face image to be recognized, inputting the face image to be recognized into the expression recognition model for expression recognition processing, and obtaining a predicted expression recognition result. Since the expression recognition method of the embodiment of the present application uses the expression recognition model trained by the training method described above, the contents of the above-mentioned expression model training method embodiment are applicable to the embodiment of the present expression recognition method, and the functions specifically implemented by the present expression recognition method embodiment are the same as those of the above-mentioned expression model training method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned expression model training method embodiment.

[0117] The following is a detailed description of the embodiments of the present invention with reference to specific application examples:

[0118] The training method of the expression recognition model or the expression recognition method of the embodiment of the present application can be applied to the field of computer vision technology and can be applied to human-computer interaction scenarios. For example, by recognizing the user's expression, the smart device can more accurately understand the user's intentions and emotions, thereby providing more personalized services and interactive experiences. Specifically, in the field of games, the system can adjust the game content or difficulty based on the player's expression feedback to provide a smarter gaming experience. In the field of education, expression recognition technology can identify the emotional state of students, adaptively adjust teaching content and methods, and improve learning outcomes. The embodiments of the present application can also be applied in the fields of social media, market research, etc. Expression recognition technology can identify the user's emotional tendencies and provide sentiment analysis support for brands, products, etc. By analyzing the user's expression data, companies can gain an in-depth understanding of the user's emotional state and needs, and provide strong support for product optimization, advertising, etc. Please refer to Figure 7 In practical applications, the embodiment of the present application first obtains a training data set of a pre-application scenario, transmits it to the model for training, and then applies the optimized segmentation model with good effect to the scene to be recognized, such as intelligent robots, intelligent monitoring, virtual reality and synthetic animation. The embodiment of the present application trains the expression recognition model with separated deformable attention convolution of the embodiment of the present application by obtaining a training data set of a pre-application scenario, obtains the optimal training model in the training process, writes this model into the hardware kernel or calls this model in the software module. Then, the face image to be recognized is obtained through the front-end camera, and this model is called to predict the expression recognition result of the face image to be recognized, and the expression recognition result is transmitted to the hardware kernel or software module to guide the hardware or software to perform corresponding operations.

[0119] It should be noted that the separated deformable attention convolution in the embodiment of the present application improves the model performance while effectively reducing the amount of calculation, especially the number of parameters. Take the common 3×3 convolution kernel as an example for detailed analysis:

[0120] In a general convolution operation, if the number of channels of the input feature map is c and the number of channels of the pre-output feature map is c1, the total number of parameters required is 3×3×c×c1. This is because each output channel needs to be fully connected to all input channels.

[0121] Separable deformable attention convolution achieves efficient parameter utilization by decomposing the convolution operation into two independent steps: layer convolution (responsible for extracting spatial features) and channel convolution (responsible for integrating information between channels). Specifically, the layer convolution uses a 3×3 convolution kernel to process each input channel to generate an intermediate feature map. The number of parameters in this step is 3×3×c. Subsequently, the channel convolution uses a 1×1 convolution kernel to integrate information between channels and output the final feature map. The number of parameters in this step is 1×1×c×c1. Therefore, the overall number of parameters of the separable deformable attention convolution is roughly 3×3×c plus a small number of additional parameters for processing global attention and offset (assuming that this part is at a constant level and does not increase significantly with the number of channels in the input feature map), and 1×1×c×c1.

[0122] Taking specific values ​​as an example, when the number of input feature map channels c = 64 and the number of pre-output feature map channels c1 = 128, the number of parameters of the general convolution is 73728 (3×3×64×128). For the separated deformable attention convolution, although the total parameters are slightly increased due to the introduction of global attention and offset (but for the sake of simplicity, it is assumed that the increment is small), the overall parameter magnitude is still close to 74112 (mainly 3×3×64+1×1×64×128), and as the number of channels increases further, the difference in the number of parameters between the two will become more significant, reflecting the advantage of separated deformable attention convolution in computational efficiency. It should be emphasized that this reduction in the number of parameters does not sacrifice model performance. On the contrary, it improves the model's adaptability and expression recognition accuracy through a more intelligent feature extraction and integration mechanism.

[0123] See also Figure 8 The embodiment of the present application also provides a training system for an expression recognition model, which can implement the training method for the expression recognition model. The training system includes:

[0124] An acquisition module 801 is used to acquire a training face image;

[0125] Prediction module 802, used for inputting the training face image into the expression recognition model to perform separation deformation attention convolution processing to obtain a prediction result;

[0126] The loss module 803 is used to calculate the prediction result according to the loss function to obtain a loss value;

[0127] The updating module 804 is used to update the parameters of the expression recognition model according to the loss value to obtain a trained expression recognition model.

[0128] It can be understood that the contents of the above-mentioned expression model training method embodiment are applicable to the present training system embodiment, the functions specifically implemented by the present training system embodiment are the same as those of the above-mentioned expression model training method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned expression model training method embodiment.

[0129] See also Fig. 9 The embodiment of the present application further provides an expression recognition system, the expression recognition system comprising:

[0130] Capturing module 901, used to obtain a face image to be recognized;

[0131] The recognition module 902 is used to input the face image to be recognized into the expression recognition model to perform expression recognition processing to obtain an expression recognition result.

[0132] It can be understood that the contents of the above-mentioned expression model training method embodiment are applicable to the present training system embodiment, the functions specifically implemented by the present training system embodiment are the same as those of the above-mentioned expression model training method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned expression model training method embodiment.

[0133] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned expression recognition model training method or expression recognition method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0134] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0136] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0137] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the training method or expression recognition method of the expression recognition model in the embodiment of this application;

[0138] Input / output interface 1003, used to implement information input and output;

[0139] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0140] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0141] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0142] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned expression recognition model training method or expression recognition method.

[0143] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The embodiments of the present application provide a training method for an expression recognition model, an expression recognition method, a system and a device. The scheme obtains a prediction result by inputting a training face image into an expression recognition model for separable deformable attention convolution processing. The scheme can introduce global attention information and a learnable offset through the separable deformable attention convolution, which not only retains the computational efficiency of the separable convolution, but also enables the convolution to adaptively find comprehensive image features that determine expression classification, reduces the problems of irregular facial angles and facial occlusions in the image, and improves the efficiency of expression recognition.

[0146] Specifically, the embodiment of the present application adopts a separable deformable attention convolution that can adaptively classify. The convolution introduces a learnable convolution position offset and global attention information. The synergy of the two in the convolution process enhances the flexibility and accuracy of the model.

[0147] The embodiment of the present application uses layer-deformable convolution to dynamically determine the points to be convolved using learnable convolution position offsets. This mechanism enables the convolution operation to no longer be limited to a fixed grid, but to adaptively focus on the feature areas in the image that are most critical for facial expression classification. This capability is particularly important in complex situations such as irregular facial angles or the presence of occlusions, because it allows the model to dynamically adjust to capture the most discriminative information.

[0148] The embodiment of the present application also uses the channel layer attention point convolution and introduces the global attention mechanism, so that the model can obtain higher-level abstract information in the image, which reflects the relative importance and relevance between different facial features. Using this global information to guide the learning of the offset not only makes the convolution operation more accurate, but also promotes the synergy between features, thereby helping to extract more comprehensive and integrated facial expression features.

[0149] Compared with traditional convolution methods, although deformable convolution also has the ability to adjust the position of convolution, it often lacks the guidance of global information; and although separable convolution decomposes the convolution operation into layer convolution and channel convolution to improve efficiency, it may sacrifice the continuity between features to a certain extent. The separable deformable attention convolution of the embodiment of the present application not only retains the computational efficiency of separable convolution by introducing global attention information and learnable offsets, but also overcomes the feature separation problem it may bring. At the same time, it uses global attention information to specify learnable offset variables, thereby achieving accurate capture and adaptive analysis of complex facial expression features. In addition, while improving the performance of the model, the embodiment of the present application also achieves an effective reduction in the amount of calculation, especially in terms of the reduction in the number of parameters.

[0150] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0151] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0152] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0154] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0155] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0156] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0157] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0160] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A training method for an expression recognition model, characterized in that: The training method comprises the following steps: Get training face images; Inputting the training face image into the expression recognition model to perform separate deformation attention convolution processing to obtain a prediction result; Calculate the prediction result according to the loss function to obtain a loss value; The parameters of the facial expression recognition model are updated according to the loss value to obtain a trained facial expression recognition model.

2. The method according to claim 1, characterized in that The step of inputting the training face image into the expression recognition model for separation deformation attention convolution processing to obtain a prediction result comprises the following steps: Performing feature extraction processing on the training face image to obtain an input feature map; Performing layer-deformable convolution processing on the input feature map to obtain an output feature map; Performing channel layer attention point convolution processing on the output feature map to obtain a target feature map; The target feature map is fully connected and activated to obtain the prediction result.

3. The method according to claim 2, characterized in that The step of performing layer deformable convolution processing on the input feature map to obtain an output feature map comprises the following steps: Assigning a separate deformable convolution kernel to each channel of the input feature map; The position offset variable and the deformed convolution kernel are combined to perform convolution calculation processing on any point on the channel to obtain the output feature map.

4. The method according to claim 2, characterized in that: The step of performing channel layer attention point convolution processing on the output feature map to obtain a target feature map comprises the following steps: Performing average pooling processing on the output feature map to obtain global attention information; Performing nonlinear convolution processing on the global attention information to obtain a first convolution result; Convolution processing is performed on each pixel of the output feature map in combination with the channel offset variable and the first convolution result to obtain the target feature map.

5. The method according to claim 1, characterized in that The step of calculating the prediction result according to the loss function to obtain the loss value comprises the following steps: Performing classification loss calculation processing on the prediction result to obtain a first loss result; Performing a central loss calculation process on the prediction result to obtain a second loss result; The first loss result and the second loss result are calculated and processed in combination with the hyperparameters to obtain the loss value.

6. A method for facial expression recognition, characterized in that: Used to perform expression recognition using an expression recognition model trained by the training method according to any one of claims 1 to 5, the expression recognition method comprising the following steps: Obtain a face image to be recognized; The face image to be recognized is input into the expression recognition model for expression recognition processing to obtain an expression recognition result.

7. A training system for an expression recognition model, characterized in that: The training system comprises: An acquisition module is used to acquire training face images; A prediction module, used for inputting the training face image into the expression recognition model to perform a separated deformation attention convolution process to obtain a prediction result; A loss module, used to calculate and process the prediction result according to a loss function to obtain a loss value; An updating module is used to update the parameters of the expression recognition model according to the loss value to obtain a trained expression recognition model.

8. An expression recognition system, characterized in that: The expression recognition system comprises: A capture module, used to obtain a face image to be identified; The recognition module is used to input the face image to be recognized into the expression recognition model for expression recognition processing to obtain an expression recognition result.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the training method of the expression recognition model described in any one of claims 1 to 5 or the expression recognition method described in claim 6 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the training method of the expression recognition model described in any one of claims 1 to 5 or the expression recognition method described in claim 6 is implemented.

Images