Facial expression representation model training method, facial expression recognition method and device

Abstract

The present disclosure provides a facial expression representation model training method, a facial expression recognition method and a device. It relates to the field of artificial intelligence technology, and in particular to computer vision, deep learning, image recognition, image retrieval and other technologies in the field of image processing. The specific implementation scheme is: obtaining a first image, a second image and a third image corresponding to each of N original images; inputting the first image into the first branch of the facial expression representation model, the second image into the second branch of the facial expression representation model, and the third image into the third branch of the facial expression representation model; obtaining the first facial expression representation output by the first branch, the second facial expression representation output by the second branch, and the third facial expression representation output by the third branch; training the facial expression representation model based on the first facial expression representation, the second facial expression representation and the third facial expression representation. According to the scheme of the present disclosure, the accuracy of facial expression representation can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to computer vision, deep learning, image recognition, image retrieval and other technologies in the field of image processing. Background Art

[0002] Facial expression recognition technology primarily relies on supervised training using large, manually annotated datasets. However, due to the high cost of facial expression annotation, datasets for expression recognition are smaller than those for general image classification, limiting the ability of deep learning models to learn from large datasets. Furthermore, due to annotation biases between datasets, facial expression annotation results for the same image may vary. Mixing multiple datasets for training can lead to even worse results, resulting in lower accuracy in facial expression representation. Summary of the Invention

[0003] The present disclosure provides a facial expression representation model training method, a facial expression recognition method and a facial expression recognition device.

[0004] According to a first aspect of the present disclosure, a method for training a facial expression representation model is provided, comprising: obtaining data to be trained, the data to be trained comprising N original images, where N is an integer not less than 1; obtaining a first image, a second image, and a third image corresponding to each of the N original images, wherein the first image is an image obtained by performing a first random augmentation process on the original image; the second image is an image obtained by performing a second random augmentation process on the original image; and the third image is an image obtained by performing a random deformation process on the first image; inputting the first image into a first branch of a facial expression representation model, the second image into a second branch of the facial expression representation model, and the third image into a third branch of the facial expression representation model; obtaining a first facial expression representation output by the first branch, a second facial expression representation output by the second branch, and a third facial expression representation output by the third branch; and training the facial expression representation model based on the first facial expression representation, the second facial expression representation, and the third facial expression representation.

[0005] According to a second aspect of the present disclosure, a facial expression recognition method is provided, comprising: obtaining an image to be recognized; inputting the image to be recognized into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model; inputting the facial expression representation into a classifier to obtain an expression classification result of the image to be recognized output by the classifier; wherein the facial expression representation model is obtained by training using the method provided in the first aspect.

[0006] According to a third aspect of the present disclosure, a facial expression representation model training device is provided, including: a first acquisition module for acquiring data to be trained, the data to be trained including N original images, where N is an integer not less than 1; a second acquisition module for acquiring a first image, a second image, and a third image corresponding to each of the N original images, the first image being an image obtained by performing a first random augmentation process on the original image; the second image being an image obtained by performing a second random augmentation process on the original image; and the third image being an image obtained by performing a random deformation process on the first image; a first input module for inputting the first image into the first branch of the facial expression representation model, the second image into the second branch of the facial expression representation model, and the third image into the third branch of the facial expression representation model; a third acquisition module for acquiring a first facial expression representation output by the first branch, a second facial expression representation output by the second branch, and a third facial expression representation output by the third branch; and a training module for training the facial expression representation model based on the first facial expression representation, the second facial expression representation, and the third facial expression representation.

[0007] According to the fourth aspect of the present disclosure, a facial expression recognition device is provided, comprising: a fifth acquisition module for acquiring an image to be recognized; a third input module for inputting the image to be recognized into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model; a recognition module for inputting the facial expression representation into a classifier to obtain an expression classification result of the image to be recognized output by the classifier; wherein the facial expression representation model is obtained by training using the method provided in the first aspect.

[0008] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; a memory communicatively connected to the at least one processor; the memory storing instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to execute the facial expression representation model training method provided in the first aspect and / or the facial expression recognition method provided in the second aspect.

[0009] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the facial expression representation model training method provided by the first aspect and / or the facial expression recognition method provided by the second aspect.

[0010] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the facial expression representation model training method provided in the first aspect and / or the facial expression recognition method provided in the second aspect.

[0011] According to the technical solution disclosed in the present invention, the accuracy of facial expression representation can be improved.

[0012] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0014] Figure 1 is a flowchart of a facial expression representation model training method according to an embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram of the principle of random deformation processing according to an embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram of the effect of random deformation processing according to an embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram of the overall structure of a facial expression representation model according to an embodiment of the present disclosure;

[0018] Figure 5 is a flowchart of a facial expression recognition method according to an embodiment of the present disclosure;

[0019] Figure 6 is a flowchart of facial expression recognition according to an embodiment of the present disclosure;

[0020] Figure 7 is a structural diagram of a facial expression representation model training device according to an embodiment of the present disclosure;

[0021] Figure 8 is a structural diagram of a facial expression recognition device according to an embodiment of the present disclosure;

[0022] Figure 9 is a schematic diagram of a scenario for training a facial expression representation model according to an embodiment of the present disclosure;

[0023] Figure 10 is a schematic diagram of a scene of facial expression recognition according to an embodiment of the present disclosure;

[0024] Figure 11 Schematic diagram of the structure of an electronic device used to implement the facial expression representation model training method and / or facial expression recognition method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] The terms "first," "second," and "third," etc. in the description, embodiments, claims, and figures of the present disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, for example, including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to the process, method, product, or apparatus.

[0027] In related technologies, due to the potential for discrepancies in labeling between datasets, resulting in different labeling results for the same image, most use a pre-trained face recognition model to initialize the backbone network and then fine-tune the model on less labeled expression data to train the facial expression representation model. The primary goal of the face recognition task is to extract identity-related features, thereby minimizing the influence of age, expression, and other factors on identity-related features. Therefore, while the pre-trained face recognition model can extract information from facial images, the expression information is suppressed and needs to be re-learned during the fine-tuning phase, resulting in low efficiency in facial expression recognition.

[0028] In related technologies, self-supervised methods can learn from unlabeled images through a proxy task. This proxy task primarily involves contrastive learning. Specifically, it enables the model to distinguish different augmentations of the same image (such as random cropping, blurring, and color perturbations) from the augmentations of other images, ensuring that the features generated by different augmentations of the same image are highly similar. This proxy task allows the model to extract feature representations relevant to the image to be recognized. Since self-supervised methods were developed for general object recognition, primarily for identifying the category of objects in the foreground, facial expression recognition uses only facial images as input, and the differences between different categories are very small. At the same time, the differences within the same category can be significant. For example, for the "happy" category, different people can display very different expressions. From a "light smile" to a "laughing out loud," although both categories are the same, the expressions displayed differ significantly. Therefore, self-supervised methods struggle to effectively learn features related to facial expressions.

[0029] In order to at least partially solve one or more of the above problems and other potential problems, the present disclosure proposes a facial expression representation solution that can improve the accuracy of facial expression representation.

[0030] The present disclosure provides a method for training a facial expression model. Figure 1 It is a flow chart of a facial expression representation model training method according to an embodiment of the present disclosure, and the facial expression representation model training method can be applied to a facial expression representation model training device. The facial expression representation model training device is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the server can be a cloud server or an ordinary server. For example, mobile devices include but are not limited to mobile phones, tablet computers, and vehicle-mounted terminals. In some possible implementations, the facial expression representation model training method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the facial expression representation model training method includes:

[0031] S101: Acquire data to be trained, where the data to be trained includes N original images;

[0032] S102: Obtain a first image, a second image, and a third image corresponding to each of N original images, wherein the first image is an image obtained by performing a first random augmentation process on the original image, the second image is an image obtained by performing a second random augmentation process on the original image, and the third image is an image obtained by performing a random deformation process on the first image;

[0033] S103: inputting the first image into the first branch of the facial expression representation model, the second image into the second branch of the facial expression representation model, and the third image into the third branch of the facial expression representation model;

[0034] S104: Acquire a first facial expression representation output by the first branch, a second facial expression representation output by the second branch, and a third facial expression representation output by the third branch;

[0035] S105: Training a facial expression representation model based on the first facial expression representation, the second facial expression representation, and the third facial expression representation.

[0036] In the embodiment of the present disclosure, N is an integer not less than 1.

[0037] In the embodiments of the present disclosure, any feasible implementation method may be used to obtain the training data. For example, the training data may be obtained from an image database. In another example, available images may be captured from the Internet as the training data. In another example, the training data may be obtained from other electronic devices. The above is merely illustrative and does not limit all possible methods for obtaining training data. This is not intended to be an exhaustive list.

[0038] In the disclosed embodiments, the original image is an image that has not undergone augmentation processing. For example, it can be a selfie of a single person. Another example is a group photo. Another example is a photo of a person and a landscape. The above description is merely illustrative and does not limit all possible types of original images. This is not intended to be an exhaustive list.

[0039] In the disclosed embodiments, random augmentation processing includes, but is not limited to, random cropping, left-right flipping, color perturbation, noise perturbation, and Gaussian blur processing. Color perturbation may include adjusting at least one of hue, saturation, contrast, and brightness. The above is merely illustrative and does not limit all possible random augmentation methods. This list is not exhaustive.

[0040] In the embodiment of the present disclosure, the first random augmentation process includes: performing one or more random augmentation processes on the image. For example, blurring the image. For another example, color perturbation is performed on the image. For another example, noise perturbation is performed on the image. The first random augmentation process may include one or more of the processes in the above examples. In order to obtain better training results, the image sizes of the first image and the original image are kept equal as much as possible. Therefore, the first augmentation process does not include random cropping. The above is only an illustrative description and does not limit all possible processing methods of the first random augmentation process. It is just not exhaustive here.

[0041] In the disclosed embodiments, the first image is obtained by performing a first random augmentation process on the original image. For example, the first image may be obtained by blurring and noise perturbing a group photo. Another example is the first image obtained by color perturbing a single-person selfie. The above description is merely illustrative and does not limit all possible ways to obtain the first image. This is not intended to be an exhaustive list.

[0042] In the embodiment of the present disclosure, the second random augmentation processing includes performing one or more random augmentation processing on the image. In practical applications, in order to make the data diverse, different second random augmentation processing can be used for different original images, and different second random augmentation processing can also be used for the same original image. For example, the image is blurred. For another example, the image is color perturbed. For another example, the image is randomly cropped. The second augmentation processing may include at least one of random cropping, left-right flipping, color perturbation, noise perturbation, and Gaussian blur processing. The above is only an exemplary description and does not limit all possible processing methods of the second random augmentation processing, but it is not exhaustive here.

[0043] In the disclosed embodiments, the second image is obtained by performing a second random augmentation process on the original image. For example, the second image can be obtained by randomly cropping and color perturbing a group photo. Another example is the second image obtained by performing noise perturbation on a single selfie. The above description is merely illustrative and does not limit all possible ways to obtain the second image. This is not intended to be an exhaustive list.

[0044] Facial expressions are mainly expressed through the movement of facial muscles, such as frowning, raising the corners of the mouth, etc. The method of face deformation can be used to simulate these muscle movements, that is, random deformation.

[0045] In the embodiment of the present disclosure, the random deformation can be specifically: define a circle with a center of c and a radius of r max The random deformation occurs only within the circular area. A new position m is defined, and the center c is moved to this new position. Simultaneously, all image pixels within the circular area are shifted a certain distance in the direction from c to m. This shift is inversely proportional to the distance from the center. That is, as the center c moves to the new position m, the distance pixels move decreases as they move farther from the center, until no pixels on the circumference move at all. By repeating this pixel shifting process, the movement of different facial muscles can be simulated. This helps improve the smoothness of the random deformation effect.

[0046] In the disclosed embodiments, the third image is obtained by randomly deforming the first image. For example, a group photo can be blurred and noise-perturbed to obtain the first image; a circular region is defined within which pixels in the first image are shifted multiple times to obtain the third image. The above description is merely illustrative and does not limit all possible ways to obtain the third image; however, this is not an exhaustive list.

[0047] In the disclosed embodiment, the first branch may include a backbone network (such as an online backbone network) and a multi-layer perceptron (MLP), wherein the MLP includes a neural network projector and a predictor. The second branch may include a backbone network (such as a target backbone network) and the MLP; and the third branch may include a backbone network (such as a target backbone network) and the MLP.

[0048] In the disclosed embodiment, the first branch outputs a first facial expression representation; the second branch outputs a second facial expression branch; and the third branch outputs a third facial expression branch. The first, second, and third facial expression representations can be feature maps or feature vectors obtained by projecting the feature maps.

[0049] Among them, the model to be trained is used to realize the prediction of the first facial expression representation, the second facial expression representation, and the third facial expression representation of the image according to the original image included in the data to be trained. Based on this, it can be understood that in the embodiment of the present disclosure, the model to be trained may include at least one feature extraction model, which is at least used to extract the features of the first image, the features of the second image, and the features of the third image; at least one prediction model, which is used to predict the first facial expression representation of the first image, the second facial expression representation of the second image, and the third facial expression representation of the third image, and the prediction model at least includes a first prediction model, a second prediction model, and a third prediction model, wherein the first prediction model is used to predict the predicted value of the first facial expression representation, the second prediction model is used to predict the predicted value of the second facial expression representation, and the third prediction model is used to predict the third facial expression representation. The facial expression representation model is obtained by training the model to be trained with the data to be trained, and therefore has the same model structure as the model to be trained, except that the model parameters are updated after training.

[0050] The technical solution of the embodiment of the present disclosure is to obtain data to be trained, which includes N original images; obtain a first image, a second image, and a third image corresponding to each of the N original images; input the first image into the first branch of the facial expression representation model, the second image into the second branch of the facial expression representation model, and the third image into the third branch of the facial expression representation model; train the facial expression representation model based on the first facial expression representation output by the first branch, the second facial expression representation output by the second branch, and the third facial expression representation output by the third branch. The first image is an image obtained by performing a first random augmentation process on the original image, the second image is an image obtained by performing a second random augmentation process on the original image, and the third image is an image obtained by performing a random deformation process on the first image; based on the facial expression representations output by the first branch, the second branch, and the third branch, the facial expression representation model effectively learns the features related to facial expressions, improves the precision of the facial expression representation model, and thus improves the accuracy of facial expression representation. Since the accuracy of facial expression representation is improved, it helps to further improve the accuracy of facial expression recognition.

[0051] In some embodiments, S102 may include:

[0052] S102a: Determine a circular area;

[0053] S102b: Pixel shifting is performed on the first images corresponding to the N original images in the circular area to obtain third images corresponding to the N original images.

[0054] Figure 2 The schematic diagram of the principle of random deformation processing in the facial expression representation model is shown in FIG. Figure 2 As shown, determine a circle with center c and radius r max All deformations occur only within the circular area. To ensure the quality of the third image, the drawing of the circular area can be limited according to the image size. For example, for a 224×224 face image, the value of the center c can be: 50≤c≤150; the radius r max The value of can be 50≤r max ≤80.

[0055] In the disclosed embodiment, a new position m is defined, and the center of the circle c is moved to the new position m. Simultaneously, all image pixels within the circular region are shifted a certain distance in the direction from c to m. This shift distance is inversely proportional to the distance from the center of the circle. That is, as the center of the circle c moves to the new position m, the farther the pixel is from the center of the circle, the smaller the shift distance, until no pixels on the circumference move at all. By repeating this pixel shifting process, the movement of different facial muscles is simulated.

[0056] In the embodiment of the present disclosure, based on any point u in the circular area, its position after deformation is set as x. That is, in the deformed face image, the pixel value corresponding to point x comes from point u before deformation, and the coordinates of point u can be solved by formula (1):

[0057]

[0058] Since the calculations between each point are independent of each other, parallel calculations can be performed based on formula (1) to obtain the randomly deformed face image.

[0059] In this way, obtaining the randomly deformed face image through parallel computing helps to efficiently obtain the third image, thereby helping to increase the speed of model training.

[0060] In some embodiments, S102b may include:

[0061] S102b′: Perform K shifts on the first images corresponding to the N original images in the circular area to obtain third images corresponding to the N original images, where K is an integer not less than 2. The Kth shift is a pixel shift based on the result of the K-1th shift.

[0062] In this disclosed embodiment, the first image corresponding to each of N original images is shifted K times within a circular area to obtain a third image corresponding to each of the N original images. If the first and second shifts occur in the forehead area, the change in the portrait's expression is not noticeable. Therefore, setting the number of pixel shifts to no less than 2 helps the facial expression representation model learn higher-quality expression features.

[0063] In the disclosed embodiment, the first movement occurs anywhere within the circular area. The moving area of the Kth movement can be moved based on the area of the K-1th movement, or can be moved to other areas of the circular area. For example, if the first movement occurs at the forehead of a portrait, and the K-1th movement occurs at the corner of the portrait's mouth, then the Kth movement can be performed in the area of the corner of the mouth in the K-1th movement, or in an area other than the corner of the mouth.

[0064] Figure 3 shows a schematic diagram of the random deformation effect, Figure 3 The first column of images in the figure is the original image, while the second, third, and fourth columns are randomly deformed images. In the first row, a neutral face is randomly deformed to express confusion, anger, and pity. In the second row, while the smile remains unchanged, the facial muscles move, conveying different expressions to a certain extent. For example, facial muscle movement is detected using action units (AUs).

[0065] In this way, through the random deformation task, the facial expression representation model can learn more effective expression features, which helps to improve the accuracy of facial expression representation.

[0066] In some embodiments, S104 may include:

[0067] S104a: Inputting the third image corresponding to each of the N original images into the backbone network of the third branch for feature extraction to obtain the third facial expression representation corresponding to each of the N original images.

[0068] In the embodiment of the present disclosure, the third branch may include a backbone network and an MLP, input the original image into the facial expression representation model, obtain the first image through a first random augmentation process, randomly deform the first image to obtain a third image; input the third image into the backbone network of the third branch to obtain a third facial expression representation.

[0069] Here, the third face representation can be a feature map, or a feature vector obtained by a projector based on the feature map.

[0070] In this way, by performing feature extraction on the third image obtained by random deformation processing and determining the third facial expression representation, richer expression feature data can be provided for training the facial expression representation model, and the accuracy of the facial expression representation model can be improved, thereby helping to improve the accuracy of the expression representation based on facial expression recognition.

[0071] In some embodiments, the facial expression representation model training method may further include:

[0072] S106: Inputting the first facial expression representation output by the first branch and the third facial expression representation output by the third branch into the fourth branch of the facial expression representation model;

[0073] S107: Obtain a first key point distribution map obtained by the fourth branch based on the first facial expression representation prediction, and a second key point distribution map obtained by the fourth branch based on the third facial expression representation prediction; wherein the first key point distribution map is used to represent the key point distribution before the image is deformed, and the second key point distribution map is used to represent the key point distribution after the image is deformed.

[0074] In the disclosed embodiment, the fourth branch is part of the facial expression representation model and is located downstream of the third branch. The fourth branch receives as input the first facial expression representation output by the first branch and the third facial expression representation output by the third branch, and outputs the first keypoint distribution map and the second keypoint distribution map.

[0075] In the embodiment of the present disclosure, the fourth branch may include a facial key point prediction (landmark detector). The first facial expression representation and the third facial expression representation are input into the landmark detector to obtain a first key point distribution map and a second key point distribution map.

[0076] Here, the first key point distribution graph and the second key point distribution graph have the same number of key points, for example, both are 68.

[0077] In the embodiment of the present disclosure, based on the comparison between the first key point distribution map and the second key point distribution map, the key points whose positions remain unchanged before and after deformation and the key points whose positions change can be obtained; the weights of the key points whose positions have not changed are set to the first weight such as 0.1, and the weights of the key points whose positions have changed are set to the second weight such as 1, which helps to encourage the facial expression representation model to focus on the moving area.

[0078] In the disclosed embodiment, based on the above-mentioned random deformation task, through the first branch, the second branch and the third branch, the features of the same face image after random deformation have a low similarity (low similarity) with the features before random deformation.

[0079] First, a face image is input. The face image undergoes a first random augmentation process to obtain a first image x1. The face image undergoes a second random augmentation process to obtain a second image x2. Through comparative learning, features z1 and z2 are extracted from the first image x1 and the second image x2, respectively. The first facial expression representation z1 is obtained by the predictor of the first branch to obtain p1, and p1 has a high similarity with z2 of the second branch. At the same time, the second facial expression representation z2 is obtained by the predictor of the second branch to obtain p2, and p2 has a high similarity with z1 of the first branch. The similarity value is calculated using formula (2):

[0080]

[0081] Next, the first image x1 is randomly deformed to obtain the third image x3. Aside from muscle movement, the third image x3 shares the same features (identity, age, posture, background, etc.) as the first image x1, helping to extract features related to facial expressions.

[0082] Here, forcing the model to extract features from the third image x3 with a lower similarity to the first image x1 can make the facial expression representation model pay more attention to muscle movement, thereby ignoring irrelevant information such as the identity and age of the face.

[0083] In the disclosed embodiment, a key point prediction task is provided, that is, based on the facial expression representation (feature map) obtained from the first image x1 and the third image x3, the positions of the key points before and after deformation are predicted respectively. The purpose of key point prediction is not to enable the facial expression representation model to accurately identify the positions of each key point, but to serve as additional information to enable the facial expression representation model to pay attention to the moving muscles. Therefore, the training method of the facial expression representation model does not require particularly accurate key point information that is manually labeled. In addition, the pseudo-labels generated by the face detection model pre-trained on other data sets can help improve the training effect of the facial expression representation model.

[0084] Figure 4 The overall structure diagram of the facial expression representation model is shown in Figure 4 As shown, the original image is input into the facial expression representation model, and the first image x1 is obtained through the first random augmentation process t, and the second image x2 is obtained through the second random augmentation t′; the first image x1 is input into the third branch for random warping to obtain the third image x3; the third image x3 is subjected to the target backbone to extract features to obtain the third facial expression representation f3, and the third facial expression representation f3 is subjected to the projector to obtain the third feature z3; the first image x1 is input into the first branch, and the first image x1 is subjected to the online backbone to extract features to obtain the first facial expression representation f1, and the first facial expression representation f1 is subjected to the projector to obtain the first feature z1, and the first feature z1 is input into the predictor of the first branch to obtain the predicted feature p1; the first facial expression representation f1 obtained based on the first branch and the third facial expression representation f3 obtained by the third branch are input into the fourth branch, and the first key point distribution map and the second key point distribution map are obtained through the landmark detector; the second image x2 is input into the second branch and the target backbone is subjected to the target backbone to extract features to obtain the first facial expression representation f1, and the first facial expression representation f1 is subjected to the projector to obtain the first feature z1, and the first feature z1 is input into the predictor of the first branch to obtain the predicted feature p1; the first facial expression representation f1 obtained based on the first branch and the third facial expression representation f3 obtained by the third branch are input into the fourth branch, and the first key point distribution map and the second key point distribution map are obtained through the landmark detector; the second image x2 is input into the second branch and the target backbone is subjected to the target backbone The backbone extracts features to obtain the second facial expression representation f2. The second facial expression representation f2 is passed through the projector to obtain the second feature z2. The second feature z2 is input to the projector to obtain the predicted feature p2.

[0085] Here, the parameters of the target backbone of the second branch can be consistent with those of the target backbone of the third branch; the online backbone of the first branch shares the parameters with the target backbone of the second branch through moving average.

[0086] In this way, through random deformation tasks and key point prediction tasks, the model can learn better expression features, improve the accuracy of facial expression representation model training, thereby improving the accuracy of facial expression representation and the accuracy of facial expression recognition.

[0087] In some embodiments, S105 may include:

[0088] S105a: Determining a first loss function based on the first facial expression representation and the second facial expression representation;

[0089] S105b: Determining a second loss function based on the first facial expression representation and the third facial expression representation;

[0090] S105c: Determine a third loss function based on the first key point distribution map and the second key point distribution map;

[0091] S105d: Determine a total loss function based on the first loss function, the second loss function, and the third loss function;

[0092] S105e: Train a facial expression representation model based on the total loss function.

[0093] In the embodiment of the present disclosure, the first loss function can be obtained by formula (3):

[0094] L cont12 =-sim(1,2) (3)

[0095] Here, determining the first loss function based on the first facial expression representation obtained by the first branch and the second facial expression representation obtained by the second branch includes: determining the first loss function based on the feature similarity between the first facial expression representation and the second facial expression representation.

[0096] In the embodiment of the present disclosure, the second loss function can be obtained by formula (4):

[0097] L cont13 =max(sim(1,3),s t ) (4)

[0098] Here, based on the first facial expression representation obtained by the first branch and the third facial expression representation obtained by the third branch, the second loss function is determined, including: based on the feature similarity of the first facial expression representation and the third facial expression representation, and a preset threshold, the second loss function is determined.

[0099] Among them, sim(1,3) is the feature similarity between the first person’s facial expression and the third person’s facial expression, s t is the preset threshold.

[0100] Here, s tThe value can be set or adjusted according to the speed or accuracy requirements. t =0.6, for example, when sim(1,3)=0.4, L cont13 =0.6; when sim(1,3)=0.8, L cont13 =0.8.

[0101] In the embodiment of the present disclosure, the third loss function can be obtained by formula (5):

[0102]

[0103] Here, based on the first key point distribution map and the second key point distribution map obtained by the fourth branch, the third loss function is determined. Among them, the mean square error (MSE) is used to measure the matching degree between the first key point distribution map and the second key point distribution map. Among them, pred i Indicates the predicted value of the key point, pseudo i Pseudo labels representing key points, w i Represents the weight given to each key point. Specifically, for key points whose positions have not changed, W i = first weight; for key points whose positions change, w i = Second weight. It's important to note that keypoint prediction here isn't intended to accurately identify the locations of keypoints, but rather to provide additional information to help the model focus on moving muscles. Using pseudo-labels generated by a face detection model pre-trained on other datasets yields excellent results. Therefore, the keypoint task here doesn't rely on manual labeling.

[0104] In the embodiment of the present disclosure, the third loss function can be obtained by formula (6):

[0105] L=L cont12 +L cont13 +λ(L landmark1 +L landmark3 ) (6)

[0106] Here, L landmark1 represents the third loss function obtained based on the first key point distribution map; L landmark3 Denotes the third loss function obtained based on the second key point distribution map, and λ denotes the weight of the third loss function.

[0107] Here, λ can be set or adjusted according to the speed or accuracy requirements. For example, λ=10.

[0108] In the embodiment of the present disclosure, S105a, S105b, and S105c can be executed simultaneously. Alternatively, S105a is executed before S105b and S105c, and S105b and S105c are executed simultaneously. Alternatively, S105b is executed before S105a and S105c, and S105a and S105c are executed simultaneously. Alternatively, S105c is executed before S105a and S105b, and S105a and S105b are executed simultaneously. Alternatively, S105a, S105b, and S105c are executed sequentially. Alternatively, S105a, S105b, and S105c are executed sequentially. Alternatively, S105a, S105c, and S105b are executed sequentially. Alternatively, S105b, S105a, and S105c are executed sequentially. Alternatively, S105b, S105a, and S105c are executed sequentially. Alternatively, S105b, S105c, and S105a are executed sequentially. Alternatively, S105b, S105c, and S105a are executed sequentially. Alternatively, the execution is performed in the order of S105c, S105a, and S105b. Alternatively, the execution is performed in the order of S105c, S105b, and S105a. The above is merely an example, and the present disclosure does not limit the execution order of S105a, S105b, and S105c, but this is not an exhaustive list.

[0109] In this way, when training the facial expression representation model, the total loss function is calculated by the first loss function, the second loss function and the third loss function. Training the facial expression representation model based on the total loss function can improve the accuracy of the facial expression representation model, thereby improving the accuracy of the facial expression representation output by the facial expression representation model.

[0110] In the disclosed embodiment, S105a specifically includes: performing spatial transformation on the first facial expression representation to obtain a transformed first facial expression representation; performing prediction based on the transformed first facial expression representation to obtain a predicted second facial expression representation; performing spatial transformation on the second facial expression representation to obtain a transformed second facial expression representation; performing prediction based on the transformed second facial expression representation to obtain a predicted first facial expression representation; determining a first loss function based on the predicted first facial expression representation, the transformed first facial expression representation, the predicted second facial expression representation, and the converted second facial expression representation.

[0111] In the embodiment of the present disclosure, the space is converted into a vector by inputting the feature map into the global average pooling (GAP). For example, the first facial expression representation f1 is a feature map, and the feature map includes three dimensions: length, width, and channel. The feature map is subjected to global average pooling, and the feature map is converted into a vector to obtain the converted first facial expression representation Z1; the converted first facial expression representation Z1 is input into the predictor of the first branch to obtain the predicted second facial expression representation p1. For example, the second facial expression representation f2 is a feature map, and the feature map includes three dimensions: length, width, and channel. The feature map is subjected to global average pooling, and the feature map is converted into a vector to obtain the converted second facial expression representation Z2; the converted first facial expression representation Z2 is input into the predictor of the second branch to obtain the predicted second facial expression representation p2.

[0112] Here, the calculation formula of the first loss function can refer to formula (3), which will not be repeated here.

[0113] In this way, determining the first loss function based on the predicted first facial expression representation, the converted first facial expression representation, the predicted second facial expression representation, and the converted second facial expression representation can improve the training accuracy of the facial expression representation model.

[0114] In the embodiment of the present disclosure, S105b specifically includes: performing spatial transformation on the first facial expression representation to obtain a transformed first facial expression representation; performing prediction based on the transformed first facial expression representation to obtain a predicted second facial expression representation; performing spatial transformation on the third facial expression representation to obtain a transformed third facial expression representation; and determining a second loss function based on the predicted second facial expression representation and the transformed third facial expression representation.

[0115] Here, the calculation formula of the second loss function can refer to formula (4), which will not be repeated here.

[0116] In this way, determining the second loss function based on the predicted second facial expression representation and the converted third facial expression representation can improve the training accuracy of the facial expression representation model.

[0117] In the embodiment of the present disclosure, S105c specifically includes: comparing the first key point distribution map and the second key point distribution map to obtain key points whose positions remain unchanged and key points whose positions change; determining the first weight corresponding to the key points whose positions remain unchanged, and the second weight corresponding to the key points whose positions change, the first weight being less than the second weight; determining the third loss function based on the key points whose positions remain unchanged, the first weight, the key points whose positions change, and the second weight.

[0118] Here, the calculation formula of the third loss function can refer to formula (5), which will not be repeated here.

[0119] In the embodiment of the present disclosure, the first key point distribution map is used to characterize the key point distribution before the image is deformed, and the second key point distribution map is used to characterize the key point distribution after the image is deformed; based on the comparison between the first key point distribution map and the second key point distribution map, the key points whose positions remain unchanged before and after the deformation and the key points whose positions change can be obtained; the key points whose positions remain unchanged before and after the deformation are assigned a first weight, and the key points whose positions change before and after the deformation are assigned a second weight, and the first weight is smaller than the second weight, which helps to encourage the facial expression representation model to focus on the moving area.

[0120] In this way, by assigning different weights to key points whose positions have not changed and key points whose positions have changed, the training set accuracy of the facial expression representation model can be improved.

[0121] It should be understood that Figure 2 、 Figure 3 and Figure 4 The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figure 2 、 Figure 3 and Figure 4 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0122] The embodiment of the present disclosure provides a method for recognizing facial expressions, which can be applied to electronic devices. Figure 5 The flowchart diagram shown illustrates a facial expression recognition method provided by an embodiment of the present disclosure. It should be noted that although a logical order is shown in the flowchart diagram, in some cases, the steps shown or described may also be performed in other orders.

[0123] S501: Acquire an image to be recognized;

[0124] S502: Inputting the image to be recognized into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model;

[0125] S503: Inputting the facial expression representation into the classifier to obtain the expression classification result of the image to be recognized output by the classifier.

[0126] The facial expression representation model is obtained by training using the facial expression representation model training method described above.

[0127] In the disclosed embodiments, the image to be identified may contain one or more people. For example, it may be a selfie of a single person. Another example may be a group photo. Another example may be a photo of a person and an animal. The above description is merely illustrative and does not limit all possible types of images to be identified. This is not intended to be an exhaustive list.

[0128] In the embodiments of the present disclosure, any feasible implementation method is used to obtain the image to be identified. For example, the image to be identified can be obtained from an image database. In another example, an image captured from the Internet can be used as the image to be identified. In another example, the image to be identified can be obtained from other electronic devices such as cameras. The above is merely illustrative and does not limit all possible methods for obtaining the image to be identified. This is not intended to be an exhaustive list.

[0129] In the embodiment of the present disclosure, the classifier may be a linear classifier or any other classifier that can achieve classification. The above is merely an example and does not limit the specific type of classifier, but is not exhaustive.

[0130] Based on the facial expression representation model trained in this paper, the backbone network of the first branch of the facial expression representation model is fine-tuned using a small amount of labeled datasets and deployed for expression classification or retrieval. Expression classification can be used to map facial expression representations to multiple pre-defined classification labels, such as anger, disgust, fear, happiness, sadness, surprise, and neutral. It can also be used for expression retrieval to find similar expressions within the same category, such as finding smiles or grins.

[0131] Exemplarily, an image to be identified is obtained from mobile phone A, where the image to be identified is a selfie portrait. The selfie portrait is input into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model; the facial expression representation is input into a linear classifier, and the expression classification result of the selfie portrait output by the linear classifier is happy.

[0132] This improves the training speed and accuracy of facial expression representation models, compared to supervised training based on larger, manually annotated datasets. By using random deformations to simulate facial muscle movement, generating different expression samples, and by scaling features before and after deformation, the model learns features related to expression, thereby learning expression information without manual annotation. This improves the accuracy of facial expression recognition by using the facial expression representation output by this facial expression representation model.

[0133] In some embodiments, S502 may include:

[0134] S502a: Input the image to be recognized into the first branch of the facial expression representation model to obtain the facial expression representation output by the first branch.

[0135] In the embodiment of the present disclosure, the facial expression representation model includes a first branch. The first branch includes a backbone network and a GAP. It should be noted that, Figure 4 The projector and predictor in are both MLPs and are only used in pre-training. Therefore, the trained facial expression representation model does not include MLP.

[0136] Here, the facial expression representation may be the facial expression representation output from the backbone network. In this case, the specific representation form of the facial expression representation is a feature map.

[0137] Here, the facial expression representation may also be the facial expression representation output from GAP. In this case, the specific representation form of the facial expression representation is a vector.

[0138] Figure 6 The flowchart of facial expression recognition is shown in FIG. Figure 6 As shown, the image to be recognized is input into the first branch of the facial expression representation model, which then outputs a facial expression representation. The image to be recognized passes through the backbone network of the first branch to obtain a feature map. This is then processed through GAP to obtain a facial expression representation (a vector), which is then passed through a linear classifier to obtain the expression classification result for the image to be recognized.

[0139] Exemplarily, an image to be recognized is obtained from computer B, where the image to be recognized is a group photo of multiple people. The group photo is input into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model. The specific representation form of the facial expression representation is a feature map. The facial expression representation is input into a linear classifier to obtain an expression classification result of each person in the group photo output by the linear classifier, where some people are smiling and some are laughing.

[0140] Exemplarily, an image to be recognized is obtained from camera C, where the image to be recognized is an image of a driver driving. The driver driving image is input into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model. The specific representation form of the facial expression representation is a vector. The facial expression representation is input into a linear classifier, and the expression classification result of the driver driving image output by the linear classifier is surprise.

[0141] In this way, by determining the facial expression representation through the first branch of the facial expression representation model, the speed of facial expression recognition can be improved.

[0142] Compared to feature reconstruction algorithms such as self-supervised learning methods (Bootstrap Your Own Latent, BYOL), twin networks (SimSiam) and contrastive learning algorithms (Momentum Contrast for Unsupervised Visual Representation Learing, MoCo), the present invention can more effectively obtain expression-related features from unsupervised data. Table 1 shows that after the above algorithms are combined with the two proxy tasks of random deformation and key point prediction of the present invention, MoCo, SimSiam and BYOL can achieve better results. Among them, in Table 1, "+Ours" means that on the basis of a certain algorithm, the two proxy tasks of random deformation and key point prediction of the present invention are added.

[0143]

[0144] Table 1

[0145] Therefore, based on the random deformation task and key point prediction task, expression-related features can be more effectively obtained from unsupervised data, the accuracy of facial expression representation model training can be improved, and thus the accuracy of facial expression recognition can be improved.

[0146] This paper proposes a highly efficient and accurate facial expression recognition solution. This solution can learn expression-related features from large amounts of data without manual annotation, improving the training efficiency of facial expression representation models. Furthermore, by effectively learning expression features, it brings higher accuracy to applications related to facial expression recognition, such as fatigue driving detection, virtual reality, and human-computer interaction systems.

[0147] The present disclosure provides a facial expression model training device, such as Figure 7As shown, the facial expression representation model training device may include: a first acquisition module 701, used to obtain data to be trained, the data to be trained includes N original images, N is an integer not less than 1; a second acquisition module 702, used to obtain a first image, a second image and a third image corresponding to each of the N original images, the first image is an image obtained by performing a first random augmentation process on the original image; the second image is an image obtained by performing a second random augmentation process on the original image; the third image is an image obtained by performing a random deformation process on the first image; a first input module 703, used to input the first image into the first branch of the facial expression representation model, the second image into the second branch of the facial expression representation model, and the third image into the third branch of the facial expression representation model; a third acquisition module 704, used to obtain a first facial expression representation output by the first branch, a second facial expression representation output by the second branch, and a third facial expression representation output by the third branch; a training module 705, used to train the facial expression representation model based on the first facial expression representation, the second facial expression representation and the third facial expression representation.

[0148] In some embodiments, the second acquisition module 702 includes: a first determination submodule for determining a circular area; a pixel movement submodule for performing pixel movement on the first image corresponding to each of the N original images in the circular area to obtain a third image corresponding to each of the N original images.

[0149] In some embodiments, the pixel shift submodule is used to: perform K shifts on the first images corresponding to each of the N original images in a circular area to obtain third images corresponding to each of the N original images, where K is an integer not less than 2; wherein the Kth shift is a pixel shift performed based on the result of the K-1th shift.

[0150] In some embodiments, the third acquisition module 704 includes: a feature extraction submodule, which is used to input the third image corresponding to each of the N original images into the backbone network of the third branch for feature extraction to obtain the third facial expression representation corresponding to each of the N original images.

[0151] In some embodiments, the facial expression model training device further includes: a second input module 706 ( Figure 7 (not shown), for inputting the first facial expression representation output by the first branch and the third facial expression representation output by the third branch into a fourth branch of the facial expression representation model; a fourth acquisition module 707 ( Figure 7(not shown in the figure), used to obtain a first key point distribution map predicted by the fourth branch based on the first facial expression representation, and a second key point distribution map predicted by the fourth branch based on the third facial expression representation; wherein the first key point distribution map is used to characterize the key point distribution before the image is deformed, and the second key point distribution map is used to characterize the key point distribution after the image is deformed.

[0152] In some embodiments, the training module 705 includes: a second determination submodule, used to determine a first loss function based on the first facial expression representation and the second facial expression representation; a third determination submodule, used to determine a second loss function based on the first facial expression representation and the third facial expression representation; a fourth determination submodule, used to determine a third loss function based on the first key point distribution map and the second key point distribution map; a fifth determination submodule, used to determine a total loss function based on the first loss function, the second loss function and the third loss function; and a training submodule, used to train a facial expression representation model based on the total loss function.

[0153] In some embodiments, the second determination submodule is used to: perform spatial transformation on the first facial expression representation to obtain a transformed first facial expression representation; perform prediction based on the transformed first facial expression representation to obtain a predicted second facial expression representation; perform spatial transformation on the second facial expression representation to obtain a transformed second facial expression representation; perform prediction based on the transformed second facial expression representation to obtain a predicted first facial expression representation; determine a first loss function based on the predicted first facial expression representation, the transformed first facial expression representation, the predicted second facial expression representation, and the transformed second facial expression representation.

[0154] In some embodiments, the third determination submodule is used to: perform spatial transformation on the first facial expression representation to obtain a transformed first facial expression representation; perform prediction based on the transformed first facial expression representation to obtain a predicted second facial expression representation; perform spatial transformation on the third facial expression representation to obtain a transformed third facial expression representation; and determine a second loss function based on the predicted second facial expression representation and the transformed third facial expression representation.

[0155] In some embodiments, the fourth determination submodule is used to: compare the first key point distribution map and the second key point distribution map to obtain key points whose positions remain unchanged and key points whose positions change; determine the first weight corresponding to the key points whose positions remain unchanged, and the second weight corresponding to the key points whose positions change, the first weight being less than the second weight; determine the third loss function based on the key points whose positions remain unchanged, the first weight, the key points whose positions change, and the second weight.

[0156] Those skilled in the art should understand that the functions of each processing module in the facial expression representation model training device of the embodiment of the present disclosure can be understood with reference to the relevant description of the facial expression representation model training method described above. Each processing module in the facial expression representation model training device of the embodiment of the present disclosure can be implemented by an analog circuit that implements the function of the embodiment of the present disclosure, or it can be implemented by running software that executes the function of the embodiment of the present disclosure on an electronic device.

[0157] The facial expression representation model training device of the embodiment of the present disclosure can improve the accuracy of the facial expression representation model.

[0158] The present disclosure provides a facial expression recognition device, such as Figure 8 As shown, the facial expression recognition device may include: a fifth acquisition module 801 for acquiring an image to be recognized; a third input module 802 for inputting the image to be recognized into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model; and a recognition module 803 for inputting the facial expression representation into a classifier to obtain an expression classification result of the image to be recognized output by the classifier. The facial expression representation model is trained using the facial expression representation model training method described above.

[0159] In some embodiments, the third input module 802 includes: a control submodule, configured to input the image to be recognized into the first branch of the facial expression representation model to obtain the facial expression representation output by the first branch.

[0160] Those skilled in the art should understand that the functions of each processing module in the facial expression recognition device of the embodiment of the present disclosure can be understood with reference to the relevant description of the facial expression recognition method mentioned above. Each processing module in the facial expression recognition device of the embodiment of the present disclosure can be implemented by an analog circuit that implements the function of the embodiment of the present disclosure, or it can be implemented by running software that executes the function of the embodiment of the present disclosure on an electronic device.

[0161] The facial expression recognition device of the embodiment of the present disclosure adopts a facial expression representation model to obtain facial expression representation, which can improve the accuracy of facial expression recognition.

[0162] The embodiment of the present disclosure provides a scene diagram of a facial expression representation model training, such as Figure 9 shown.

[0163] As previously mentioned, the facial expression representation model training method provided in the embodiments of the present disclosure is applied to electronic devices. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0164] Specifically, the electronic device can perform the following operations: obtain data to be trained, which includes N original images; obtain a first image, a second image and a third image corresponding to each of the N original images, where the first image is an image obtained by performing a first random augmentation process on the original image; the second image is an image obtained by performing a second random augmentation process on the original image; and the third image is an image obtained by performing a random deformation process on the first image; input the first image into the first branch of the facial expression representation model, the second image into the second branch of the facial expression representation model, and the third image into the third branch of the facial expression representation model; obtain a first facial expression representation output by the first branch, a second facial expression representation output by the second branch, and a third facial expression representation output by the third branch; and train the facial expression representation model based on the first facial expression representation, the second facial expression representation and the third facial expression representation.

[0165] The training data, including N original images, can be obtained from an image data source. The image data source can be various forms of data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants (PDAs), servers, blade servers, mainframe computers, and other suitable computers. The image data source can also represent various forms of mobile devices, such as personal digital assistants (PDAs), cellular phones, smartphones, wearable devices, and other similar computing devices. Furthermore, the image data source and the user terminal can be the same device.

[0166] It should be understood that Figure 9 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 9 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0167] The embodiment of the present disclosure also provides a scene diagram of facial expression recognition, such as Figure 10 shown.

[0168] As previously mentioned, the facial expression recognition method provided by the embodiments of the present disclosure is applied to electronic devices. The term "electronic device" is intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The term "electronic device" may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0169] Specifically, the electronic device can perform the following operations: obtain an image to be recognized; input the image to be recognized into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model; input the facial expression representation into a classifier to obtain an expression classification result of the image to be recognized output by the classifier; wherein the facial expression representation model is obtained by training through the facial expression representation model training method described above.

[0170] The image to be recognized can be obtained from an image data source. The image data source can be various forms of data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The image data source can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. Furthermore, the image data source and the user terminal can be the same device.

[0171] It should be understood that Figure 10 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 10 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0172] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0173] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0174] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0175] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0176] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0177] The computing unit 1101 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the facial expression representation model training method / facial expression recognition method. For example, in some embodiments, the facial expression representation model training method / facial expression recognition method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the facial expression representation model training method / facial expression recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the facial expression representation model training method / facial expression recognition method by any other appropriate means (e.g., by means of firmware).

[0178] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0179] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0180] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory RAM, a read-only memory ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0182] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0183] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0184] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0185] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training a facial expression representation model, comprising: Acquire data to be trained, where the data to be trained includes N original images; Obtaining a first image, a second image, and a third image corresponding to each of the N original images, wherein the first image is an image obtained by performing a first random augmentation process on the original image; the second image is an image obtained by performing a second random augmentation process on the original image; and the third image is an image obtained by performing a random deformation process on the first image; Inputting the first image into a first branch of a facial expression representation model, inputting the second image into a second branch of the facial expression representation model, and inputting the third image into a third branch of the facial expression representation model; Obtaining a first facial expression representation output by the first branch, a second facial expression representation output by the second branch, and a third facial expression representation output by the third branch; The facial expression representation model is trained based on the first facial expression representation, the second facial expression representation, and the third facial expression representation.

2. The method according to claim 1, wherein Acquiring a third image corresponding to each of the N original images includes: Determine a circular area; Pixel shifting is performed on the first image corresponding to each of the N original images in the circular area to obtain the third image corresponding to each of the N original images.

3. The method according to claim 2, wherein: The performing pixel shifting on the first image corresponding to each of the N original images in the circular area to obtain the third image corresponding to each of the N original images includes: The first images corresponding to the N original images are moved K times in the circular area to obtain the third images corresponding to the N original images, where K is an integer not less than 2; wherein the Kth movement is a pixel movement based on the result of the K-1th movement.

4. The method according to claim 1, wherein Obtaining a third facial expression representation output by the third branch includes: The third image corresponding to each of the N original images is input into the backbone network of the third branch for feature extraction to obtain the third facial expression representation corresponding to each of the N original images.

5. The method according to claim 1, further comprising: Inputting the first facial expression representation output by the first branch and the third facial expression representation output by the third branch into the fourth branch of the facial expression representation model; Obtain a first key point distribution map obtained by the fourth branch based on the first facial expression representation prediction, and a second key point distribution map obtained by the fourth branch based on the third facial expression representation prediction; wherein the first key point distribution map is used to characterize the key point distribution before the image is deformed, and the second key point distribution map is used to characterize the key point distribution after the image is deformed.

6. The method according to claim 5, wherein: The training of the facial expression representation model based on the first facial expression representation, the second facial expression representation, and the third facial expression representation includes: determining a first loss function based on the first facial expression representation and the second facial expression representation; determining a second loss function based on the first facial expression representation and the third facial expression representation; determining a third loss function based on the first key point distribution map and the second key point distribution map; Determining a total loss function based on the first loss function, the second loss function, and the third loss function; Based on the total loss function, the facial expression representation model is trained.

7. The method according to claim 6, wherein: The determining a first loss function based on the first facial expression representation and the second facial expression representation includes: performing spatial transformation on the first facial expression representation to obtain a transformed first facial expression representation; Performing a prediction based on the converted first facial expression representation to obtain a predicted second facial expression representation; performing spatial transformation on the second facial expression representation to obtain a transformed second facial expression representation; Performing prediction based on the converted second facial expression representation to obtain a predicted first facial expression representation; The first loss function is determined based on the predicted first facial expression representation, the converted first facial expression representation, the predicted second facial expression representation, and the converted second facial expression representation.

8. The method according to claim 6, wherein: The determining of a second loss function based on the first facial expression representation and the third facial expression representation includes: performing spatial transformation on the first facial expression representation to obtain a transformed first facial expression representation; Performing a prediction based on the converted first facial expression representation to obtain a predicted second facial expression representation; performing spatial transformation on the third facial expression representation to obtain a transformed third facial expression representation; The second loss function is determined based on the predicted second facial expression representation and the converted third facial expression representation.

9. The method according to claim 6, wherein: The determining a third loss function based on the first key point distribution map and the second key point distribution map includes: Comparing the first key point distribution map with the second key point distribution map to obtain key points with unchanged positions and key points with changed positions; Determine a first weight corresponding to the key point whose position remains unchanged, and a second weight corresponding to the key point whose position changes, wherein the first weight is smaller than the second weight; The third loss function is determined based on the key points that do not change in position, the first weights, the key points that change in position, and the second weights.

10. A method for recognizing facial expressions, comprising: Obtain the image to be recognized; Inputting the image to be recognized into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model; The facial expression representation is input into a classifier to obtain an expression classification result of the image to be recognized output by the classifier; wherein the facial expression representation model is obtained by training using the method described in any one of claims 1 to 9.

11. The method according to claim 10, wherein: The step of inputting the image to be recognized into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model comprises: The image to be recognized is input into the first branch of the facial expression representation model to obtain the facial expression representation output by the first branch.

12. A facial expression model training device, comprising: A first acquisition module is used to acquire data to be trained, where the data to be trained includes N original images; a second acquisition module, configured to acquire a first image, a second image, and a third image corresponding to each of the N original images, wherein the first image is an image obtained by performing a first random augmentation process on the original image; the second image is an image obtained by performing a second random augmentation process on the original image; and the third image is an image obtained by performing a random deformation process on the first image; a first input module, configured to input the first image into a first branch of a facial expression representation model, the second image into a second branch of the facial expression representation model, and the third image into a third branch of the facial expression representation model; a third acquisition module, configured to acquire the first facial expression representation output by the first branch, the second facial expression representation output by the second branch, and the third facial expression representation output by the third branch; A training module is used to train the facial expression representation model based on the first facial expression representation, the second facial expression representation and the third facial expression representation.

13. The device according to claim 12, wherein The second acquisition module includes: A first determining submodule is used to determine a circular area; The pixel shift submodule is configured to perform pixel shift on the first image corresponding to each of the N original images in the circular area to obtain the third image corresponding to each of the N original images.

14. The device according to claim 13, wherein The pixel shift submodule is used to: The first images corresponding to the N original images are moved K times in the circular area to obtain the third images corresponding to the N original images, where K is an integer not less than 2; wherein the Kth movement is a pixel movement based on the result of the K-1th movement.

15. The device according to claim 12, wherein The third acquisition module includes: The feature extraction submodule is used to input the third image corresponding to each of the N original images into the backbone network of the third branch for feature extraction, so as to obtain the third facial expression representation corresponding to each of the N original images.

16. The apparatus according to claim 12, further comprising: a second input module, configured to input the first facial expression representation output by the first branch and the third facial expression representation output by the third branch into a fourth branch of the facial expression representation model; A fourth acquisition module is used to obtain a first key point distribution map obtained by the fourth branch based on the first facial expression representation prediction, and a second key point distribution map obtained by the fourth branch based on the third facial expression representation prediction; wherein the first key point distribution map is used to characterize the key point distribution before the image is deformed, and the second key point distribution map is used to characterize the key point distribution after the image is deformed.

17. The device according to claim 16, wherein The training module includes: A second determining submodule, configured to determine a first loss function based on the first facial expression representation and the second facial expression representation; A third determining submodule, configured to determine a second loss function based on the first facial expression representation and the third facial expression representation; a fourth determining submodule, configured to determine a third loss function based on the first key point distribution map and the second key point distribution map; a fifth determining submodule, configured to determine a total loss function based on the first loss function, the second loss function, and the third loss function; A training submodule is used to train the facial expression representation model based on the total loss function.

18. The device according to claim 17, wherein The second determining submodule is configured to: performing spatial transformation on the first facial expression representation to obtain a transformed first facial expression representation; Performing a prediction based on the converted first facial expression representation to obtain a predicted second facial expression representation; performing spatial transformation on the second facial expression representation to obtain a transformed second facial expression representation; Performing prediction based on the converted second facial expression representation to obtain a predicted first facial expression representation; The first loss function is determined based on the predicted first facial expression representation, the converted first facial expression representation, the predicted second facial expression representation, and the converted second facial expression representation.

19. The device according to claim 17, wherein The third determining submodule is configured to: performing spatial transformation on the first facial expression representation to obtain a transformed first facial expression representation; Performing a prediction based on the converted first facial expression representation to obtain a predicted second facial expression representation; performing spatial transformation on the third facial expression representation to obtain a transformed third facial expression representation; The second loss function is determined based on the predicted second facial expression representation and the converted third facial expression representation.

20. The apparatus according to claim 17, wherein The fourth determining submodule is configured to: Comparing the first key point distribution map with the second key point distribution map to obtain key points with unchanged positions and key points with changed positions; Determine a first weight corresponding to the key point whose position remains unchanged, and a second weight corresponding to the key point whose position changes, wherein the first weight is smaller than the second weight; The third loss function is determined based on the key points that do not change in position, the first weights, the key points that change in position, and the second weights.

21. A facial expression recognition device, comprising: A fifth acquisition module, configured to acquire an image to be identified; A third input module is used to input the image to be recognized into a facial expression representation model to obtain a facial expression representation output by the facial expression representation model; A recognition module, configured to input the facial expression representation into a classifier to obtain an expression classification result of the image to be recognized output by the classifier; wherein the facial expression representation model is obtained by training using the method described in any one of claims 1 to 9.

22. The device according to claim 21, wherein The third input module includes: The control submodule is configured to input the image to be recognized into the first branch of the facial expression representation model to obtain the facial expression representation output by the first branch.

23. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

25. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images