A method, apparatus, device, and medium for dividing facial motion units into regions.

Abstract

This invention relates to the field of image processing technology, and more particularly to a method, apparatus, device, and medium for region segmentation of facial action units. The application first acquires a region segmentation model and a target face image to be segmented; simultaneously, it acquires a manually labeled first face image and an unlabeled training set of expression images. Based on the labeling result of the first face image, it performs region labeling on the images in the expression image training set to obtain the target labeling result for the corresponding training image; then, it uses the region segmentation model to predict the region segmentation of the images in the expression image training set to obtain the predicted labeling result for the corresponding training image; furthermore, based on the target labeling result and the predicted labeling result, it obtains a trained region segmentation model, and uses the trained region segmentation model to segment the target face image to obtain the region segmentation result; finally, with minimal manual annotation, the accuracy of region segmentation for facial action units is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and medium for dividing facial motion units into regions. Background Technology

[0002] Facial Action Unit Detection (FAUD), as a pre-processing technique for facial expression analysis, aims to detect the location and state of different Action Units (AUs). Different AUs can be understood as different facial muscle regions, thus indirectly reflecting changes in facial expressions. Currently, most FAUD methods focus on deep learning frameworks and can be broadly categorized into three types: pre-trained FAUD methods, attention-based FAUD methods, and multi-task-based FAUD methods.

[0003] Among them, pre-trained facial action unit detection methods mainly utilize transfer learning techniques to first transfer other facial feature knowledge (e.g., face verification and expression recognition) to the FAUD task, and then fine-tune the parameters on the AU dataset to update the parameters of the last few layers, thereby reducing the model's dependence on AU annotations. However, due to the certain cross-domain differences between the pre-training dataset and the fine-tuning dataset, this method usually requires a lot of manual parameter tuning or search strategies to find the optimal training configuration, thus generating more training costs.

[0004] Attention-based facial action unit detection methods primarily remove irrelevant content (e.g., background, neck, hair, etc.) from images by constructing facial masks, allowing the model to focus more on regions of interest and enhancing its ability to perceive facial structures. However, since the shapes formed by facial keypoints are not strictly equivalent to facial action unit regions, the learned facial keypoint features may not accurately reflect changes in facial action units.

[0005] Multi-task-based facial action unit detection methods primarily utilize multi-head mechanisms to enhance the diversity of facial representations, thereby reducing the risk of model overfitting. However, since multi-task learning performance typically depends on the correlation between different tasks and the number of training samples, model performance can easily degrade when the differences between multi-tasks are too large or the number of training samples is insufficient.

[0006] The three facial action unit detection methods mentioned above are based on discrete label annotation. They aim to improve the facial action recognition performance of the model from the perspectives of overcoming annotation dependence, overcoming background interference, and improving feature diversity. However, the discrete label annotation method is difficult to reflect real and subtle facial expression changes, which can lead to a mismatch between the annotation information and the expression details. For example, there are domain shift problems caused by the difference between the pre-training dataset and the fine-tuning dataset, region shift problems caused by the difference between facial key points and facial action units, and factor shift problems caused by the difference between AU expression features and identity factors. Therefore, how to improve the accuracy of the annotation information of facial action units has become an urgent problem to be solved. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a method, apparatus, device and medium for dividing the region of facial motion units, in order to solve the problem of low accuracy of annotation information of facial motion units.

[0008] In a first aspect, embodiments of the present invention provide a method for region division of a facial action unit, the region division method comprising:

[0009] Obtain the region segmentation model and the target face image to be segmented;

[0010] Obtain a training set of manually calibrated first face images and uncalibrated expression images. Based on the calibration result of the first face image, perform region calibration on the images in the expression image training set to obtain the target calibration result of the corresponding training image.

[0011] The region segmentation model is used to perform region segmentation prediction on the images in the expression image training set to obtain the prediction and calibration results of the corresponding training images;

[0012] Based on the target calibration result and the prediction calibration result, adjust the parameters of the region partitioning model until the preset conditions are met to obtain the trained region partitioning model;

[0013] The trained region segmentation model is used to segment the target face image into regions, resulting in the region segmentation result corresponding to the target face image.

[0014] Secondly, embodiments of the present invention provide a region division device for a facial motion unit, the region division device comprising:

[0015] The information acquisition module is used to acquire the region segmentation model and the target face image to be segmented;

[0016] The result calibration module is used to acquire a manually calibrated first face image and an uncalibrated expression image training set, and to perform region calibration on the images in the expression image training set based on the calibration result of the first face image to obtain the target calibration result of the corresponding training image.

[0017] The result prediction module is used to perform region segmentation prediction on the images in the expression image training set using the region segmentation model, and obtain the prediction and calibration results of the corresponding training images.

[0018] The parameter adjustment module is used to adjust the parameters of the region partitioning model according to the target calibration result and the prediction calibration result until the preset conditions are met, so as to obtain the trained region partitioning model.

[0019] The region segmentation module is used to segment the target face image into regions using the trained region segmentation model, thereby obtaining the region segmentation result corresponding to the target face image.

[0020] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the region partitioning method as described in the first aspect.

[0021] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the region partitioning method as described in the first aspect.

[0022] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0023] This invention acquires a region segmentation model and a target face image to be segmented; it acquires a manually labeled first face image and an unlabeled training set of facial expression images; based on the labeling result of the first face image, it performs region labeling on the images in the facial expression image training set to obtain the target labeling result for the corresponding training image; it uses the region segmentation model to predict the region segmentation of the images in the facial expression image training set to obtain the predicted labeling result for the corresponding training image; based on the target labeling result and the predicted labeling result, it adjusts the parameters of the region segmentation model until a preset condition is met to obtain a trained region segmentation model; it uses the trained region segmentation model to segment the target face image to obtain the region segmentation result for the corresponding target face image. Under conditions of minimal manual annotation, it uses the manually labeled first face image to obtain the target labeling result for each training image in the unlabeled facial expression image training set; based on the target labeling result and the predicted labeling result output by the region segmentation model, it adjusts the parameters of the region segmentation model, thereby improving the performance of the region segmentation model and thus enhancing the accuracy of facial action unit region segmentation. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an application environment for a facial motion unit region division method provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart illustrating a method for dividing a facial action unit into regions, as provided in Embodiment 1 of the present invention.

[0027] Figure 3 This is a schematic diagram of a two-dimensional texture coordinate transformation provided in Embodiment 1 of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a facial motion unit region division device provided in Embodiment 2 of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0031] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0032] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0034] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0036] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0037] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0038] The method for dividing facial motion units according to Embodiment 1 of this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0039] See Figure 2 This is a flowchart illustrating a method for dividing a facial action unit into regions according to Embodiment 1 of the present invention. The above-described region division method can be applied to... Figure 1 The client-side computer connects to the server to obtain relevant backend data, such as the target face image to be segmented and the regions used for subsequent facial motion unit segmentation of the target face image. Additionally, the server can provide the client with a model trained using a training set of manually labeled first face images and unlabeled expression images, allowing the client to call the model for computation. Figure 2 As shown, this region division method may include the following steps:

[0040] Step S201: Obtain the region segmentation model and the target face image to be segmented.

[0041] The region segmentation model refers to a neural network used to segment images into regions. The target face image to be segmented refers to the original face image that has not been segmented by facial action units. Therefore, the region segmentation model and the target face image to be segmented are obtained so that the target face image can be segmented into regions by facial action units in the subsequent process.

[0042] Step S202: Obtain a training set of manually calibrated first face image and uncalibrated expression images. Based on the calibration result of the first face image, perform region calibration on the images in the expression image training set to obtain the target calibration result of the corresponding training image.

[0043] In this context, the manually labeled first face image refers to the first face image whose facial action units have been manually labeled. The unlabeled expression image training set refers to the training set consisting of at least one expression image without labeled facial action units, i.e., an unlabeled expression image training set. Training images refer to the expression images in the unlabeled expression image training set. The labeling result refers to the annotation information corresponding to the facial action units, i.e., the labeled regions corresponding to the facial action units in the image. Therefore, by obtaining the manually labeled first face image and the unlabeled expression image training set, and performing region labeling on each training image in the expression image training set based on the labeling result of the first face image, the target labeling result of the corresponding training image is obtained. By manually labeling a small number of images, the automatic generation of the labeling for any image is achieved.

[0044] Optionally, based on the calibration results of the first face image, region calibration is performed on the images in the expression image training set to obtain the target calibration results for the corresponding training images, including:

[0045] Based on the calibration results of the first face image, determine the three-dimensional region mask corresponding to each calibrated region in the first face image;

[0046] Extract the three-dimensional vertex position information of the first face image, and use a preset two-dimensional texture transformation model to tile the three-dimensional vertex position information to obtain the first tiling result;

[0047] Based on the mapping relationship between the three-dimensional vertex position information and each three-dimensional region mask, each three-dimensional region mask is aligned in the first tiling result to obtain the aligned two-dimensional region mask.

[0048] For any training image, extract the three-dimensional vertex position information of the training image, and use a preset two-dimensional texture transformation model to tile the three-dimensional vertex position information of the training image to obtain a second tiling result.

[0049] Based on the mapping relationship between the three-dimensional vertex position information and each three-dimensional region mask, the target three-dimensional region mask corresponding to the aligned two-dimensional region mask in the second tiling result is obtained. Using the target three-dimensional region mask, the initial calibration result of the training image is generated.

[0050] The preset 2D texture transformation model refers to a 2D texture coordinate (U-VEEZ, UV) transformation component, used to determine how to place a texture image on a 3D model surface, thereby achieving coordinate transformation from image space to UV space. Therefore, for facial motion units in the first face image, firstly, the regions corresponding to the facial motion units in the first face image are manually calibrated using a manual calibration component to obtain the calibration result of the first face image. Then, based on the calibration result, the 3D region mask corresponding to each calibrated region in the first face image in image space is obtained. Next, the image vertices of the first face image are estimated using a 3D vertex estimator component to obtain a 3D position map. It is worth noting that the 3D vertex estimator component has a U-shaped structure, capable of predicting the 3D geometric structure of the image, thus establishing a mapping relationship from image pixel coordinates to UV coordinates. Based on the 3D position map, the 3D vertex position information of the first face image is extracted, and a preset... The UV transformation model tiled the 3D vertex position information to obtain the first tiling result. Then, based on the mapping relationship between the 3D vertex position information and each 3D region mask, each 3D region mask was aligned in the first tiling result to obtain an aligned 2D region mask. The aligned 2D region mask corresponds exactly to each facial action region in the first face image. The 3D region mask corresponding to the first face image was mapped to a relatively unchanging UV space, so that all pixels of the first face image would be tiled at any fixed position in the UV space, thus obtaining the aligned 2D region mask. This makes the labeling of the regions corresponding to the facial action units in each training image in the subsequent expression image training set more accurate.

[0051] In one feasible embodiment, reference is made to Figure 3 This is a schematic diagram of a two-dimensional texture coordinate transformation provided in an embodiment of the present invention. It is assumed that the image space contains a three-dimensional region mask, the UV space contains an aligned two-dimensional region mask, and the index space contains a 3D position map. Simultaneously, it defines: [ζ,ξ]:[α1,α2,…,α...] d [] represents the element relationships in different spaces, where [ζ,ξ] represents the position coordinates of the element, and [α1,α2,…,α] represents the position coordinates of the element. d [] represents the element value, and d represents the number of channels. During the forward process of UV transformation, all elements in the 3D position map are traversed first. i ,v i ]:[x i ,y i Then, in the 3D region mask, the region located in [x] i ,y i The element value [r,g,b] of [r,g,b] is extracted and assigned to the aligned 2D region mask [u]. i ,v iAt position [r, g, b], through iterative loops, the 3D region mask is remapped to an aligned 2D region mask.

[0052] Similarly, for images in the facial expression image training set, a preset two-dimensional texture transformation model is used to achieve coordinate inversion from UV space to image space. Therefore, for any training image in the facial expression image training set, the image vertices of the training image are estimated using a 3D vertex estimator component to obtain a three-dimensional position map. Based on the three-dimensional position map, the three-dimensional vertex position information of the training image is extracted. The three-dimensional vertex position information of the training image is tiled using a preset UV transformation model to obtain a second tiling result. Based on the mapping relationship between the three-dimensional vertex position information and each three-dimensional region mask, the target three-dimensional region mask corresponding to the aligned two-dimensional region mask in the second tiling result is obtained. Using the target three-dimensional region mask, the initial calibration result of the training image is generated, which is to say, the region division of facial action units in the training image is completed. By calibrating facial action units on a small number of first face images, automatic calibration of any face image is achieved based on the calibration results, reducing the cost of manual annotation.

[0053] Optionally, after generating the initial calibration results for the training images, the following steps are included:

[0054] Using preset ellipse parameters, each calibration region in the initial calibration result is approximated by an ellipse, and the initial calibration result after ellipse approximation is confirmed to be the target calibration result.

[0055] Ellipse approximation refers to adjusting an irregular shape into an elliptical shape. Therefore, ellipse approximation is performed on each calibration region in the initial calibration result using preset ellipse parameters. Specifically, it is assumed that the region contour of any calibration region is... Among them, (x i ,y i ) represents a point on the region's contour, and T represents the number of contour points; typically, the standard equation of an ellipse is derived from the ellipse's center (c x ,c y The major and minor axes are (a, b), and the rotation angle θ determines the ellipse equation. The ellipse parameters are obtained by fitting the data to the ellipse equation based on the positions of all contour points on the calibration region's contour. Then, an ellipse approximation is performed on the calibration region based on these parameters to obtain the corresponding elliptical region E. reg ={c x ,c ySimilarly, each calibration region in the initial calibration result is approximated by an ellipse to obtain the elliptical region corresponding to each calibration region. Then, the elliptical region is confirmed as the target calibration result. That is, the initial calibration result after elliptical approximation is the target calibration result. The approximate contours of different calibration regions are approximated by ellipses to convert irregular calibration regions into rotatable elliptical regions, thereby reducing the complexity of region fitting.

[0056] Step S203: Use the region segmentation model to perform region segmentation prediction on the images in the expression image training set to obtain the prediction and calibration results of the corresponding training images.

[0057] The prediction and calibration result refers to the annotation information corresponding to the pre-inferred facial action unit, that is, the region corresponding to the pre-inferred facial action unit in the image. Therefore, the region segmentation model is used to perform region segmentation prediction on each training image in the expression image training set to obtain the prediction and calibration result of the corresponding training image.

[0058] Optionally, the region segmentation model includes a downsampling module, a residual module, and a region regression module. The region segmentation model is used to predict regions in the image training set of facial expression images, obtaining the prediction and labeling results for the corresponding training images, including:

[0059] For any training image, the downsampling module is used to extract features from the training image to obtain the first common feature;

[0060] The residual module is used to perform residual calculation on the first common feature to obtain the first residual tensor;

[0061] The region regression module is used to perform region division prediction on the first residual tensor, and the prediction calibration results are obtained.

[0062] The downsampling module is used to extract image features, the residual module contains a residual block to measure the difference between the predicted and observed values ​​to enhance the expressive power of the region segmentation model, and the region regression module consists of two convolutional layers and one fully connected layer to predict the regional location of facial action units. Therefore, for any training image in the expression image training set, the downsampling module is used to extract features from the training image to obtain the first common feature. Then, the first common feature is split into two parts according to the channel, namely common feature one and common feature two. The residual module is then used to perform residual calculation on common feature one to obtain the first residual tensor. Finally, the region regression module is used to perform region segmentation prediction on the first residual tensor to obtain the prediction calibration result of the training image.

[0063] Step S204: Adjust the parameters of the region partitioning model according to the target calibration results and the prediction calibration results until the preset conditions are met, and obtain the trained region partitioning model.

[0064] The trained region partitioning model refers to a region partitioning model that has been trained. In other words, it is a region partitioning model that has been trained by adjusting the parameters of the region partitioning model through the training set during the training process. Therefore, the parameters of the region partitioning model are adjusted according to the target calibration results and the prediction calibration results until the preset conditions are met, and the trained region partitioning model is obtained.

[0065] Optionally, the region partitioning module includes an identity decoupling module and a state classification module. Before adjusting the parameters of the region partitioning model based on the target calibration results and prediction calibration results, it includes:

[0066] Get B labeled facial expression images, where B is a positive integer;

[0067] For any facial expression image, the downsampling module is used to extract features from the facial expression image to obtain the second common feature, and the residual module is used to calculate the residual of the second common feature to obtain the second residual tensor.

[0068] The identity decoupling module is used to perform region detection on the second common feature to generate the expected feature vector of the expression image. The second residual tensor and the expected feature vector are then input into the state classification module for region reshaping to obtain the activation feature vector of the expression image.

[0069] Obtain the activation feature vector of each facial expression image, and count all region categories based on the activation feature vector of each facial expression image;

[0070] For any region category, identify facial expression images that contain the region category as positive samples and identify facial expression images that do not contain the region category as negative samples, and obtain positive and negative samples for all region categories.

[0071] Here, the B labeled expression images refer to B expression images with known facial action units. The identity decoupling module is used to detect the regions corresponding to the facial action units. The state classification module consists of two convolutional layers and one fully connected layer, used to reconstruct the regions of the facial action units. Therefore, for any expression image, the downsampling module is used to extract features from the expression image to obtain the second common feature. Then, the second common feature is split into two parts according to the channel, namely the first common feature and the second common feature. Then, the residual module is used to calculate the residual of the first common feature of the second common feature to obtain the second residual tensor. The second common feature is input into the identity decoupling module for channel dimensionality reduction to obtain the region detection result. Based on the detected region, the expected feature vector of the expression image is generated. Finally, the generated feature vector is processed... The expected feature vector is further reshaped by inputting the second residual tensor and the expected feature vector into the state classification module for dimension swapping and dimension stretching operations, reshaping the expected feature vector into an activation feature vector. Here, different elements in the activation feature vector represent the probability that the corresponding region is activated. For example, there are a total of 5 facial action units, namely unit 1, unit 2, unit 3, unit 4 and unit 5. The feature activation vector of the corresponding expression image is [1, 0, 1, 0, 1]. The first 1 indicates that the expression image contains the region corresponding to unit 1, the first 0 indicates that the expression image does not contain the region corresponding to unit 2, the second 1 indicates that the expression image contains the region corresponding to unit 3, the second 0 indicates that the expression image does not contain the region corresponding to unit 4, and the third 1 indicates that the expression image contains the region corresponding to unit 5.

[0072] Similarly, the activation feature vector of each facial expression image is obtained. Based on the activation feature vector of each facial expression image, all region categories, that is, the categories of facial action units, are counted. For any region category, facial expression images containing the region category are identified as positive samples, and facial expression images not containing the region category are identified as negative samples. Positive and negative samples under all region categories are obtained to form positive and negative sample pairs for the facial expression images, which is convenient for subsequent comparative learning. For example, there are a total of 5 facial action units, namely unit 1, unit 2, and unit 3. The activation feature vector of facial expression image A is [1, 0, 1], the activation feature vector of facial expression image B is [0, 1, 0], and the activation feature vector of facial expression image C is [1, 0, 0]. For unit 1, it can be seen that facial expression images A and B are positive samples, and facial expression image C is a negative sample. For unit 2, it can be seen that facial expression images A and C are negative samples, and facial expression image B is a positive sample. Similarly, for unit 3, it can be seen that facial expression images B and C are negative samples, and facial expression image A is a positive sample.

[0073] Optionally, the parameters of the region partitioning model can be adjusted based on the target calibration results and the prediction calibration results, including:

[0074] Calculate the regional regression loss based on the regional differences between the target calibration results and the predicted calibration results;

[0075] The first residual tensor is constrained using the initial calibration results to obtain the labeled mask loss;

[0076] Obtain the actual feature vector corresponding to the label of each facial expression image, and calculate the activation loss based on the feature difference between the actual feature vector and the activation feature vector;

[0077] By comparing and learning positive and negative samples under all region categories, we obtain the identity decoupling loss.

[0078] The parameters of the region partitioning model are adjusted based on the region regression loss, label mask loss, activation loss, and identity decoupling loss.

[0079] The loss functions used in the training process of the region segmentation model include region regression loss, label mask loss, activation loss, and identity decoupling loss. For the region regression loss, each labeled region in the target labeling result is an elliptical region, and each elliptical region corresponds to 5 elliptical parameters, namely {c x ,c y If we define a, b, θ, then we can form a parameter set by combining the five elliptical parameters of each elliptical region to represent the positional information of the corresponding calibration region. Then, we can form a target position vector by combining the parameter sets corresponding to all calibration regions in the target calibration result. Similarly, we can obtain the set of five elliptical parameters corresponding to each calibration region in the prediction calibration result, and form a prediction position vector by combining the set of all calibration regions in the prediction calibration result. We can calculate the L2 norm loss based on the difference between the target position vector and the prediction position vector corresponding to the training image, and confirm that the L2 norm loss is the region regression loss. By evaluating the regional difference between the predicted region and the real region, we can guide the training of the region segmentation model.

[0080] For the labeling mask loss, the target calibration result is obtained by approximating the initial calibration result with an ellipse. The predicted calibration result is obtained by region division prediction based on the first residual tensor. The feature vector of the calibration region corresponding to the initial calibration result is obtained, and the L2 norm loss between the feature vector and the first residual tensor is calculated. The L2 norm loss is confirmed as the labeling mask loss. By constraining the first residual tensor, it is made as close as possible to the initial calibration result, thereby strengthening the region regression loss.

[0081] For activation loss, the corresponding actual feature vector is obtained according to the label of each expression image. It can generally be represented as a vector in the form of [1,0,…,1]. The L2 norm loss between the actual feature vector and the activation feature vector is calculated. The L2 norm loss is confirmed as the activation loss. By transforming the multi-label classification problem into a vector regression problem, the activation loss is used to measure the difference between the predicted label and the true label.

[0082] To achieve identity decoupling, from the perspective of contrastive learning, the identity decoupling loss is obtained by contrastive learning of positive and negative samples under all region categories. This ensures that the same facial action unit has a small Euclidean distance in face images of people with different identities, thereby reducing the influence of identity factors on the representation of the same facial action unit. As a result, the learned facial action unit has a certain degree of identity invariance, that is, the representation of the same type of facial action unit is consistent in face images of different people.

[0083] This method integrates region regression loss, label masking loss, activation loss, and identity decoupling loss, and adjusts the parameters of the region partitioning model based on the integration result to achieve unified end-to-end training. Specifically, it calculates the sum of the region regression loss, label masking loss, activation loss, and identity decoupling loss, uses this sum as the supervision loss of the region partitioning model, and then uses gradient descent to correct the parameters of the region partitioning model in reverse until the supervision loss converges, resulting in a trained region partitioning model.

[0084] Optionally, comparative learning is performed on positive and negative samples across all region categories to obtain the identity decoupling loss, including:

[0085] For any region category, obtain the anchor features and positive sample features of each positive sample under the region category, as well as the negative sample features of each negative sample;

[0086] Based on the feature difference between the anchor feature and the positive sample feature corresponding to each positive sample, a first difference result is obtained; based on the feature difference between the positive sample feature of each positive sample and the negative sample feature of each negative sample, a second difference result is obtained.

[0087] Obtain the first and second difference results for all region categories, and calculate the identity decoupling loss based on all first and second difference results.

[0088] By contrastive learning, facial action units are freed from identity interference, enabling them to possess identity invariance. In other words, for any category of facial action unit, the differences in the state represented by that category on different people's faces are reduced.

[0089] Specifically, for any region category, the L2 norm loss between the anchor feature and the positive sample feature corresponding to each positive sample is calculated. The L2 norm loss of all positive samples is used to form the first difference result. At the same time, the L2 norm loss between each positive sample and each negative sample corresponding to the positive sample feature and the negative sample feature is calculated. The L2 norm loss between all positive samples and each negative sample is used to form the second difference result. Similarly, the first difference result and the second difference result under all region categories are obtained. All the first difference results and all the second difference results are integrated to obtain the identity decoupling loss. Through contrastive learning, the common features between similar instances are learned, and the differences between different instances are distinguished, thereby reducing the influence of different identities on the representation of the same facial action unit.

[0090] In one optional embodiment, the formula for calculating the identity decoupling loss is:

[0091]

[0092] Among them, L id For identity decoupling loss, N au B represents the total number of region categories. kr B represents the number of positive samples in the k-th region category. kf Let be the number of negative samples in the k-th region category. Let be the anchor feature of the t-th positive sample under the k-th region category. Let be the positive sample features of the t-th positive sample under the k-th region category. Let be the negative sample feature of the s-th negative sample under the k-th region category, ‖*‖2 is the L2 norm, max is the maximum value function, and ∈ is a hyperparameter used to control the minimum boundary between positive and negative samples.

[0093] Step S205: Use the trained region segmentation model to segment the target face image into regions, and obtain the region segmentation result of the corresponding target face image.

[0094] Among them, the trained region segmentation model already has a powerful region segmentation capability, and it can accurately segment regions for new face images. Therefore, using the trained region segmentation model to segment the target face image results in a very high accuracy rate.

[0095] This invention provides an embodiment of the method for obtaining a constructed region segmentation model and a target face image to be segmented. It acquires a manually labeled first face image and an unlabeled set of facial expression images. Based on the labeling results of the first face image, it performs region labeling on each training image in the facial expression image training set, obtaining the target labeling results for the corresponding training images. The region segmentation model is then used to predict the region segmentation of each training image in the facial expression image training set, obtaining the predicted labeling results for the corresponding training images. Based on the target labeling results and the predicted labeling results, the parameters of the region segmentation model are adjusted until preset conditions are met, resulting in a trained region segmentation model. The trained region segmentation model is then used to segment the target face image, obtaining the region segmentation results for the corresponding target face image. With minimal manual annotation, the target labeling results for each training image in the unlabeled facial expression image training set are obtained using the manually labeled first face image. Based on the target labeling results and the predicted labeling results output by the region segmentation model, the parameters of the region segmentation model are adjusted, improving the performance of the region segmentation model and thus enhancing the accuracy of facial action unit region segmentation.

[0096] Corresponding to the region division method in the above embodiment, Figure 4 The diagram shows a structural block diagram of a facial motion unit region division device according to Embodiment 2 of the present invention. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0097] See Figure 4 The area division device includes:

[0098] The information acquisition module 41 is used to acquire the region segmentation model and the target face image to be segmented.

[0099] The result calibration module 42 is used to acquire a manually calibrated first face image and an uncalibrated expression image training set, and to perform region calibration on the images in the expression image training set based on the calibration result of the first face image to obtain the target calibration result of the corresponding training image.

[0100] The result prediction module 43 is used to perform region segmentation prediction on the images in the expression image training set using a region segmentation model, and obtain the prediction and calibration results of the corresponding training images.

[0101] The parameter adjustment module 44 is used to adjust the parameters of the region partitioning model according to the target calibration results and the prediction calibration results until the preset conditions are met, so as to obtain the trained region partitioning model.

[0102] The region segmentation module 45 is used to segment the target face image into regions using a trained region segmentation model, and obtain the region segmentation result of the corresponding target face image.

[0103] Optionally, the result calibration module 42 includes:

[0104] The mask acquisition unit is used to determine the three-dimensional region mask corresponding to each labeled region in the first face image based on the labeling result of the first face image.

[0105] The first tiling unit is used to extract the three-dimensional vertex position information of the first face image, and to perform tiling processing on the three-dimensional vertex position information using a preset two-dimensional texture transformation model to obtain the first tiling result.

[0106] The mask alignment unit is used to align each three-dimensional region mask in the first tiling result according to the mapping relationship between the three-dimensional vertex position information and each three-dimensional region mask, so as to obtain the aligned two-dimensional region mask.

[0107] The second tiling unit is used to extract the three-dimensional vertex position information of any training image, and to tile the three-dimensional vertex position information of the training image using a preset two-dimensional texture transformation model to obtain the second tiling result.

[0108] The initial calibration unit is used to obtain the target 3D region mask corresponding to the aligned 2D region mask in the second tiling result based on the mapping relationship between the 3D vertex position information and each 3D region mask, and to generate the initial calibration result of the training image using the target 3D region mask.

[0109] Optionally, the result calibration module 42 includes:

[0110] The ellipse fitting unit is used to approximate each calibration region in the initial calibration result with an ellipse using preset ellipse parameters after generating the initial calibration result of the training image, and to confirm that the initial calibration result after ellipse approximation is the target calibration result.

[0111] Optionally, if the region partitioning model includes a downsampling module, a residual module, and a region regression module, then the result prediction module 43 includes:

[0112] The feature extraction unit is used to extract features from any training image using the downsampling module to obtain the first common feature;

[0113] The residual calculation unit is used to perform residual calculation on the first common feature using the residual module to obtain the first residual tensor;

[0114] The region prediction unit is used to perform region division prediction on the first residual tensor using the region regression module to obtain the prediction calibration result.

[0115] Optionally, if the region division module includes an identity decoupling module and a state classification module, then the parameter adjustment module 44 includes:

[0116] The labeled image acquisition unit is used to acquire B labeled facial expression images before adjusting the parameters of the region segmentation model based on the target calibration result and the prediction calibration result, where B is an integer greater than zero.

[0117] The residual tensor acquisition unit is used to extract features from any expression image using the downsampling module to obtain the second common feature, and to calculate the residual of the second common feature using the residual module to obtain the second residual tensor.

[0118] The activation vector acquisition unit is used to perform region detection on the second common feature using the identity decoupling module, generate the expected feature vector of the expression image, input the second residual tensor and the expected feature vector into the state classification module for region reshaping, and obtain the activation feature vector of the expression image.

[0119] The region category statistics unit is used to obtain the activation feature vector of each expression image and to count all region categories based on the activation feature vector of each expression image.

[0120] The sample pair acquisition unit is used to identify facial expression images containing the region category as positive samples and facial expression images not containing the region category as negative samples for any region category, and to acquire positive and negative samples under all region categories.

[0121] Optionally, the parameter adjustment module 44 includes:

[0122] The regional loss calculation unit is used to calculate the regional regression loss based on the regional differences between the target calibration results and the prediction calibration results.

[0123] The labeling loss calculation unit is used to constrain the first residual tensor using the initial calibration results to obtain the labeling mask loss;

[0124] The feature loss calculation unit is used to obtain the actual feature vector corresponding to the label of each expression image, and calculate the activation loss based on the feature difference between the actual feature vector and the activation feature vector.

[0125] The decoupling loss calculation unit is used to compare and learn positive and negative samples under all region categories to obtain the identity decoupling loss;

[0126] The model parameter adjustment unit is used to adjust the parameters of the region partitioning model based on the region regression loss, label mask loss, activation loss, and identity decoupling loss.

[0127] Optionally, the decoupling loss calculation unit includes:

[0128] The feature acquisition subunit is used to acquire the anchor features and positive sample features of each positive sample under any region category, as well as the negative sample features of each negative sample, for any region category.

[0129] The difference analysis subunit is used to obtain a first difference result based on the feature difference between the anchor point feature and the positive sample feature corresponding to each positive sample, and to obtain a second difference result based on the feature difference between the positive sample feature of each positive sample and the negative sample feature of each negative sample.

[0130] The difference processing subunit is used to obtain the first difference result and the second difference result under all region categories, and calculate the identity decoupling loss based on all the first difference results and all the second difference results.

[0131] It should be noted that the information interaction and execution process between the above modules, units, and sub-units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0132] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above-described embodiments of the region partitioning methods.

[0133] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0134] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0135] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0136] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0137] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0140] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for dividing the region of a facial action unit, characterized in that, The region division method includes: Obtain the region segmentation model and the target face image to be segmented; Obtain a training set of manually labeled first face images and unlabeled expression images. Based on the labeling result of the first face image, perform region labeling on the images in the expression image training set to obtain the target labeling result of the corresponding training image. The region segmentation model is used to perform region segmentation prediction on the images in the expression image training set to obtain the prediction and calibration results of the corresponding training images; Based on the target calibration result and the prediction calibration result, adjust the parameters of the region partitioning model until the preset conditions are met to obtain the trained region partitioning model; The trained region segmentation model is used to segment the target face image into regions, and the region segmentation result corresponding to the target face image is obtained. The step of performing region labeling on the images in the expression image training set based on the labeling result of the first face image to obtain the target labeling result of the corresponding training image includes: Based on the calibration results of the first face image, determine the three-dimensional region mask corresponding to each calibrated region in the first face image; Extract the three-dimensional vertex position information of the first face image, and use a preset two-dimensional texture transformation model to tile the three-dimensional vertex position information to obtain the first tiling result; Based on the mapping relationship between the three-dimensional vertex position information and each three-dimensional region mask, each three-dimensional region mask is aligned in the first tiling result to obtain the aligned two-dimensional region mask. For any training image, the three-dimensional vertex position information of the training image is extracted, and the three-dimensional vertex position information of the training image is tiled using a preset two-dimensional texture transformation model to obtain a second tiling result; Based on the mapping relationship between the three-dimensional vertex position information and each three-dimensional region mask, the target three-dimensional region mask corresponding to the aligned two-dimensional region mask in the second tiling result is obtained, and the initial calibration result of the training image is generated using the target three-dimensional region mask. After generating the initial calibration results of the training images, the process includes: The initial calibration result is approximated by an ellipse using preset ellipse parameters, and the initial calibration result after ellipse approximation is confirmed as the target calibration result.

2. The region division method according to claim 1, characterized in that, The region segmentation model includes a downsampling module, a residual module, and a region regression module. The step of using the region segmentation model to predict regions in the images of the facial expression image training set, and obtaining the prediction and labeling results for the corresponding training images, includes: For any training image, the downsampling module is used to extract features from the training image to obtain the first common feature; The residual module is used to perform residual calculation on the first common feature to obtain the first residual tensor; The region regression module is used to perform region division prediction on the first residual tensor to obtain the prediction calibration result.

3. The region division method according to claim 2, characterized in that, The region partitioning model includes an identity decoupling module and a state classification module. Before adjusting the parameters of the region partitioning model based on the target calibration result and the prediction calibration result, it includes: Get B labeled facial expression images, where B is a positive integer; For any facial expression image, the downsampling module is used to extract features from the facial expression image to obtain a second common feature, and the residual module is used to calculate the residual of the second common feature to obtain a second residual tensor; The identity decoupling module is used to perform region detection on the second common feature to generate the expected feature vector of the expression image. The second residual tensor and the expected feature vector are then input into the state classification module for region reshaping to obtain the activation feature vector of the expression image. Obtain the activation feature vector of each of the facial expression images, and count all region categories based on the activation feature vector of each of the facial expression images; For any region category, the facial expression image containing the region category is identified as a positive sample, and the facial expression image not containing the region category is identified as a negative sample. The positive samples and negative samples under all region categories are obtained.

4. The region division method according to claim 3, characterized in that, The step of adjusting the parameters of the region partitioning model based on the target calibration result and the prediction calibration result includes: Calculate the regional regression loss based on the regional differences between the target calibration result and the prediction calibration result; The first residual tensor is constrained using the initial calibration results to obtain the labeled mask loss; Obtain the actual feature vector corresponding to the label of each of the facial expression images, and calculate the activation loss based on the feature difference between the actual feature vector and the activation feature vector; By comparing and learning the positive and negative samples under all region categories, the identity decoupling loss is obtained. The parameters of the region partitioning model are adjusted based on the region regression loss, the labeling mask loss, the activation loss, and the identity decoupling loss.

5. The region division method according to claim 4, characterized in that, The step of comparing and learning the positive and negative samples under all region categories to obtain the identity decoupling loss includes: For any region category, obtain the anchor point features and positive sample features of each positive sample under the region category, as well as the negative sample features of each negative sample; A first difference result is obtained based on the feature difference between the anchor point feature and the positive sample feature corresponding to each positive sample; and a second difference result is obtained based on the feature difference between the positive sample feature of each positive sample and the negative sample feature of each negative sample. Obtain the first difference results and the second difference results for all region categories, and calculate the identity decoupling loss based on all the first difference results and all the second difference results.

6. A facial motion unit region division device, characterized in that, The region division device includes: The information acquisition module is used to acquire the region segmentation model and the target face image to be segmented; The result calibration module is used to acquire a manually calibrated first face image and an uncalibrated expression image training set, and to perform region calibration on the images in the expression image training set based on the calibration result of the first face image to obtain the target calibration result of the corresponding training image. The result prediction module is used to perform region segmentation prediction on the images in the expression image training set using the region segmentation model, and obtain the prediction and calibration results of the corresponding training images. The parameter adjustment module is used to adjust the parameters of the region partitioning model according to the target calibration result and the prediction calibration result until the preset conditions are met, so as to obtain the trained region partitioning model. The region segmentation module is used to segment the target face image into regions using the trained region segmentation model, and obtain the region segmentation result corresponding to the target face image; The result calibration module includes: The mask acquisition unit is used to determine the three-dimensional region mask corresponding to each calibrated region in the first face image based on the calibration result of the first face image. The first tiling unit is used to extract the three-dimensional vertex position information of the first face image, and to perform tiling processing on the three-dimensional vertex position information using a preset two-dimensional texture transformation model to obtain the first tiling result. The mask alignment unit is used to align each three-dimensional region mask in the first tiling result according to the mapping relationship between the three-dimensional vertex position information and each three-dimensional region mask, so as to obtain an aligned two-dimensional region mask. The second tiling unit is used to extract the three-dimensional vertex position information of any training image, and to perform tiling processing on the three-dimensional vertex position information of the training image using a preset two-dimensional texture transformation model to obtain a second tiling result. The initial calibration unit is used to obtain the target three-dimensional region mask corresponding to the aligned two-dimensional region mask in the second tiling result according to the mapping relationship between the three-dimensional vertex position information and each three-dimensional region mask, and to generate the initial calibration result of the training image using the target three-dimensional region mask; The result calibration module includes: An ellipse fitting unit is used to approximate each calibration region in the initial calibration result with an ellipse using preset ellipse parameters, and to confirm that the initial calibration result after ellipse approximation is the target calibration result.

7. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the region division method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the region division method as described in any one of claims 1 to 5.

Images