Facial movement unit detection method, apparatus and device

Abstract

The application provides a face movement unit detection method, device and equipment, the method comprising: inputting a source domain image into a detection network to obtain a source domain feature, and inputting the source domain feature into a first classifier to obtain a source domain prediction vector; inputting a target domain image into the detection network to obtain a target domain feature, and inputting the target domain feature into the first classifier to obtain a target domain prediction vector; determining a source domain distribution vector based on the source domain prediction vector, and determining a target domain distribution vector based on the target domain prediction vector; inputting the source domain distribution vector into a second classifier to obtain a first classification result, inputting the target domain distribution vector into the second classifier to obtain a second classification result, determining a first loss value based on the first classification result and the second classification result, adjusting network parameters of the detection network and the first classifier based on the first loss value, and generating a target network model based on the adjusted detection network and the first classifier. According to the application, the training time is short, and the resource consumption is small.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus and device for detecting facial motion units. Background Technology

[0002] Facial Action Units (AUs) are the basic units that make up facial expressions. They describe the degree of movement of different parts of the face during an expression. There are a total of 44 facial action units, which together make up all possible facial expressions. For example, facial action units can be used to describe different aspects of facial expressions, such as the degree to which the eyes are open, the degree to which the eyebrows are raised, and the degree to which the mouth is open.

[0003] To identify facial motion units in an image to be detected, a facial motion unit detection model can be pre-trained. The image to be detected is input into the facial motion unit detection model, which then outputs the facial motion units in the image. To make the facial motion unit detection model applicable to all scenarios, a dataset for each scenario needs to be collected, and each sample image in the dataset needs to be labeled. The facial motion unit detection model is then trained based on all sample images in the dataset.

[0004] However, due to the diverse range of real-world scenarios, labeling the datasets for each scenario requires a significant amount of time. This means it takes a long time to train the facial motion unit detection model, resulting in prolonged training time and low efficiency. Furthermore, it requires substantial resources (such as computational resources) to complete the labeling process, leading to high resource consumption. Consequently, the labeling cost is high, and the labeling efficiency is low. Summary of the Invention

[0005] This application provides a method for detecting facial motion units, the method comprising:

[0006] The source domain image is input into the detection network to obtain source domain features, and the source domain features are input into the first classifier to obtain the source domain prediction vector; the target domain image is input into the detection network to obtain target domain features, and the target domain features are input into the first classifier to obtain the target domain prediction vector.

[0007] The source domain distribution vector is determined based on the source domain prediction vector, and the target domain distribution vector is determined based on the target domain prediction vector. The source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0008] The source domain distribution vector is input to the second classifier to obtain the first classification result, the target domain distribution vector is input to the second classifier to obtain the second classification result, the first loss value is determined based on the first classification result and the second classification result, the network parameters of the detection network and the first classifier are adjusted based on the first loss value, and the target network model is generated based on the adjusted detection network and the first classifier.

[0009] The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

[0010] This application provides a facial motion unit detection device, the device comprising:

[0011] The acquisition module is used to input a source domain image into a detection network to obtain source domain features, and input the source domain features into a first classifier to obtain a source domain prediction vector; input a target domain image into a detection network to obtain target domain features, and input the target domain features into a first classifier to obtain a target domain prediction vector;

[0012] The determination module is used to determine the source domain distribution vector based on the source domain prediction vector and the target domain distribution vector based on the target domain prediction vector. The source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0013] The acquisition module is further configured to input the source domain distribution vector to the second classifier to obtain a first classification result, and input the target domain distribution vector to the second classifier to obtain a second classification result;

[0014] The determining module is further configured to determine a first loss value based on the first classification result and the second classification result, adjust the network parameters of the detection network and the first classifier based on the first loss value, and generate a target network model based on the adjusted detection network and the first classifier;

[0015] The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

[0016] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the facial motion unit detection method of the above example.

[0017] As can be seen from the above technical solutions, in this embodiment, a target network model (i.e., a facial motion unit detection model) is trained based on source domain images and target domain images. The source domain images are labeled images, and the target domain images are unlabeled images. This means that only a portion of the scene dataset (the source domain dataset) needs to be labeled, while the remaining scene dataset (the target domain dataset) does not need to be labeled. Therefore, the labeling work can be completed in a short time, resulting in a short training time and high training efficiency. Only a small amount of resources (such as computing resources) are required to complete the labeling work, resulting in low resource consumption. The labeling cost is low, and the labeling efficiency is high. The facial motion unit detection model can be trained based on the source domain distribution vector and the target domain distribution vector. Since the source domain distribution vector and the target domain distribution vector reflect the invariant relationship characteristics of facial motion units, training the facial motion unit detection model based on these invariant relationship characteristics improves the accuracy of facial motion unit detection and enhances the performance of facial motion unit recognition in the target domain. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0019] Figure 1 This is a flowchart illustrating a facial motion unit detection method according to one embodiment of this application;

[0020] Figure 2A This is a schematic diagram of the structure of the initial network model in one embodiment of this application;

[0021] Figure 2B This is a schematic diagram of the training process of the initial network model in one embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of the initial network model in one embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of the initial network model in one embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of the initial network model in one embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the facial motion unit detection device in one embodiment of this application;

[0026] Figure 7 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0027] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0029] This application proposes a facial motion unit detection method, which can be applied to any type of device. See [link to relevant documentation]. Figure 1 The diagram shown is a flowchart of the method, which may include:

[0030] Step 101: Input the source domain image into the detection network to obtain source domain features, and input the source domain features into the first classifier to obtain the source domain prediction vector; input the target domain image into the detection network to obtain target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0031] Step 102: Determine the source domain distribution vector based on the source domain prediction vector, and determine the target domain distribution vector based on the target domain prediction vector. The source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0032] Step 103: Input the source domain distribution vector into the second classifier to obtain the first classification result, input the target domain distribution vector into the second classifier to obtain the second classification result, determine the first loss value based on the first and second classification results, adjust the network parameters of the detection network and the first classifier based on the first loss value, and generate a target network model based on the adjusted detection network and the first classifier. The target network model is used to determine the target facial motion units corresponding to the image to be detected.

[0033] For example, determining the source domain distribution vector based on the source domain prediction vector and the target domain distribution vector based on the target domain prediction vector can include, but is not limited to, using the following formulas to determine the source domain distribution vector and the target domain distribution vector: Among them, if If the source domain prediction vector is given, then... This is the matrix transpose of the source domain prediction vectors, and `diag` is used to denote the normalization of the matrix's diagonal. Let be the source domain distribution vector; furthermore, if If the target domain is the predicted vector, then... This is the matrix transpose of the target domain prediction vector; `diag` is used to denote the normalization of the matrix's diagonal. This is the distribution vector of the target domain.

[0034] For example, determining the first loss value based on the first classification result and the second classification result may include, but is not limited to, using the following formula to determine the first loss value: in, The first loss value, The source domain distribution vector, This is the first classification result, which represents the predicted probability value of the source domain or target domain corresponding to the source domain distribution vector. The target domain distribution vector, The second classification result represents the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

[0035] For example, after inputting the source domain features into the first classifier to obtain the source domain prediction vector, a second loss value can be determined based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein, the source domain prediction vector includes N predicted probability values ​​corresponding to N facial motion units, and the label vector includes N true values ​​corresponding to N facial motion units, where N is a positive integer. Furthermore, adjusting the network parameters of the detection network and the first classifier based on the first loss value can include, but is not limited to, adjusting the network parameters of the detection network and the first classifier based on the first loss value and the second loss value.

[0036] For example, after inputting the source domain image into the detection network to obtain source domain features and the target domain image into the detection network to obtain target domain features, the source domain features can be input into a third classifier to obtain a third classification result, and the target domain features can be input into the third classifier to obtain a fourth classification result. A third loss value is then determined based on the third and fourth classification results. Furthermore, adjusting the network parameters of the detection network and the first classifier based on the first and second loss values ​​can include, but is not limited to: adjusting the network parameters of the detection network based on the first, second, and third loss values, and adjusting the network parameters of the first classifier based on the first and second loss values.

[0037] For example, the detection network may include, but is not limited to, a feature extraction network and a global pooling network. It may also input a stitched image into the feature extraction network to obtain stitched features. This stitched image may include K image regions, where K can be a positive integer greater than 1. For each image region, the image region originates from either a source domain image or a target domain image. The stitched features are then input into a fourth classifier to obtain a fifth classification result corresponding to each image region in the stitched image. A fourth loss value is determined based on the fifth classification results corresponding to the K image regions. Furthermore, the network parameters of the detection network are adjusted based on the first, second, and third loss values, which may include, but is not limited to: adjusting the network parameters of the feature extraction network based on the first, second, third, and fourth loss values, and adjusting the network parameters of the global pooling network based on the first, second, and third loss values.

[0038] For example, adjusting the network parameters of the first classifier based on the first loss value and the second loss value, adjusting the network parameters of the global pooling network based on the first loss value, the second loss value and the third loss value, and adjusting the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value and the fourth loss value may include, but is not limited to: calculating the first gradient value corresponding to the first loss value, the second gradient value corresponding to the second loss value, the third gradient value corresponding to the third loss value and the fourth gradient value corresponding to the fourth loss value; based on this, the network parameters of the first classifier may be adjusted based on the inverted gradient values ​​of the first gradient value and the second gradient value; the network parameters of the global pooling network may be adjusted based on the inverted gradient values ​​of the first gradient value, the second gradient value and the third gradient value; and the network parameters of the feature extraction network may be adjusted based on the inverted gradient values ​​of the first gradient value, the second gradient value, the third gradient value and the fourth gradient value.

[0039] For example, after obtaining the first gradient value, the third gradient value, and the fourth gradient value, the network parameters of the second classifier can be adjusted based on the first gradient value; the network parameters of the third classifier can be adjusted based on the third gradient value; and the network parameters of the fourth classifier can be adjusted based on the fourth gradient value.

[0040] For example, after generating a target network model (i.e., a facial motion unit detection model) based on the adjusted detection network and the first classifier, the image to be detected can be input into the detection network in the target network model to obtain the features to be detected; the features to be detected can be input into the first classifier in the target network model to obtain the prediction vector to be detected, which includes N predicted probability values ​​corresponding to N facial motion units; based on the N predicted probability values ​​corresponding to N facial motion units, facial motion units with prediction probability values ​​greater than a preset threshold are identified as the target facial motion units corresponding to the image to be detected.

[0041] As can be seen from the above technical solutions, in this embodiment, a target network model (i.e., a facial motion unit detection model) is trained based on source domain images and target domain images. The source domain images are labeled images, and the target domain images are unlabeled images. This means that only a portion of the scene dataset (the source domain dataset) needs to be labeled, while the remaining scene dataset (the target domain dataset) does not need to be labeled. Therefore, the labeling work can be completed in a short time, resulting in a short training time and high training efficiency. Only a small amount of resources (such as computing resources) are required to complete the labeling work, resulting in low resource consumption. The labeling cost is low, and the labeling efficiency is high. The facial motion unit detection model can be trained based on the source domain distribution vector and the target domain distribution vector. Since the source domain distribution vector and the target domain distribution vector reflect the invariant relationship characteristics of facial motion units, training the facial motion unit detection model based on these invariant relationship characteristics improves the accuracy of facial motion unit detection and enhances the performance of facial motion unit recognition in the target domain.

[0042] The technical solutions of the embodiments of this application will be described below in conjunction with specific application scenarios.

[0043] Facial motion units (FMUs) are the basic units that make up facial expressions, describing the degree of movement of different parts of the face. In order to identify facial motion units in an image to be detected, a facial motion unit detection model can be trained. For example, a dataset for each scene can be collected, and each sample image in the dataset can be labeled. Then, a facial motion unit detection model can be trained based on all the sample images in the dataset.

[0044] However, due to the diverse range of real-world scenarios, labeling the datasets for each scenario requires a significant amount of time. This means it takes a long time to train the facial motion unit detection model, resulting in prolonged training time and low efficiency. Furthermore, it requires substantial resources (such as computational resources) to complete the labeling process, leading to high resource consumption. Consequently, the labeling cost is high, and the labeling efficiency is low.

[0045] To address the aforementioned findings, this application's embodiments train a facial motion unit detection model based on source and target domain images. Only a portion of the scene dataset (the source domain dataset) is labeled, while the remaining scene dataset (the target domain dataset) is not labeled. This allows for labeling in a short time, resulting in low labeling costs and high efficiency. Furthermore, when training the facial motion unit detection model, features invariant to the global and local feature domains, as well as the invariant relationship between facial motion units, are fully considered. This improves the accuracy of facial motion unit detection, enhances the performance of facial motion unit recognition in the target domain, and improves the performance of expression analysis.

[0046] Among these, the features based on global feature domain invariance require that the global features extracted from the source domain image and the global features extracted from the target domain image be inseparable. The features based on local feature domain invariance require that the features corresponding to the region of interest remain unchanged in both the source and target domain images. Based on the relational invariance of facial motion units (FMUs), the relationships between different FMUs must be fully considered. For example, since FMUs are the smallest units of facial expression, when an expression appears, multiple FMUs are often activated, indicating a positive correlation between different FMUs. Conversely, when a smiling expression appears, a crying expression cannot appear simultaneously, indicating a negative correlation between different FMUs; that is, when one FMU appears, other FMUs will not appear. In summary, the relationships between different FMUs are determined by physiological structure and are not affected by the scene. Therefore, based on the relational invariance of FMUs, a model training method based on the relational invariance of FMUs can be proposed to fully consider the relationships between FMUs and improve the accuracy of FMU detection.

[0047] See Figure 2A The diagram shown illustrates the structure of the initial network model (facial motion unit detection model). This initial network model may include a feature extraction network, a global pooling network, a first classifier, a second classifier, a third classifier, and a fourth classifier. The feature extraction network and the global pooling network can form the detection network. The functions of the feature extraction network, the global pooling network, the first classifier, the second classifier, the third classifier, and the fourth classifier are explained in subsequent embodiments and will not be repeated here.

[0048] For example, an initial network model can be trained to obtain a target network model. The target network model may include a pre-trained feature extraction network, a global pooling network, and a first classifier. See [link to relevant documentation]. Figure 2B The diagram shown illustrates the initial network model training process, which may include:

[0049] Step 201: Obtain the source domain dataset and the target domain dataset.

[0050] For example, datasets from multiple scenes can be obtained. The labeled dataset is called the source domain dataset, and the sample images in the source domain dataset are called source domain images. The unlabeled dataset is called the target domain dataset, and the sample images in the target domain dataset are called target domain images. In summary, it can be seen that only the source domain datasets for a portion of the scenes need to be labeled, while the target domain datasets for the remaining scenes do not need to be labeled.

[0051] For example, the source domain dataset can be represented as This represents the k-th source domain image in the source domain dataset. This represents the label vector corresponding to the k-th source domain image. This label vector can include N ground truth values ​​corresponding to N facial motion units, where N can be a positive integer, such as 44. For each ground truth value (i.e., label value) corresponding to a facial motion unit, if the ground truth value is the first value (e.g., 1), it indicates that the facial motion unit is activated in the k-th source domain image; if the ground truth value is the second value (e.g., 0), it indicates that the facial motion unit is not activated in the k-th source domain image. K s This represents the total number of source domain images in the source domain dataset.

[0052] For example, the target domain dataset can be represented as This is used to represent the k-th target domain image in the target domain dataset. Obviously, no corresponding label vector exists for the target domain images in the target domain dataset. t Used to represent the total number of target domain images in the target domain dataset.

[0053] In summary, the source domain images in the source domain dataset are labeled data, while the target domain images in the target domain dataset are unlabeled data. The target network model can be trained using the labeled data from the source domain dataset and the unlabeled data from the target domain dataset, thereby improving the recognition performance of the lower face motion units in the target domain.

[0054] Step 202: Input the source domain images (such as multiple source domain images) in the source domain dataset into the detection network to obtain source domain features, and input the source domain features into the first classifier to obtain the source domain prediction vector.

[0055] For example, the detection network can include a feature extraction network and a global pooling network. The feature extraction network can be a neural network, a deep learning network, etc., and its structure is not limited, as long as it can achieve the feature extraction function. The global pooling network can be a neural network, a deep learning network, etc., and its structure is not limited, as long as it can achieve the pooling function. For example, the global pooling network can be GAP (Global Average Pooling).

[0056] For example, all source domain images in the source domain dataset can be used. The input is fed into a feature extraction network, which extracts features from the source domain image. The extracted features are then fed into a global pooling network, which performs global pooling to obtain the source domain image. Corresponding source domain features This represents the source domain features corresponding to the first source domain image. This represents the source domain feature corresponding to the second source domain image, and so on.

[0057] For example, the first classifier can be denoted as C. au The first classifier C au This is used to classify facial motion units and output the predicted probability value of each facial motion unit. For example, after obtaining the source domain features corresponding to the source domain image, these source domain features can be input into the first classifier C. au The first classifier C au Based on the source domain features, a source domain prediction vector corresponding to the source domain image is obtained. For example, assuming there are a total of 44 facial motion units, the source domain prediction vector includes 44 prediction probability values. The first prediction probability value represents the prediction probability value of the first facial motion unit appearing, the second prediction probability value represents the prediction probability value of the second facial motion unit appearing, and so on.

[0058] For example, when obtaining source domain features Next, source domain features Input to the first classifier C au The source domain prediction vector is obtained. The source domain prediction vector is the model's prediction of the probability of facial motion units appearing. The model predicts the facial motion unit vector for the first source domain image (which can be called the source domain prediction vector corresponding to the first source domain image), with dimension N, where N represents the number of facial motion units (e.g., 44), and so on.

[0059] Step 203: Input the target domain images (such as multiple target domain images) in the target domain dataset into the detection network to obtain target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0060] For example, all target domain images in the target domain dataset can be used. The input is fed into a feature extraction network, which extracts features from the target domain image. The extracted features are then fed into a global pooling network, which performs global pooling to obtain the target domain image. Corresponding target domain features This represents the target domain features corresponding to the first target domain image. This represents the target domain feature corresponding to the second target domain image, and so on.

[0061] For example, after obtaining the target domain features corresponding to the target domain image, these target domain features can be input into the first classifier C. au The first classifier C au Based on the features of the target domain, a target domain prediction vector corresponding to the target domain image is obtained, which includes N prediction probability values.

[0062] For example, in obtaining the features of the target domain Next, the target domain features Input to the first classifier C au The target domain prediction vector is obtained. The model predicts the facial motion unit vector for the first target domain image (which can be called the target domain prediction vector corresponding to the first target domain image), with dimension N, where N represents the number of facial motion units, and so on.

[0063] Step 204: Determine the source domain distribution vector based on the source domain prediction vector, and determine the target domain distribution vector based on the target domain prediction vector. The source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0064] For example, the source domain distribution vector and the target domain prediction vector can be denoted as... It is an N*N matrix, with For example, the value range of i is 1-N, and the value range of j is 1-N. This represents the co-occurrence probability of the i-th facial motion unit and the j-th facial motion unit, which is the correlation between the i-th facial motion unit and the j-th facial motion unit. If there is a positive correlation, it is 1, and if there is a negative correlation, it is -1.

[0065] Obviously, the greater the probability of the i-th facial motion unit and the j-th facial motion unit co-occurring, the better. The closer the value of is to 1, the smaller the probability of the i-th facial motion unit and the j-th facial motion unit co-occurring. The closer the value of is to -1. If i and j are the same, then The value is 1.

[0066] For example, after obtaining the source domain prediction vector, the source domain distribution vector can be determined based on this vector. There are no restrictions on the method of determination, as long as the above relationship is satisfied. Similarly, after obtaining the target domain prediction vector, the target domain distribution vector can be determined based on this vector. There are no restrictions on the method of determination, as long as the above relationship is satisfied. For instance, the source domain distribution vector and the target domain distribution vector can be determined using the following formula:

[0067] For example, based on the aforementioned source domain prediction vector The source domain distribution vector can then be determined using the following formula. in, For the source domain prediction vector, The matrix transpose of the source domain prediction vector. Let be the source domain distribution vector, and diag denotes normalization of the matrix diagonally.

[0068] For example, predicting vectors based on the target domain The target domain distribution vector can be determined using the following formula. in, Predict vectors for the target domain. The matrix transpose of the target domain prediction vector. Let be the target domain distribution vector, and diag denotes the normalization of the matrix diagonal.

[0069] In summary, the source domain distribution vector is obtained. and target domain distribution vector It is an N*N matrix, with For example, This represents the co-occurrence probability of the i-th facial motion unit and the j-th facial motion unit, i.e., the correlation between the i-th facial motion unit and the j-th facial motion unit. It is an N*N matrix, with For example, This represents the co-occurrence probability of the i-th facial motion unit and the j-th facial motion unit, i.e., the correlation between the i-th facial motion unit and the j-th facial motion unit.

[0070] For example, the source domain distribution vector and the target domain distribution vector can reflect the relationship-invariant properties of facial motion units, that is, reflect the relationship between different facial motion units, such as a positive correlation between different facial motion units, or a negative correlation between different facial motion units.

[0071] Step 205: Input the source domain distribution vector into the second classifier to obtain a first classification result, and input the target domain distribution vector into the second classifier to obtain a second classification result. Determine a first loss value based on the first and second classification results. For example, the first classification result can be used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or target domain, and the second classification result can be used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or target domain.

[0072] For example, the second classifier can be denoted as C. d Second classifier C d It is used to implement the classification function of source domain and target domain, and to output the predicted probability value of source domain or target domain.

[0073] For example, after obtaining the source domain distribution vector corresponding to the source domain image. Then, the source domain distribution vector can be... Input to the second classifier C d The second classifier C d Based on source domain distribution vector The process yields the first classification result corresponding to the source domain image. The second classifier C... d It can be based on the source domain distribution vector Determine the predicted probability values ​​for the source domain and the target domain. If the predicted probability value for the source domain is greater than that for the target domain, the predicted probability value for the source domain (i.e., the first classification result) can be output. Obviously, the higher the predicted probability value for the source domain, the more accurate the prediction result. If the predicted probability value for the source domain is not greater than that for the target domain, the predicted probability value for the target domain (i.e., the first classification result) can be output. Obviously, the higher the predicted probability value for the target domain, the less accurate the prediction result.

[0074] After obtaining the target domain distribution vector, the target domain distribution vector is input into the second classifier C. d The second classifier C d The second classification result is obtained by processing the target domain distribution vector. The second classifier C... d The predicted probability values ​​for the source domain and the target domain are determined based on the target domain distribution vector. If the predicted probability value for the source domain is greater than that for the target domain, the predicted probability value for the source domain (i.e., the second classification result) is output. Obviously, the higher the predicted probability value for the source domain, the less accurate the prediction result. If the predicted probability value for the source domain is not greater than that for the target domain, the predicted probability value for the target domain (i.e., the second classification result) is output. Obviously, the higher the predicted probability value for the target domain, the more accurate the prediction result.

[0075] For example, after obtaining the first classification result and the second classification result, a first loss value can be determined based on the first classification result and the second classification result. For instance, a first loss function can be pre-configured. There are no restrictions on this first loss function, as long as it is related to the first classification result and the second classification result. An example of the first loss function can be seen in the following formula, that is, the first loss value can be determined using the following formula:

[0076] In the above formula, The first loss value, The source domain distribution vector, The first classification result is the result of the second classifier C. d The predicted probability value output for the source domain distribution vector. The target domain distribution vector, The result of the second classification is the second classifier C. d The predicted probability value is output for the target domain distribution vector. Clearly, as can be seen from the first loss function, to control the first loss value to be as small as possible, the second classifier C... d Able to distinguish and Soon Identify as the corresponding source domain, and It is identified as the corresponding target domain.

[0077] For example, since the calculation is performed on all data The cost is high; therefore, during training, a mini-batch strategy can be adopted, calculating the computation time for each batch of samples. There might be bias. To address this issue, a second classifier can be used, and adversarial training can be employed to improve its performance. and The distributions are close, meaning the predicted values ​​are similar. and The goal is to make them inseparable, thus bringing them closer together. In this way, the relationship between facial motion units can be constrained to remain unchanged in both the source and target domains.

[0078] Step 206: Determine the second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein, the source domain prediction vector may include N predicted probability values ​​corresponding to N facial motion units, and the label vector may include N true values ​​corresponding to N facial motion units, where N is a positive integer.

[0079] For example, since the source domain prediction vector includes N predicted probability values ​​corresponding to N facial motion units, and the label vector includes N true values ​​corresponding to N facial motion units, a difference vector corresponding to the source domain image can be calculated. This difference vector includes N differences corresponding to N facial motion units. The first difference is the difference between the first predicted probability value and the first true value, the second difference is the difference between the second predicted probability value and the second true value, and so on. Obviously, the smaller the difference, the more accurate the prediction result. A second loss value can be determined based on the difference vector corresponding to the source domain image. For example, a second loss function can be pre-configured. There are no restrictions on this second loss function, as long as it is related to the difference vector. That is, the smaller the difference vector, the smaller the output of the second loss function (i.e., the second loss value).

[0080] Step 207: Input the source domain features into the third classifier to obtain the third classification result, input the target domain features into the third classifier to obtain the fourth classification result, and determine the third loss value based on the third and fourth classification results. The third classification result is used to represent the predicted probability value of the source domain features corresponding to the source domain or target domain, and the fourth classification result is used to represent the predicted probability value of the target domain features corresponding to the source domain or target domain.

[0081] For example, the third classifier can be denoted as C. s Third classifier C s This is used to perform source and target domain classification, and to output the predicted probability value of the source domain or the predicted probability value of the target domain. For example, it is used to obtain the source domain features corresponding to the source domain image. Then, the source domain features are input into the third classifier C. s The third classifier C s Based on the source domain features, a third classification result corresponding to the source domain image is obtained. Third classifier C sBased on the source domain features, the predicted probability values ​​for the source domain and the target domain are determined. If the predicted probability value for the source domain is greater than that for the target domain, the predicted probability value for the source domain is output (third-class classification result). A higher predicted probability value for the source domain indicates a more accurate prediction. If the predicted probability value for the source domain is not greater than that for the target domain, the predicted probability value for the target domain is output (third-class classification result). A higher predicted probability value for the target domain indicates a less accurate prediction. After obtaining the target domain features... Then, the target domain features are input into the third classifier C. s The third classifier C s Based on the target domain features, a fourth classification result is obtained. The third classifier C... s The predicted probability values ​​for the source domain and the target domain are determined based on the features of the target domain. If the predicted probability value for the source domain is greater than that for the target domain, the predicted probability value for the source domain (fourth classification result) is output. A higher predicted probability value for the source domain indicates a less accurate prediction. If the predicted probability value for the source domain is not greater than that for the target domain, the predicted probability value for the target domain (fourth classification result) is output. A higher predicted probability value for the target domain indicates a more accurate prediction.

[0082] For example, after obtaining the third and fourth classification results, a third loss value can be determined based on these results. For instance, a third loss function can be pre-configured. This third loss function is not restricted, as long as it is related to the third and fourth classification results. An example of a third loss function can be seen in the following formula, which can be used to determine the third loss value:

[0083] In the above formula, This is the third loss value. Features of the source domain The result is the third classification result, which is the third classifier C. s The predicted probability value output for the source domain features. For target domain features, The result of the fourth classification is the result of the third classifier C. s The predicted probability value is output based on the features of the target domain. Clearly, as can be seen from the third loss function, to control the third loss value to be as small as possible, the third classifier C... s Able to distinguish and Soon Identify as the corresponding source domain, and The source domain features are identified as the corresponding target domain. In summary, for properties where global features are invariant, the source domain features can be... and target domain features Send to the third classifier C s Let the third classifier C s To determine whether a feature originates from the source domain or the target domain, an adversarial approach is used to achieve the goal of making the global feature domain indivisible.

[0084] Step 208: Input the stitched image into the feature extraction network to obtain stitched features, and input the stitched features into the fourth classifier to obtain the fifth classification result corresponding to each image region in the stitched image, and determine the fourth loss value based on the fifth classification results corresponding to K image regions.

[0085] For example, a stitched image (multiple stitched images, hereinafter referred to as a single stitched image) can be constructed based on a source domain dataset and / or a target domain dataset. This stitched image can include K image regions, where K is a positive integer greater than 1, such as 3, 5, 8, etc., without restriction. For each image region, the image region originates from either the source domain image or the target domain image. For instance, a frame of image (source domain image or target domain image) is randomly selected from the source domain dataset or the target domain dataset, and a sub-image corresponding to the image region is extracted from that frame. This sub-image is then used as the image region in the stitched image. After obtaining K image regions, the K image regions can be combined to form a stitched image.

[0086] After obtaining the stitched image, it can be input into the feature extraction network, which processes the stitched image to obtain the stitched features corresponding to the stitched image. That is, the feature extraction network extracts the stitched features corresponding to the stitched image and inputs the stitched features into the fourth classifier.

[0087] For example, the fourth classifier can be denoted as C. r The fourth classifier C r This is used to perform source and target domain classification, outputting the predicted probability value of each image region corresponding to either the source or target domain. For example, after obtaining the stitched features, these features are input into the fourth classifier C. r The fourth classifier C r Based on the concatenated features, a fifth classification result is obtained for each image region. This fifth classification result is the predicted probability value of the source or target domain corresponding to that image region, i.e., the fourth classifier C. r Based on the stitching features, the predicted probability values ​​of the source domain and the target domain for each image region are determined, and the predicted probability value of the source domain or the target domain is output. The implementation process is described in the third classifier, and will not be repeated here.

[0088] For example, after obtaining the fifth classification result for each image region, the fourth loss value can be determined based on the fifth classification result. For instance, the fourth loss function can be pre-configured. There are no restrictions on this fourth loss function, as long as it is related to the fifth classification result. An example of the fourth loss function can be seen in the following formula, which can be used to determine the fourth loss value:

[0089]

[0090] In the above formula, This is the fourth loss value. For the i-th splicing feature, This is the fifth classification result corresponding to the first image region, which is the result of the fourth classifier C. r The predicted probability value output for the first image region, This is the fifth classification result corresponding to the second image region, which is the result of the fourth classifier C. r The predicted probability value is output for the second image region, and so on. This is the fifth classification result corresponding to the Kth image region, which is the result of the fourth classifier C. r The predicted probability value is output for the Kth image region. Clearly, as can be seen from the fourth loss function, to control the fourth loss value to be as small as possible, the fourth classifier C... r It can distinguish whether each image region corresponds to the source domain or the target domain. In summary, for features that are invariant to local features, the fourth classifier C is also trained. r The features before global pooling are fed into the fourth classifier C. r Let the fourth classifier C r Predict which domain each image region comes from, and achieve the goal of local feature domain inseparability through adversarial methods.

[0091] Step 209: Adjust the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value; adjust the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value; adjust the network parameters of the first classifier based on the first loss value and the second loss value; adjust the network parameters of the second classifier based on the first loss value; adjust the network parameters of the third classifier based on the third loss value; and adjust the network parameters of the fourth classifier based on the fourth loss value.

[0092] For example, the first gradient value corresponding to the first loss value, the second gradient value corresponding to the second loss value, the third gradient value corresponding to the third loss value, and the fourth gradient value corresponding to the fourth loss value can be calculated. For instance, the first gradient value, the second gradient value, the third gradient value, and the fourth gradient value can be calculated using the gradient descent method, or other algorithms can be used to calculate the first gradient value, the second gradient value, the third gradient value, and the fourth gradient value. There are no restrictions on this, as long as the gradient value can be calculated based on the loss value.

[0093] Then, the network parameters of the feature extraction network can be adjusted based on the inverted gradient values ​​of the first, second, third, and fourth gradients. There are no restrictions on this parameter adjustment process. The network parameters of the global pooling network can be adjusted based on the inverted gradient values ​​of the first, second, and third gradients. The network parameters of the first classifier can be adjusted based on the inverted gradient values ​​of the first and second gradients. The network parameters of the second classifier can be adjusted based on the first gradient value. The network parameters of the third classifier can be adjusted based on the third gradient value. The network parameters of the fourth classifier can be adjusted based on the fourth gradient value.

[0094] For example, the first loss value is used to control the relation-invariant properties of facial motion units, the third loss value is used to control the features that are invariant in the global feature domain, and the fourth loss value is used to control the features that are invariant in the local feature domain. Obviously, when adjusting the network parameters of the model based on the first, third, and fourth loss values, the relation-invariant properties of facial motion units, the features that are invariant in the global feature domain, and the features that are invariant in the local feature domain can be repeatedly considered, thereby improving the accuracy of facial motion unit detection, enhancing the performance of facial motion unit recognition in the target domain, and improving the performance of expression analysis.

[0095] Step 210: Determine whether the adjusted network model has converged. If not, use the adjusted network model as the initial network model and return to step 202. If yes, proceed to step 211.

[0096] For example, the prediction accuracy of the adjusted network model can be determined based on the validation dataset. If the prediction accuracy is greater than the convergence threshold (configured empirically, such as 98%), the adjusted network model is considered to have converged. If the prediction accuracy is not greater than the convergence threshold, the adjusted network model is considered to have not converged.

[0097] For example, if the number of iterations of the network model is greater than a preset threshold (which can be configured based on experience), then the adjusted network model is determined to have converged; otherwise, the adjusted network model is determined not to have converged.

[0098] For example, if the iteration duration of the network model is greater than the preset duration threshold (which can be configured based on experience), then it is determined that the adjusted network model has converged; otherwise, it is determined that the adjusted network model has not converged.

[0099] Of course, the above are just a few examples, and this embodiment does not impose any restrictions on the convergence conditions.

[0100] Step 211: Generate the target network model based on the adjusted feature extraction network, global pooling network and first classifier. That is, the adjusted feature extraction network, adjusted global pooling network and adjusted first classifier are combined to form the target network model (i.e., the facial motion unit detection model).

[0101] See Figure 3 The diagram shown is another structural illustration of the initial network model (facial motion unit detection model). The initial network model may include a feature extraction network, a global pooling network, a first classifier, and a second classifier. The feature extraction network and the global pooling network can form a detection network. For example, the initial network model can be trained to obtain the target network model. This training process may include:

[0102] Step S11: Obtain the source domain dataset and the target domain dataset.

[0103] Step S12: Input the source domain images (such as multiple source domain images) in the source domain dataset into the detection network to obtain source domain features, and input the source domain features into the first classifier to obtain the source domain prediction vector.

[0104] Step S13: Input the target domain images (such as multiple target domain images) in the target domain dataset into the detection network to obtain target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0105] Step S14: Determine the source domain distribution vector based on the source domain prediction vector, and determine the target domain distribution vector based on the target domain prediction vector. The source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0106] Step S15: Input the source domain distribution vector into the second classifier to obtain a first classification result, and input the target domain distribution vector into the second classifier to obtain a second classification result. Determine a first loss value based on the first and second classification results. For example, the first classification result can be used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or target domain, and the second classification result can be used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or target domain.

[0107] Step S16: Determine the second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein, the source domain prediction vector may include N predicted probability values ​​corresponding to N facial motion units, and the label vector may include N true values ​​corresponding to N facial motion units, where N is a positive integer.

[0108] Step S17: Adjust the network parameters of the feature extraction network based on the first loss value and the second loss value, adjust the network parameters of the global pooling network based on the first loss value and the second loss value, adjust the network parameters of the first classifier based on the first loss value and the second loss value, and adjust the network parameters of the second classifier based on the first loss value.

[0109] Step S18: Determine whether the adjusted network model has converged. If not, use the adjusted network model as the initial network model and return to step S12. If yes, proceed to step S19.

[0110] Step S19: Generate a target network model based on the adjusted feature extraction network, global pooling network and first classifier. That is, the adjusted feature extraction network, adjusted global pooling network and adjusted first classifier are combined to form a target network model (i.e., facial motion unit detection model).

[0111] See Figure 4 The diagram shown illustrates another structural design of the initial network model, which includes a feature extraction network, a global pooling network, a first classifier, a second classifier, and a third classifier. For example, the initial network model can be trained to obtain the target network model. This training process may include:

[0112] Step S21: Obtain the source domain dataset and the target domain dataset.

[0113] Step S22: Input the source domain images (such as multiple source domain images) in the source domain dataset into the detection network to obtain source domain features, and input the source domain features into the first classifier to obtain the source domain prediction vector.

[0114] Step S23: Input the target domain images (such as multiple target domain images) in the target domain dataset into the detection network to obtain target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0115] Step S24: Determine the source domain distribution vector based on the source domain prediction vector, and determine the target domain distribution vector based on the target domain prediction vector. The source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0116] Step S25: Input the source domain distribution vector into the second classifier to obtain a first classification result, and input the target domain distribution vector into the second classifier to obtain a second classification result. Determine a first loss value based on the first and second classification results. For example, the first classification result can be used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or target domain, and the second classification result can be used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or target domain.

[0117] Step S26: Determine the second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein, the source domain prediction vector may include N predicted probability values ​​corresponding to N facial motion units, and the label vector may include N true values ​​corresponding to N facial motion units, where N is a positive integer.

[0118] Step S27: Input the source domain features into the third classifier to obtain the third classification result, input the target domain features into the third classifier to obtain the fourth classification result, and determine the third loss value based on the third and fourth classification results. The third classification result is used to represent the predicted probability value of the source domain features corresponding to the source domain or target domain, and the fourth classification result is used to represent the predicted probability value of the target domain features corresponding to the source domain or target domain.

[0119] Step S28: Adjust the network parameters of the feature extraction network based on the first loss value, the second loss value, and the third loss value; adjust the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value; adjust the network parameters of the first classifier based on the first loss value and the second loss value; adjust the network parameters of the second classifier based on the first loss value; adjust the network parameters of the third classifier based on the third loss value.

[0120] Step S29: Determine whether the adjusted network model has converged. If not, use the adjusted network model as the initial network model and return to step S22. If yes, proceed to step S30.

[0121] Step S30: Generate a target network model based on the adjusted feature extraction network, global pooling network and first classifier. That is, the adjusted feature extraction network, adjusted global pooling network and adjusted first classifier are combined to form a target network model (i.e., facial motion unit detection model).

[0122] See Figure 5 The diagram shown illustrates another structural design of the initial network model, which includes a feature extraction network, a global pooling network, a first classifier, a second classifier, and a fourth classifier. For example, the initial network model can be trained to obtain the target network model. This training process may include:

[0123] Step S31: Obtain the source domain dataset and the target domain dataset.

[0124] Step S32: Input the source domain images (such as multiple source domain images) in the source domain dataset into the detection network to obtain source domain features, and input the source domain features into the first classifier to obtain the source domain prediction vector.

[0125] Step S33: Input the target domain images (such as multiple target domain images) in the target domain dataset into the detection network to obtain target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0126] Step S34: Determine the source domain distribution vector based on the source domain prediction vector, and determine the target domain distribution vector based on the target domain prediction vector. The source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0127] Step S35: Input the source domain distribution vector into the second classifier to obtain a first classification result, and input the target domain distribution vector into the second classifier to obtain a second classification result. Determine a first loss value based on the first and second classification results. For example, the first classification result can be used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or target domain, and the second classification result can be used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or target domain.

[0128] Step S36: Determine the second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein, the source domain prediction vector may include N predicted probability values ​​corresponding to N facial motion units, and the label vector may include N true values ​​corresponding to N facial motion units, where N is a positive integer.

[0129] Step S37: Input the stitched image into the feature extraction network to obtain stitched features, and input the stitched features into the fourth classifier to obtain the fifth classification result corresponding to each image region in the stitched image, and determine the fourth loss value based on the fifth classification results corresponding to K image regions.

[0130] Step S38: Adjust the network parameters of the feature extraction network based on the first loss value, the second loss value, and the fourth loss value; adjust the network parameters of the global pooling network based on the first loss value and the second loss value; adjust the network parameters of the first classifier based on the first loss value and the second loss value; adjust the network parameters of the second classifier based on the first loss value; and adjust the network parameters of the fourth classifier based on the fourth loss value.

[0131] Step S39: Determine whether the adjusted network model has converged. If not, use the adjusted network model as the initial network model and return to step S32. If yes, proceed to step S40.

[0132] Step S40: Generate a target network model based on the adjusted feature extraction network, global pooling network, and first classifier. That is, the adjusted feature extraction network, adjusted global pooling network, and adjusted first classifier are combined to form a target network model (i.e., a facial motion unit detection model).

[0133] In one possible implementation, after training the target network model, the target facial motion units corresponding to the image to be detected can be determined based on the target network model. For example, the image to be detected can be input into the detection network (such as a feature extraction network and a global pooling network) in the target network model to obtain the features to be detected. Then, the features to be detected can be input into the first classifier in the target network model to obtain the prediction vector to be detected. For the processing of the detection network and the first classifier, please refer to the above embodiment.

[0134] For example, the target network model can be applied to fields involving facial expression analysis, such as lie detectors, driver assistance systems, human-computer interaction, and virtual reality, without limitation. It can use images to be detected in these scenarios and input them into the target network model to obtain the predicted vector. It should be noted that all data involved in this specification were obtained and used with the knowledge and authorization of the relevant users.

[0135] For example, the prediction vector to be detected may include N predicted probability values ​​corresponding to N facial motion units. Based on the N predicted probability values ​​corresponding to the N facial motion units, facial motion units with predicted probability values ​​greater than a preset threshold can be identified as target facial motion units corresponding to the image to be detected. For instance, if the predicted probability value corresponding to facial motion unit 1 is greater than the preset threshold, then facial motion unit 1 is identified as the target facial motion unit corresponding to the image to be detected; if the predicted probability value corresponding to facial motion unit 2 is not greater than the preset threshold, then facial motion unit 2 is not considered a target facial motion unit, and so on.

[0136] As can be seen from the above technical solutions, in this embodiment, a facial motion unit detection model can be trained based on the source domain distribution vector and the target domain distribution vector. The source domain distribution vector and the target domain distribution vector reflect the relational invariance of facial motion units. Training the facial motion unit detection model based on the relational invariance of facial motion units improves the accuracy of facial motion unit detection and enhances the performance of facial motion unit recognition in the target domain. This embodiment proposes a multi-scale unsupervised domain-adaptive facial motion unit detection method. It constrains feature learning through three scales: relational invariance of facial motion units, global feature invariance, and local feature invariance. This improves the performance of the facial motion unit detection model in the target domain without labeled samples, saving significant labeling costs.

[0137] Based on the same concept as the above method, this application proposes a facial motion unit detection device, see [link to relevant documentation]. Figure 6 The diagram shown is a structural schematic of the device, which may include:

[0138] The acquisition module 61 is used to input a source domain image into a detection network to obtain source domain features, and input the source domain features into a first classifier to obtain a source domain prediction vector; input a target domain image into a detection network to obtain target domain features, and input the target domain features into a first classifier to obtain a target domain prediction vector;

[0139] The determining module 62 is used to determine the source domain distribution vector based on the source domain prediction vector and the target domain distribution vector based on the target domain prediction vector. The source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0140] The acquisition module 61 is further configured to input the source domain distribution vector to the second classifier to obtain a first classification result, and input the target domain distribution vector to the second classifier to obtain a second classification result;

[0141] The determining module 62 is further configured to determine a first loss value based on the first classification result and the second classification result, adjust the network parameters of the detection network and the first classifier based on the first loss value, and generate a target network model based on the adjusted detection network and the first classifier;

[0142] The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

[0143] For example, when determining the source domain distribution vector based on the source domain prediction vector, the determining module 62 specifically determines the target domain distribution vector based on the target domain prediction vector by using the following formulas: like If the source domain prediction vector is given, then... Let be the matrix transpose of the source domain prediction vector, and ... Let be the source domain distribution vector; if If the target domain is the predicted vector, then... Let diag be the matrix transpose of the target domain prediction vector, where diag denotes normalization of the matrix diagonal. is the distribution vector of the target domain.

[0144] For example, when determining the first loss value based on the first classification result and the second classification result, the determining module 62 is specifically used to: determine the first loss value using the following formula: The first loss value, The source domain distribution vector, The first classification result represents the predicted probability value of the source domain or target domain corresponding to the source domain distribution vector. The target domain distribution vector, The second classification result represents the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

[0145] For example, the determining module 62 is further configured to determine a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein, the source domain prediction vector includes N predicted probability values ​​corresponding to N facial motion units, and the label vector includes N true values ​​corresponding to N facial motion units, where N is a positive integer; when the determining module 62 adjusts the network parameters of the detection network and the first classifier based on the first loss value, it is specifically configured to: adjust the network parameters of the detection network and the first classifier based on the first loss value and the second loss value.

[0146] For example, the acquisition module 61 is further configured to input the source domain features into a third classifier to obtain a third classification result, and input the target domain features into a third classifier to obtain a fourth classification result; the determination module 62 is further configured to determine a third loss value based on the third classification result and the fourth classification result; when the determination module 62 adjusts the network parameters of the detection network and the first classifier based on the first loss value and the second loss value, it is specifically configured to: adjust the network parameters of the detection network based on the first loss value, the second loss value and the third loss value, and adjust the network parameters of the first classifier based on the first loss value and the second loss value.

[0147] For example, the detection network includes a feature extraction network and a global pooling network. The acquisition module 61 is further configured to input the stitched image into the feature extraction network to obtain stitched features. The stitched image includes K image regions, where K is a positive integer greater than 1. For each image region, the image region originates from a source domain image or a target domain image. The stitched features are input into a fourth classifier to obtain a fifth classification result corresponding to each image region in the stitched image. The determination module 62 is further configured to determine a fourth loss value based on the fifth classification results corresponding to the K image regions. When the determination module 62 adjusts the network parameters of the detection network based on the first loss value, the second loss value, and the third loss value, it is specifically configured to: adjust the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value; and adjust the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value.

[0148] For example, when the determining module 62 adjusts the network parameters of the first classifier based on the first loss value and the second loss value, adjusts the network parameters of the global pooling network based on the first loss value, the second loss value and the third loss value, and adjusts the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value and the fourth loss value, it specifically performs the following steps: calculating the first gradient value corresponding to the first loss value, the second gradient value corresponding to the second loss value, the third gradient value corresponding to the third loss value and the fourth gradient value corresponding to the fourth loss value; adjusting the network parameters of the first classifier based on the inverted gradient value of the first gradient value and the second gradient value; adjusting the network parameters of the global pooling network based on the inverted gradient value of the first gradient value, the second gradient value and the inverted gradient value of the third gradient value; and adjusting the network parameters of the feature extraction network based on the inverted gradient value of the first gradient value, the second gradient value, the inverted gradient value of the third gradient value and the inverted gradient value of the fourth gradient value.

[0149] The determining module 62 is further configured to adjust the network parameters of the second classifier based on the first gradient value; adjust the network parameters of the third classifier based on the third gradient value; and adjust the network parameters of the fourth classifier based on the fourth gradient value.

[0150] The facial motion unit detection device further includes: a detection module, used to input the image to be detected into the detection network in the target network model to obtain the features to be detected, and input the features to be detected into the first classifier in the target network model to obtain the prediction vector to be detected, wherein the prediction vector to be detected includes N prediction probability values ​​corresponding to N facial motion units; based on the N prediction probability values ​​corresponding to the N facial motion units, facial motion units with prediction probability values ​​greater than a preset threshold are determined as the target facial motion units corresponding to the image to be detected.

[0151] Based on the same concept as the above method, this application proposes an electronic device, see [link to previous application]. Figure 7 As shown, the electronic device includes a processor 71 and a machine-readable storage medium 72, the machine-readable storage medium 72 storing machine-executable instructions that can be executed by the processor 71; the processor 71 is used to execute the machine-executable instructions to implement the facial motion unit detection method disclosed in the above example of this application.

[0152] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the facial motion unit detection method disclosed in the above examples of this application.

[0153] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0154] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer entity or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0155] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0156] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting facial motion units, characterized in that, The method includes: The source domain image is input into the detection network to obtain source domain features, and the source domain features are input into the first classifier to obtain the source domain prediction vector; the target domain image is input into the detection network to obtain target domain features, and the target domain features are input into the first classifier to obtain the target domain prediction vector. The source domain distribution matrix is ​​determined based on the source domain prediction vector, and the target domain distribution matrix is ​​determined based on the target domain prediction vector. The source domain distribution matrix includes the co-occurrence probability of any two facial motion units, and the target domain distribution matrix includes the co-occurrence probability of any two facial motion units. The source domain distribution matrix is ​​input to the second classifier to obtain a first classification result, which represents the predicted probability value of the source domain or target domain corresponding to the source domain distribution matrix; the target domain distribution matrix is ​​input to the second classifier to obtain a second classification result, which represents the predicted probability value of the source domain or target domain corresponding to the target domain distribution matrix; a first loss value is determined based on the first and second classification results; the network parameters of the detection network and the first classifier are adjusted based on the first loss value; and a target network model is generated based on the adjusted detection network and the first classifier. The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

2. The method according to claim 1, characterized in that, The step of determining the source domain distribution matrix based on the source domain prediction vector and determining the target domain distribution matrix based on the target domain prediction vector includes: The source domain distribution matrix and the target domain distribution matrix are determined using the following formulas: Among them, if Let be the source domain prediction vector, then Let be the matrix transpose of the source domain prediction vector, and ... The source domain distribution matrix; like Let the target domain prediction vector be... Let diag be the matrix transpose of the target domain prediction vector, where diag denotes normalization of the matrix diagonal. Let be the distribution matrix of the target domain.

3. The method according to claim 1, characterized in that, The step of determining the first loss value based on the first classification result and the second classification result includes: determining the first loss value using the following formula: For the first loss value, The source domain distribution matrix is... The first classification result, The target domain distribution matrix, This is the result of the second classification.

4. The method according to any one of claims 1-3, characterized in that, After inputting the source domain features into the first classifier to obtain the source domain prediction vector, the method further includes: determining a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein, the source domain prediction vector includes N predicted probability values ​​corresponding to N facial motion units, and the label vector includes N true values ​​corresponding to N facial motion units, where N is a positive integer; The adjustment of the network parameters of the detection network and the first classifier based on the first loss value includes: adjusting the network parameters of the detection network and the first classifier based on the first loss value and the second loss value.

5. The method according to claim 4, characterized in that, After inputting the source domain image into the detection network to obtain source domain features and inputting the target domain image into the detection network to obtain target domain features, the method further includes: inputting the source domain features into a third classifier to obtain a third classification result, inputting the target domain features into a third classifier to obtain a fourth classification result, and determining a third loss value based on the third classification result and the fourth classification result; The network parameters of the detection network and the first classifier are adjusted based on the first loss value and the second loss value, including: adjusting the network parameters of the detection network based on the first loss value, the second loss value and the third loss value, and adjusting the network parameters of the first classifier based on the first loss value and the second loss value.

6. The method according to claim 5, characterized in that, The detection network includes a feature extraction network and a global pooling network. The method further includes: inputting the stitched image into the feature extraction network to obtain stitched features, wherein the stitched image includes K image regions, and for each image region, the image region originates from a source domain image or a target domain image; inputting the stitched features into a fourth classifier to obtain a fifth classification result corresponding to each image region in the stitched image; and determining a fourth loss value based on the fifth classification results corresponding to the K image regions. The adjustment of the network parameters of the detection network based on the first loss value, the second loss value, and the third loss value includes: adjusting the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value, and adjusting the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value.

7. The method according to claim 6, characterized in that, The network parameters of the first classifier are adjusted based on the first and second loss values; the network parameters of the global pooling network are adjusted based on the first, second, and third loss values; and the network parameters of the feature extraction network are adjusted based on the first, second, third, and fourth loss values, including: Calculate the first gradient value corresponding to the first loss value, the second gradient value corresponding to the second loss value, the third gradient value corresponding to the third loss value, and the fourth gradient value corresponding to the fourth loss value; Based on the inverted gradient value of the first gradient value and the second gradient value, the network parameters of the first classifier are adjusted; based on the inverted gradient value of the first gradient value, the second gradient value, and the inverted gradient value of the third gradient value, the network parameters of the global pooling network are adjusted; based on the inverted gradient value of the first gradient value, the second gradient value, the inverted gradient value of the third gradient value, and the inverted gradient value of the fourth gradient value, the network parameters of the feature extraction network are adjusted.

8. The method according to claim 7, characterized in that, After calculating the first gradient value corresponding to the first loss value, the second gradient value corresponding to the second loss value, the third gradient value corresponding to the third loss value, and the fourth gradient value corresponding to the fourth loss value, the method further includes: The network parameters of the second classifier are adjusted based on the first gradient value; The network parameters of the third classifier are adjusted based on the third gradient value; The network parameters of the fourth classifier are adjusted based on the fourth gradient value.

9. The method according to any one of claims 1-3, characterized in that, After generating the target network model based on the adjusted detection network and the first classifier, the method further includes: The image to be detected is input into the detection network in the target network model to obtain the features to be detected; The feature to be detected is input into the first classifier in the target network model to obtain the prediction vector to be detected, which includes N prediction probability values ​​corresponding to N facial motion units. Based on the N predicted probability values ​​corresponding to the N facial motion units, the facial motion units with predicted probability values ​​greater than a preset threshold are identified as the target facial motion units corresponding to the image to be detected.

10. A facial motion unit detection device, characterized in that, The device includes: The acquisition module is used to input a source domain image into a detection network to obtain source domain features, and input the source domain features into a first classifier to obtain a source domain prediction vector; input a target domain image into a detection network to obtain target domain features, and input the target domain features into a first classifier to obtain a target domain prediction vector; The determination module is used to determine a source domain distribution matrix based on the source domain prediction vector and a target domain distribution matrix based on the target domain prediction vector. The source domain distribution matrix includes the co-occurrence probability of any two facial motion units, and the target domain distribution matrix includes the co-occurrence probability of any two facial motion units. The acquisition module is further configured to input the source domain distribution matrix into a second classifier to obtain a first classification result, wherein the first classification result represents the predicted probability value of the source domain or target domain corresponding to the source domain distribution matrix; and input the target domain distribution matrix into a second classifier to obtain a second classification result, wherein the second classification result represents the predicted probability value of the source domain or target domain corresponding to the target domain distribution matrix. The determining module is further configured to determine a first loss value based on the first classification result and the second classification result, adjust the network parameters of the detection network and the first classifier based on the first loss value, and generate a target network model based on the adjusted detection network and the first classifier; The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

11. The apparatus according to claim 10, Its features are, in, The determining module determines the source domain distribution matrix based on the source domain prediction vector, and specifically determines the target domain distribution matrix based on the target domain prediction vector by using the following formulas: ;like Let be the source domain prediction vector, then Let be the matrix transpose of the source domain prediction vector, and ... Let be the source domain distribution matrix; if Let the target domain prediction vector be... Let diag be the matrix transpose of the target domain prediction vector, where diag denotes normalization of the matrix diagonal. The target domain distribution matrix; Specifically, when the determining module determines the first loss value based on the first classification result and the second classification result, it is used to determine the first loss value using the following formula: ; The first loss value, The source domain distribution matrix is... The first classification result, The target domain distribution matrix, This is the second classification result; The determining module is further configured to determine a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein the source domain prediction vector includes N predicted probability values ​​corresponding to N facial motion units, and the label vector includes N true values ​​corresponding to N facial motion units, where N is a positive integer; when the determining module adjusts the network parameters of the detection network and the first classifier based on the first loss value, it is specifically configured to: adjust the network parameters of the detection network and the first classifier based on the first loss value and the second loss value; The acquisition module is further configured to input the source domain features into a third classifier to obtain a third classification result, and input the target domain features into a third classifier to obtain a fourth classification result; the determination module is further configured to determine a third loss value based on the third classification result and the fourth classification result; when the determination module adjusts the network parameters of the detection network and the first classifier based on the first loss value and the second loss value, it is specifically configured to: adjust the network parameters of the detection network based on the first loss value, the second loss value and the third loss value, and adjust the network parameters of the first classifier based on the first loss value and the second loss value; The detection network includes a feature extraction network and a global pooling network. The acquisition module is further configured to input the stitched image into the feature extraction network to obtain stitched features. The stitched image includes K image regions, where K is a positive integer greater than 1. For each image region, the image region originates from a source domain image or a target domain image. The stitched features are input into a fourth classifier to obtain a fifth classification result corresponding to each image region in the stitched image. The determination module is further configured to determine a fourth loss value based on the fifth classification results corresponding to the K image regions. When the determination module adjusts the network parameters of the detection network based on the first loss value, the second loss value, and the third loss value, it specifically adjusts the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value, and adjusts the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value. Specifically, the determining module adjusts the network parameters of the first classifier based on the first loss value and the second loss value, adjusts the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value, and adjusts the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value. Specifically, it calculates the first gradient value corresponding to the first loss value, the second gradient value corresponding to the second loss value, the third gradient value corresponding to the third loss value, and the fourth gradient value corresponding to the fourth loss value; adjusts the network parameters of the first classifier based on the inverted gradient value of the first gradient value and the second gradient value; adjusts the network parameters of the global pooling network based on the inverted gradient value of the first gradient value, the second gradient value, and the inverted gradient value of the third gradient value; and adjusts the network parameters of the feature extraction network based on the inverted gradient value of the first gradient value, the second gradient value, the inverted gradient value of the third gradient value, and the inverted gradient value of the fourth gradient value. The determining module is further configured to adjust the network parameters of the second classifier based on the first gradient value; adjust the network parameters of the third classifier based on the third gradient value; and adjust the network parameters of the fourth classifier based on the fourth gradient value. The facial motion unit detection device further includes: a detection module, used to input the image to be detected into the detection network in the target network model to obtain the features to be detected, and input the features to be detected into the first classifier in the target network model to obtain the prediction vector to be detected, wherein the prediction vector to be detected includes N prediction probability values ​​corresponding to N facial motion units; based on the N prediction probability values ​​corresponding to the N facial motion units, facial motion units with prediction probability values ​​greater than a preset threshold are determined as the target facial motion units corresponding to the image to be detected.

12. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-9.

Images