Facial motion unit detection method, device and equipment

Abstract

The invention provides a facial motion unit detection method, device and equipment, and the method comprises the steps: inputting a source domain image to a detection network to obtain a source domain feature, and inputting the source domain feature to a first classifier to obtain a source domain prediction vector; inputting the target domain image to a detection network to obtain a target domain feature, and inputting the target domain feature to a first classifier to obtain a target domain prediction vector; determining a source domain distribution vector based on the source domain prediction vector, and determining a target domain distribution vector based on the target domain prediction vector; inputting the source domain distribution vector to a second classifier to obtain a first classification result, inputting the target domain distribution vector to the second classifier to obtain a second classification result, determining a first loss value based on the first classification result and the second classification result, and adjusting network parameters of the detection network and the first classifier based on the first loss value to obtain a detection result. And generating a target network model based on the adjusted detection network and the first classifier. According to the scheme, the training time is short, and the resource consumption is low.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a facial motion unit detection method, device and equipment. Background Art

[0002] Facial Action Unit (AU) is the basic unit of facial expression, describing the degree of activity of different parts of the face. There are 44 types of AUs, which make up all possible facial expressions. For example, AUs can be used to describe different aspects of facial expressions, such as the degree of eye opening, eyebrow raising, and mouth opening.

[0003] In order to identify facial motion units in the image to be detected, a facial motion unit detection model can be pre-trained, and the image to be detected is input into the facial motion unit detection model, and the facial motion unit detection model outputs the facial motion units in the image to be detected. In order to make the facial motion unit detection model applicable to all scenes, it is necessary to collect a data set for each scene, annotate each sample image in the data set, and then train the facial motion unit detection model based on all sample images in the data set.

[0004] However, due to the large number of actual scenes, it takes a lot of time to complete the labeling work when labeling the data set of each scene, that is, it takes a long time to train the facial motion unit detection model, the training time is long, and the training efficiency is low. A large amount of resources (such as computing resources) are required to complete the labeling work, and the resource consumption is large. The labeling cost is high and the labeling efficiency is low. Summary of the invention

[0005] The present application provides a facial motion unit detection method, the method comprising:

[0006] Inputting the source domain image into the detection network to obtain source domain features, and inputting the source domain features into the first classifier to obtain a source domain prediction vector; inputting the target domain image into the detection network to obtain target domain features, and inputting the target domain features into the first classifier to obtain a target domain prediction vector;

[0007] Determine a source domain distribution vector based on the source domain prediction vector, and determine a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes a common occurrence probability of any two facial motion units, and the target domain distribution vector includes a common occurrence probability of any two facial motion units;

[0008] Inputting the source domain distribution vector to a second classifier to obtain a first classification result, inputting the target domain distribution vector to a second classifier to obtain a second classification result, determining a first loss value based on the first classification result and the second classification result, adjusting network parameters of a detection network and a first classifier based on the first loss value, and generating a target network model based on the adjusted detection network and the first classifier;

[0009] The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

[0010] The present application provides a facial motion unit detection device, the device comprising:

[0011] An acquisition module is used to input a source domain image into a detection network to obtain source domain features, and input the source domain features into a first classifier to obtain a source domain prediction vector; input a target domain image into a detection network to obtain target domain features, and input the target domain features into a first classifier to obtain a target domain prediction vector;

[0012] A determination module, configured to determine a source domain distribution vector based on the source domain prediction vector, and determine a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes a probability of co-occurrence of any two facial motion units, and the target domain distribution vector includes a probability of co-occurrence of any two facial motion units;

[0013] The acquisition module is further used to input the source domain distribution vector to the second classifier to obtain a first classification result, and input the target domain distribution vector to the second classifier to obtain a second classification result;

[0014] The determination module is further used to determine a first loss value based on the first classification result and the second classification result, adjust network parameters of the detection network and the first classifier based on the first loss value, and generate a target network model based on the adjusted detection network and the first classifier;

[0015] The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

[0016] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the facial motion unit detection method of the above example.

[0017] It can be seen from the above technical solutions that in the embodiment of the present application, the target network model (i.e., the facial motion unit detection model) is obtained by training based on the source domain image and the target domain image, the source domain image is a labeled image, and the target domain image is an unlabeled image, that is, only the data set of some scenes (source domain data set) needs to be labeled, and the data set of the remaining scenes (target domain data set) does not need to be labeled, so that only a small amount of time is needed to complete the labeling work, that is, only a small amount of time is needed to train the facial motion unit detection model, the training time is relatively short, and the training efficiency is relatively high. Only a small amount of resources (such as computing resources) are needed to complete the labeling work, and the resource consumption is relatively small. The labeling cost is relatively low, and the labeling efficiency is relatively high. The facial motion unit detection model can be obtained by training based on the source domain distribution vector and the target domain distribution vector, and the source domain distribution vector and the target domain distribution vector can reflect the relationship invariant characteristics of the facial motion unit. The facial motion unit detection model is obtained by training based on the relationship invariant characteristics of the facial motion unit, and the accuracy of facial motion unit detection is improved, and the performance of facial motion unit recognition under the target domain is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0019] Figure 1 is a flowchart of a facial motion unit detection method in one embodiment of the present application;

[0020] Figure 2A is a schematic diagram of the structure of an initial network model in one embodiment of the present application;

[0021] Figure 2B is a schematic diagram of the training process of the initial network model in one embodiment of the present application;

[0022] Figure 3 is a schematic diagram of the structure of an initial network model in one embodiment of the present application;

[0023] Figure 4 is a schematic diagram of the structure of an initial network model in one embodiment of the present application;

[0024] Figure 5 is a schematic diagram of the structure of an initial network model in one embodiment of the present application;

[0025] Figure 6 is a structural schematic diagram of a facial motion unit detection device in one embodiment of the present application;

[0026] Figure 7 It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. The singular forms of "a", "said" and "the" used in the present application and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0028] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination".

[0029] In the embodiment of the present application, a facial motion unit detection method is proposed, which can be applied to any type of device. Figure 1 FIG. 4 is a flow chart of the method, which may include:

[0030] Step 101: input a source domain image into a detection network to obtain source domain features, and input the source domain features into a first classifier to obtain a source domain prediction vector; input a target domain image into a detection network to obtain target domain features, and input the target domain features into a first classifier to obtain a target domain prediction vector.

[0031] Step 102: determine a source domain distribution vector based on the source domain prediction vector, and determine a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0032] Step 103: input the source domain distribution vector to the second classifier to obtain a first classification result, input the target domain distribution vector to the second classifier to obtain a second classification result, determine a first loss value based on the first classification result and the second classification result, adjust the network parameters of the detection network and the first classifier based on the first loss value, and generate a target network model based on the adjusted detection network and the first classifier. The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

[0033] Exemplarily, determining the source domain distribution vector based on the source domain prediction vector and determining the target domain distribution vector based on the target domain prediction vector may include, but is not limited to: the source domain distribution vector and the target domain distribution vector may be determined using the following formula: Among them, if is the source domain prediction vector, then is the matrix transpose of the source domain prediction vector, and diag is used to indicate the normalization of the diagonal of the matrix. is the source domain distribution vector; in addition, if is the target domain prediction vector, then is the matrix transpose of the target domain prediction vector, and diag is used to indicate the normalization of the diagonal of the matrix. is the target domain distribution vector.

[0034] Exemplarily, determining the first loss value based on the first classification result and the second classification result may include, but is not limited to: determining the first loss value using the following formula: in, is the first loss value, is the source domain distribution vector, is the first classification result, which is used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or the target domain. is the target domain distribution vector, is the second classification result, which is used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

[0035] Exemplarily, after the source domain feature is input into the first classifier to obtain the source domain prediction vector, the second loss value can also be determined based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein the source domain prediction vector includes N prediction probability values ​​corresponding to N facial motion units, and the label vector includes N true values ​​corresponding to N facial motion units, and N is a positive integer. In addition, adjusting the network parameters of the detection network and the first classifier based on the first loss value may include but is not limited to: adjusting the network parameters of the detection network and the first classifier based on the first loss value and the second loss value.

[0036] Exemplarily, after the source domain image is input into the detection network to obtain the source domain features, and the target domain image is input into the detection network to obtain the target domain features, the source domain features can also be input into the third classifier to obtain the third classification result, and the target domain features can be input into the third classifier to obtain the fourth classification result, and the third loss value is determined based on the third classification result and the fourth classification result. In addition, adjusting the network parameters of the detection network and the first classifier based on the first loss value and the second loss value may include but is not limited to: adjusting the network parameters of the detection network based on the first loss value, the second loss value and the third loss value, and adjusting the network parameters of the first classifier based on the first loss value and the second loss value.

[0037] Exemplarily, the detection network may include but is not limited to a feature extraction network and a global pooling network, and the stitched image may be input into the feature extraction network to obtain a stitched feature, the stitched image may include K image regions, and K may be a positive integer greater than 1, and for each image region, the image region is derived from a source domain image or a target domain image; the stitched feature is input into a fourth classifier to obtain a fifth classification result corresponding to each image region in the stitched image, and a fourth loss value is determined based on the fifth classification results corresponding to the K image regions. In addition, adjusting the network parameters of the detection network based on the first loss value, the second loss value, and the third loss value may include but is not limited to: adjusting the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value, and adjusting the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value.

[0038] Exemplarily, adjusting the network parameters of the first classifier based on the first loss value and the second loss value, adjusting the network parameters of the global pooling network based on the first loss value, the second loss value and the third loss value, and adjusting the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value and the fourth loss value may include but are not limited to: calculating a first gradient value corresponding to the first loss value, a second gradient value corresponding to the second loss value, a third gradient value corresponding to the third loss value, and a fourth gradient value corresponding to the fourth loss value; based on this, the network parameters of the first classifier may be adjusted based on the reverse gradient value of the first gradient value and the second gradient value; the network parameters of the global pooling network are adjusted based on the reverse gradient value of the first gradient value, the second gradient value and the reverse gradient value of the third gradient value; the network parameters of the feature extraction network are adjusted based on the reverse gradient value of the first gradient value, the second gradient value, the reverse gradient value of the third gradient value and the reverse gradient value of the fourth gradient value.

[0039] Exemplarily, after obtaining the first gradient value, the third gradient value and the fourth gradient value, the network parameters of the second classifier can be adjusted based on the first gradient value; the network parameters of the third classifier can be adjusted based on the third gradient value; and the network parameters of the fourth classifier can be adjusted based on the fourth gradient value.

[0040] Exemplarily, after generating a target network model (i.e., a facial motion unit detection model) based on the adjusted detection network and the first classifier, the image to be detected can be input into the detection network in the target network model to obtain features to be detected; the features to be detected are input into the first classifier in the target network model to obtain a prediction vector to be detected, and the prediction vector to be detected includes N prediction probability values ​​corresponding to N facial motion units; based on the N prediction probability values ​​corresponding to the N facial motion units, the facial motion unit whose prediction probability value is greater than a preset threshold is determined as the target facial motion unit corresponding to the image to be detected.

[0041] It can be seen from the above technical solutions that in the embodiment of the present application, the target network model (i.e., the facial motion unit detection model) is obtained by training based on the source domain image and the target domain image, the source domain image is a labeled image, and the target domain image is an unlabeled image, that is, only the data set of some scenes (source domain data set) needs to be labeled, and the data set of the remaining scenes (target domain data set) does not need to be labeled, so that only a small amount of time is needed to complete the labeling work, that is, only a small amount of time is needed to train the facial motion unit detection model, the training time is relatively short, and the training efficiency is relatively high. Only a small amount of resources (such as computing resources) are needed to complete the labeling work, and the resource consumption is relatively small. The labeling cost is relatively low, and the labeling efficiency is relatively high. The facial motion unit detection model can be obtained by training based on the source domain distribution vector and the target domain distribution vector, and the source domain distribution vector and the target domain distribution vector can reflect the relationship invariant characteristics of the facial motion unit. The facial motion unit detection model is obtained by training based on the relationship invariant characteristics of the facial motion unit, and the accuracy of facial motion unit detection is improved, and the performance of facial motion unit recognition under the target domain is improved.

[0042] The technical solution of the embodiment of the present application is described below in conjunction with specific application scenarios.

[0043] Facial motion units are the basic units that make up facial expressions and describe the degree of activity of different parts of facial expressions. In order to identify facial motion units in the image to be detected, a facial motion unit detection model can be trained. For example, a data set for each scene is collected, and each sample image in the data set is annotated, and then the facial motion unit detection model is trained based on all the sample images in the data set.

[0044] However, due to the large number of actual scenes, it takes a lot of time to complete the labeling work when labeling the data set of each scene, that is, it takes a long time to train the facial motion unit detection model, the training time is long, and the training efficiency is low. A large amount of resources (such as computing resources) are required to complete the labeling work, and the resource consumption is large. The labeling cost is high and the labeling efficiency is low.

[0045] In view of the above findings, in an embodiment of the present application, a facial motion unit detection model is obtained based on training of source domain images and target domain images, and only the data sets of some scenes (source domain data sets) are annotated, and the data sets of the remaining scenes (target domain data sets) are not annotated, so that the annotation work can be completed in a short time, the annotation cost is relatively low, and the annotation efficiency is relatively high. Among them, when training the facial motion unit detection model, the features that are invariant in the global feature domain, the features that are invariant in the local feature domain, and the relationship-invariant characteristics of the facial motion unit are fully considered, so that the accuracy of facial motion unit detection can be improved, the performance of facial motion unit recognition in the target domain is improved, and the performance of expression analysis is improved.

[0046] Among them, based on the feature of global feature domain invariance, the global features extracted from the source domain image are required to be inseparable from the global features extracted from the target domain image. Based on the feature of local feature domain invariance, the features corresponding to the region of interest are required to be invariant in the source domain image and the target domain image. Based on the invariant characteristics of the relationship between facial motion units, it is required to fully consider the relationship between different facial motion units. For example, since facial motion units are the smallest units of facial expressions, when an expression appears, multiple facial motion units are often activated, that is, there is a positive correlation between different facial motion units. For another example, when a smiling expression appears, it is impossible for a crying expression to appear at the same time, that is, there is a negative correlation between different facial motion units, that is, when a certain facial motion unit appears, some facial motion units will not appear. In summary, it can be seen that the relationship between different facial motion units is determined by the physiological structure and is not affected by the scene. Therefore, based on the invariant characteristics of the relationship between facial motion units, the relationship between facial motion units can be fully considered, and a model training method based on the invariant characteristics of the relationship between facial motion units is proposed to improve the accuracy of facial motion unit detection.

[0047] See also Figure 2A As shown, it is a schematic diagram of the structure of the initial network model (facial motion unit detection model), the initial network model may include a feature extraction network, a global pooling network, a first classifier, a second classifier, a third classifier and a fourth classifier, and the feature extraction network and the global pooling network may constitute a detection network. For the functions of the feature extraction network, the global pooling network, the first classifier, the second classifier, the third classifier and the fourth classifier, please refer to the subsequent embodiments, which will not be repeated here.

[0048] Exemplarily, the initial network model can be trained to obtain a target network model, which may include a trained feature extraction network, a global pooling network, and a first classifier, see Figure 2B FIG. 1 is a schematic diagram of the training process of the initial network model, and the training process may include:

[0049] Step 201: Obtain a source domain dataset and a target domain dataset.

[0050] For example, data sets of multiple scenes can be obtained, and the labeled data sets are called source domain data sets, the sample images in the source domain data sets are called source domain images, the unlabeled data sets are called target domain data sets, and the sample images in the target domain data sets are called target domain images. From the above, it can be seen that only the source domain data sets of some scenes need to be labeled, and the target domain data sets of the remaining scenes do not need to be labeled.

[0051] For example, the source domain dataset can be represented as represents the kth source domain image in the source domain dataset, represents the label vector corresponding to the k-th source domain image, and the label vector may include N true values ​​corresponding to N facial motion units, where N may be a positive integer, such as 44. For the true value (i.e., label value) corresponding to each facial motion unit, if the true value corresponding to the facial motion unit is the first value (such as 1), it indicates that the facial motion unit is activated in the k-th source domain image; if the true value corresponding to the facial motion unit is the second value (such as 0), it indicates that the facial motion unit is not activated in the k-th source domain image. K s Represents the total number of source domain images in the source domain dataset.

[0052] For example, the target domain dataset can be represented as It is used to represent the kth target domain image in the target domain dataset. Obviously, there is no corresponding label vector for the target domain image in the target domain dataset. t Used to represent the total number of target domain images in the target domain dataset.

[0053] In summary, it can be seen that the source domain images of the source domain dataset are labeled data, and the target domain images of the target domain dataset are unlabeled data. The labeled data of the source domain dataset and the unlabeled data of the target domain dataset can be used to train the target network model to improve the recognition performance of facial motion units in the target domain.

[0054] Step 202: input a source domain image (such as a plurality of source domain images) in the source domain data set into a detection network to obtain a source domain feature, and input the source domain feature into a first classifier to obtain a source domain prediction vector.

[0055] Exemplarily, the detection network may include a feature extraction network and a global pooling network. The feature extraction network may be a neural network, a deep learning network, etc., and there is no restriction on the structure of the feature extraction network, as long as the feature extraction function can be realized. The global pooling network may be a neural network, a deep learning network, etc., and there is no restriction on the structure of the global pooling network, as long as the pooling function can be realized. For example, the global pooling network may be GAP (Global Average Pooling).

[0056] For example, all source domain images in the source domain dataset can be The feature extraction network extracts features from the source domain image and inputs the extracted features into the global pooling network, which performs global pooling on the extracted features to obtain the source domain image. Corresponding source domain features represents the source domain features corresponding to the first source domain image, Represents the source domain features corresponding to the second source domain image, and so on.

[0057] For example, the first classifier can be denoted as C au , the first classifier C au It is used to implement the classification function of facial motion units and output the predicted probability value of each facial motion unit. For example, after obtaining the source domain feature corresponding to the source domain image, the source domain feature can be input to the first classifier C au , by the first classifier C au Based on the source domain features, a source domain prediction vector corresponding to the source domain image is obtained. For example, assuming that there are 44 facial motion units in total, the source domain prediction vector includes 44 prediction probability values, the first prediction probability value represents the prediction probability value of the first facial motion unit appearing, the second prediction probability value represents the prediction probability value of the second facial motion unit appearing, and so on.

[0058] For example, after obtaining the source domain features Afterwards, the source domain features Input to the first classifier C au , get the source domain prediction vector The source domain prediction vector is the model's prediction of the probability of occurrence of facial motion units. is the facial motion unit prediction vector of the model for the first source domain image (which can be called the source domain prediction vector corresponding to the first source domain image), with a dimension of N, where N represents the number of facial motion units (such as 44), and so on.

[0059] Step 203: input the target domain image (such as multiple target domain images) in the target domain data set into the detection network to obtain the target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0060] For example, all target domain images in the target domain dataset can be The feature extraction network is used to extract features from the target domain image, and the extracted features are input to the global pooling network, which performs global pooling on the extracted features to obtain the target domain image. Corresponding target domain features represents the target domain features corresponding to the first target domain image, Represents the target domain features corresponding to the second target domain image, and so on.

[0061] For example, after obtaining the target domain features corresponding to the target domain image, the target domain features can be input to the first classifier C au , by the first classifier C au Based on the target domain features, processing is performed to obtain a target domain prediction vector corresponding to the target domain image, where the target domain prediction vector includes N prediction probability values.

[0062] For example, after obtaining the target domain features Afterwards, the target domain features Input to the first classifier C au , get the target domain prediction vector is the facial motion unit prediction vector of the model for the first target domain image (which can be called the target domain prediction vector corresponding to the first target domain image), with a dimension of N, where N represents the number of facial motion units, and so on.

[0063] Step 204: Determine a source domain distribution vector based on the source domain prediction vector, and determine a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0064] For example, the source domain distribution vector and the target domain prediction vector can be recorded as is an N*N matrix, For example, the value range of i is 1-N, the value range of j is 1-N, It represents the co-occurrence probability of the ith facial motion unit and the jth facial motion unit, that is, it represents the correlation between the ith facial motion unit and the jth facial motion unit. If the correlation is positive, it is 1, and if the correlation is negative, it is -1.

[0065] Obviously, if the co-occurrence probability of the i-th facial motion unit and the j-th facial motion unit is greater, The closer the value of is to 1, the smaller the probability of co-occurrence of the i-th facial motion unit and the j-th facial motion unit, then The closer the value of is to -1. If i and j are the same, then is 1.

[0066] Exemplarily, after obtaining the source domain prediction vector, the source domain distribution vector can be determined based on the source domain prediction vector. There is no limitation on the determination method, as long as the above relationship can be satisfied. After obtaining the target domain prediction vector, the target domain distribution vector can be determined based on the target domain prediction vector. There is no limitation on the determination method, as long as the above relationship can be satisfied. For example, the source domain distribution vector and the target domain distribution vector can be determined using the following formula:

[0067] For example, based on the above source domain prediction vector The source domain distribution vector can be determined using the following formula in, is the source domain prediction vector, is the matrix transpose of the source domain prediction vector, is the source domain distribution vector, and diag represents the normalization of the diagonal of the matrix.

[0068] For example, based on the target domain prediction vector The target domain distribution vector can be determined using the following formula: in, Predict the vector for the target domain, is the matrix transpose of the target domain prediction vector, is the target domain distribution vector, and diag represents the normalization of the diagonal of the matrix.

[0069] In summary, the source domain distribution vector is obtained and the target domain distribution vector is an N*N matrix, with For example, represents the co-occurrence probability of the ith facial motion unit and the jth facial motion unit, that is, the correlation between the ith facial motion unit and the jth facial motion unit. is an N*N matrix, with For example, represents the co-occurrence probability of the ith facial motion unit and the jth facial motion unit, that is, the correlation between the ith facial motion unit and the jth facial motion unit.

[0070] Exemplarily, the source domain distribution vector and the target domain distribution vector can reflect the relational invariance characteristics of facial motion units, that is, reflect the relationship between different facial motion units, such as the existence of a positive correlation between different facial motion units, or the existence of a negative correlation between different facial motion units.

[0071] Step 205: input the source domain distribution vector to the second classifier to obtain a first classification result, input the target domain distribution vector to the second classifier to obtain a second classification result, and determine a first loss value based on the first classification result and the second classification result. Exemplarily, the first classification result can be used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or the target domain, and the second classification result can be used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

[0072] For example, the second classifier can be denoted as C d , the second classifier C d It is used to realize the classification function of the source domain and the target domain, and to output the predicted probability value of the source domain or the predicted probability value of the target domain.

[0073] For example, after obtaining the source domain distribution vector corresponding to the source domain image Afterwards, the source domain distribution vector Input to the second classifier C d , by the second classifier C d Based on the source domain distribution vector Processing is performed to obtain the first classification result corresponding to the source domain image. Among them, the second classifier C d The vector can be distributed based on the source domain Determine the predicted probability value of the source domain and the predicted probability value of the target domain. If the predicted probability value of the source domain is greater than the predicted probability value of the target domain, the predicted probability value of the source domain (i.e., the first classification result) can be output. Obviously, the larger the predicted probability value of the source domain, the more accurate the prediction result. If the predicted probability value of the source domain is not greater than the predicted probability value of the target domain, the predicted probability value of the target domain (i.e., the first classification result) can be output. Obviously, the larger the predicted probability value of the target domain, the less accurate the prediction result.

[0074] After obtaining the target domain distribution vector, the target domain distribution vector is input to the second classifier C d , by the second classifier C d Based on the target domain distribution vector, the second classification result is obtained. d The predicted probability value of the source domain and the predicted probability value of the target domain are determined based on the target domain distribution vector. If the predicted probability value of the source domain is greater than the predicted probability value of the target domain, the predicted probability value of the source domain (i.e., the second classification result) is output. Obviously, the larger the predicted probability value of the source domain, the less accurate the prediction result. If the predicted probability value of the source domain is not greater than the predicted probability value of the target domain, the predicted probability value of the target domain (i.e., the second classification result) is output. Obviously, the larger the predicted probability value of the target domain, the more accurate the prediction result.

[0075] Exemplarily, after obtaining the first classification result and the second classification result, the first loss value can be determined based on the first classification result and the second classification result. For example, a first loss function is pre-configured, and there is no restriction on the first loss function, as long as the first loss function is related to the first classification result and the second classification result. An example of the first loss function can be shown in the following formula, that is, the first loss value can be determined by the following formula:

[0076] In the above formula, is the first loss value, is the source domain distribution vector, is the first classification result, and the first classification result is the second classifier C d The predicted probability value output for the source domain distribution vector. is the target domain distribution vector, is the second classification result, and the second classification result is the second classifier C d The predicted probability value output for the target domain distribution vector. Obviously, through the first loss function, it can be seen that in order to control the first loss value as small as possible, the second classifier C d Can distinguish and Coming Soon Identify as the corresponding source domain and Identify as the corresponding target domain.

[0077] For example, since The cost is very high, so during the training process, a small batch sample strategy can be adopted to calculate each batch sample There may be bias. To solve this problem, a second classifier can be used to make and The distribution of is close, that is, the predicted and They need to be inseparable to achieve the goal of making the two close. In this way, the relationship between facial motion units can be constrained to remain unchanged in the source domain and the target domain.

[0078] Step 206: determine a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein the source domain prediction vector may include N prediction probability values ​​corresponding to N facial motion units, and the label vector may include N true values ​​corresponding to the N facial motion units, where N is a positive integer.

[0079] Exemplarily, since the source domain prediction vector includes N predicted probability values ​​corresponding to N facial motion units, and the label vector includes N true values ​​corresponding to N facial motion units, the difference vector corresponding to the source domain image can be calculated, and the difference vector includes N differences corresponding to N facial motion units, the first difference is the difference between the first predicted probability value and the first true value, the second difference is the difference between the second predicted probability value and the second true value, and so on. Obviously, the smaller the difference, the more accurate the prediction result. The second loss value can be determined based on the difference vector corresponding to the source domain image. For example, the second loss function is pre-configured, and there is no restriction on the second loss function, as long as the second loss function is related to the difference vector, that is, the smaller the difference vector, the smaller the output of the second loss function (that is, the second loss value).

[0080] Step 207: Input the source domain feature to the third classifier to obtain a third classification result, input the target domain feature to the third classifier to obtain a fourth classification result, and determine a third loss value based on the third classification result and the fourth classification result. The third classification result is used to represent the predicted probability value of the source domain feature corresponding to the source domain or the target domain, and the fourth classification result is used to represent the predicted probability value of the target domain feature corresponding to the source domain or the target domain.

[0081] For example, the third classifier can be denoted as C s , the third classifier C s It is used to realize the classification function of the source domain and the target domain, and to output the predicted probability value of the source domain or the predicted probability value of the target domain. For example, after obtaining the source domain features corresponding to the source domain image After that, the source domain features are input to the third classifier C s , by the third classifier C s Based on the source domain features, the third classification result corresponding to the source domain image is obtained. The third classifier C sThe predicted probability value of the source domain and the predicted probability value of the target domain are determined based on the source domain features. If the predicted probability value of the source domain is greater than the predicted probability value of the target domain, the predicted probability value of the source domain (the third classification result) is output. If the predicted probability value of the source domain is larger, the more accurate the prediction result is. If the predicted probability value of the source domain is not larger than the predicted probability value of the target domain, the predicted probability value of the target domain (the third classification result) is output. If the predicted probability value of the target domain is larger, the less accurate the prediction result is. After obtaining the target domain features After that, the target domain features are input to the third classifier C s , by the third classifier C s Based on the target domain features, the fourth classification result is obtained. s The predicted probability value of the source domain and the predicted probability value of the target domain are determined based on the target domain features. If the predicted probability value of the source domain is greater than the predicted probability value of the target domain, the predicted probability value of the source domain (the fourth classification result) is output. If the predicted probability value of the source domain is larger, the predicted result is less accurate. If the predicted probability value of the source domain is not greater than the predicted probability value of the target domain, the predicted probability value of the target domain (the fourth classification result) is output. If the predicted probability value of the target domain is larger, the predicted result is more accurate.

[0082] Exemplarily, after obtaining the third classification result and the fourth classification result, the third loss value can be determined based on the third classification result and the fourth classification result. For example, a third loss function is pre-configured, and there is no restriction on the third loss function, as long as the third loss function is related to the third classification result and the fourth classification result. An example of the third loss function can be shown in the following formula, that is, the third loss value can be determined by the following formula:

[0083] In the above formula, is the third loss value, is the source domain feature, is the third classification result, and the third classification result is the third classifier C s The predicted probability value output for the source domain feature. is the target domain feature, This is the fourth classification result. The fourth classification result is the third classifier C s The predicted probability value output for the target domain feature. Obviously, through the third loss function, it can be seen that in order to control the third loss value as small as possible, the third classifier C s Can distinguish and Coming Soon Identify as the corresponding source domain and In summary, for the characteristics of global features that remain unchanged, the source domain features can be and target domain features Sent to the third classifier C s , let the third classifier C s Determine whether the feature comes from the source domain or the target domain, and achieve the goal of making the global feature domain inseparable through adversarial means.

[0084] Step 208: input the stitched image into the feature extraction network to obtain a stitched feature, and input the stitched feature into the fourth classifier to obtain a fifth classification result corresponding to each image area in the stitched image, and determine a fourth loss value based on the fifth classification results corresponding to the K image areas.

[0085] Exemplarily, a stitched image (multiple stitched images, one stitched image is used as an example below) can be constructed based on the source domain data set and / or the target domain data set, and the stitched image can include K image regions, where K is a positive integer greater than 1, such as 3, 5, 8, etc., and there is no limitation on this. For each image region, the image region comes from the source domain image or the target domain image. For example, a frame of image (source domain image or target domain image) is randomly selected from the source domain data set or the target domain data set, and a sub-image corresponding to the image region is intercepted from the frame image, and then the sub-image is used as the image region in the stitched image. After obtaining K image regions, the K image regions can be combined into a stitched image.

[0086] After obtaining the stitched image, the stitched image can be input into a feature extraction network, which processes the stitched image to obtain stitching features corresponding to the stitched image, that is, the feature extraction network extracts the stitching features corresponding to the stitched image and inputs the stitching features into the fourth classifier.

[0087] For example, the fourth classifier can be denoted as C r , the fourth classifier C r It is used to realize the classification function of the source domain and the target domain, and to output the predicted probability value of each image region corresponding to the source domain or the target domain. For example, after obtaining the splicing feature, the splicing feature is input to the fourth classifier C r , by the fourth classifier C r Based on the splicing features, the fifth classification result corresponding to each image region is obtained. The fifth classification result is the predicted probability value of the image region corresponding to the source domain or the target domain, that is, the fourth classifier C r Based on the splicing features, the predicted probability value of the source domain and the predicted probability value of the target domain corresponding to each image region are determined, and the predicted probability value of the source domain or the predicted probability value of the target domain is output. The implementation process refers to the third classifier and will not be repeated here.

[0088] Exemplarily, after obtaining the fifth classification result corresponding to each image area, the fourth loss value can be determined based on the fifth classification result. For example, the fourth loss function can be pre-configured, and there is no restriction on the fourth loss function, as long as the fourth loss function is related to the fifth classification result. An example of the fourth loss function can be shown in the following formula, that is, the fourth loss value can be determined by the following formula:

[0089]

[0090] In the above formula, is the fourth loss value, is the i-th splicing feature, is the fifth classification result corresponding to the first image area, which is the result of the fourth classifier C r The predicted probability value output for the first image region, is the fifth classification result corresponding to the second image area, which is the result of the fourth classifier C r The predicted probability value output for the second image region, and so on, is the fifth classification result corresponding to the Kth image region, which is the fourth classifier C r The predicted probability value output for the Kth image region. Obviously, it can be seen from the fourth loss function that in order to control the fourth loss value as small as possible, the fourth classifier C r It can distinguish whether each image region corresponds to the source domain or the target domain. In summary, for the features where local features remain unchanged, the fourth classifier C is also trained r , send the features before global pooling to the fourth classifier C r , let the fourth classifier C r Predict which domain each image region comes from, and through an adversarial approach, achieve the goal of making local feature domains inseparable.

[0091] Step 209: adjust the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value and the fourth loss value; adjust the network parameters of the global pooling network based on the first loss value, the second loss value and the third loss value; adjust the network parameters of the first classifier based on the first loss value and the second loss value; adjust the network parameters of the second classifier based on the first loss value; adjust the network parameters of the third classifier based on the third loss value; adjust the network parameters of the fourth classifier based on the fourth loss value.

[0092] Exemplarily, a first gradient value corresponding to the first loss value, a second gradient value corresponding to the second loss value, a third gradient value corresponding to the third loss value, and a fourth gradient value corresponding to the fourth loss value can be calculated. For example, the gradient descent method can be used to calculate the first gradient value, the second gradient value, the third gradient value, and the fourth gradient value, or other algorithms can be used to calculate the first gradient value, the second gradient value, the third gradient value, and the fourth gradient value. There is no limitation to this, as long as the gradient value can be calculated based on the loss value.

[0093] Then, the network parameters of the feature extraction network can be adjusted based on the reverse gradient value of the first gradient value (i.e., the gradient value after reversing the first gradient value), the second gradient value, the reverse gradient value of the third gradient value, and the reverse gradient value of the fourth gradient value, and there is no restriction on the network parameter adjustment process. The network parameters of the global pooling network can be adjusted based on the reverse gradient value of the first gradient value, the second gradient value, and the reverse gradient value of the third gradient value. The network parameters of the first classifier can be adjusted based on the reverse gradient value of the first gradient value and the second gradient value. The network parameters of the second classifier can be adjusted based on the first gradient value. The network parameters of the third classifier can be adjusted based on the third gradient value. The network parameters of the fourth classifier can be adjusted based on the fourth gradient value.

[0094] Exemplarily, the first loss value is used to control the relationship invariant characteristics of facial motion units, the third loss value is used to control the features that are invariant in the global feature domain, and the fourth loss value is used to control the features that are invariant in the local feature domain. Obviously, when the network parameters of the model are adjusted based on the first loss value, the third loss value, and the fourth loss value, the relationship invariant characteristics of the facial motion units, the features that are invariant in the global feature domain, and the features that are invariant in the local feature domain can be repeatedly considered, thereby improving the accuracy of facial motion unit detection, improving the performance of facial motion unit recognition in the target domain, and improving the performance of expression analysis.

[0095] Step 210 , determine whether the adjusted network model has converged. If not, use the adjusted network model as the initial network model and return to step 202 . If yes, execute step 211 .

[0096] For example, the prediction accuracy corresponding to the adjusted network model can be determined based on the validation data set. If the prediction accuracy is greater than a convergence threshold (configured according to experience, such as 98%), it is determined that the adjusted network model has converged. If the prediction accuracy is not greater than the convergence threshold, it is determined that the adjusted network model has not converged.

[0097] For another example, if the number of iterations of the network model is greater than a preset number threshold (which can be configured based on experience), it is determined that the adjusted network model has converged; otherwise, it is determined that the adjusted network model has not converged.

[0098] For another example, if the iteration duration of the network model is greater than a preset duration threshold (which can be configured based on experience), it is determined that the adjusted network model has converged; otherwise, it is determined that the adjusted network model has not converged.

[0099] Of course, the above are just a few examples, and this embodiment does not limit the convergence condition.

[0100] Step 211, generating a target network model based on the adjusted feature extraction network, the global pooling network and the first classifier, that is, the adjusted feature extraction network, the adjusted global pooling network and the adjusted first classifier form a target network model (i.e., a facial motion unit detection model).

[0101] See also Figure 3 As shown, another structural diagram of the initial network model (facial motion unit detection model), the initial network model may include a feature extraction network, a global pooling network, a first classifier, and a second classifier, and the feature extraction network and the global pooling network may constitute a detection network. Exemplarily, the initial network model may be trained to obtain a target network model, and the training process may include:

[0102] Step S11: Acquire a source domain dataset and a target domain dataset.

[0103] Step S12: input a source domain image (such as a plurality of source domain images) in the source domain data set into a detection network to obtain a source domain feature, and input the source domain feature into a first classifier to obtain a source domain prediction vector.

[0104] Step S13: input the target domain image (such as multiple target domain images) in the target domain data set into the detection network to obtain the target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0105] Step S14: determining a source domain distribution vector based on the source domain prediction vector, and determining a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0106] Step S15: input the source domain distribution vector to the second classifier to obtain a first classification result, input the target domain distribution vector to the second classifier to obtain a second classification result, and determine a first loss value based on the first classification result and the second classification result. Exemplarily, the first classification result can be used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or the target domain, and the second classification result can be used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

[0107] Step S16, determining a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein the source domain prediction vector may include N prediction probability values ​​corresponding to N facial motion units, and the label vector may include N true values ​​corresponding to the N facial motion units, and N is a positive integer.

[0108] Step S17: adjusting the network parameters of the feature extraction network based on the first loss value and the second loss value, adjusting the network parameters of the global pooling network based on the first loss value and the second loss value, adjusting the network parameters of the first classifier based on the first loss value and the second loss value, and adjusting the network parameters of the second classifier based on the first loss value.

[0109] Step S18, determine whether the adjusted network model has converged, if not, use the adjusted network model as the initial network model and return to step S12, if yes, execute step S19.

[0110] Step S19, generating a target network model based on the adjusted feature extraction network, the global pooling network and the first classifier, that is, the adjusted feature extraction network, the adjusted global pooling network and the adjusted first classifier form a target network model (i.e., a facial motion unit detection model).

[0111] See also Figure 4 FIG. 1 is another structural diagram of the initial network model, wherein the initial network model includes a feature extraction network, a global pooling network, a first classifier, a second classifier, and a third classifier. Exemplarily, the initial network model may be trained to obtain a target network model, and the training process may include:

[0112] Step S21: Acquire a source domain dataset and a target domain dataset.

[0113] Step S22: input a source domain image (such as a plurality of source domain images) in the source domain data set into a detection network to obtain a source domain feature, and input the source domain feature into a first classifier to obtain a source domain prediction vector.

[0114] Step S23: input the target domain image (such as multiple target domain images) in the target domain data set into the detection network to obtain the target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0115] Step S24: determining a source domain distribution vector based on the source domain prediction vector, and determining a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0116] Step S25: input the source domain distribution vector to the second classifier to obtain a first classification result, input the target domain distribution vector to the second classifier to obtain a second classification result, and determine a first loss value based on the first classification result and the second classification result. Exemplarily, the first classification result can be used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or the target domain, and the second classification result can be used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

[0117] Step S26, determining a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein the source domain prediction vector may include N prediction probability values ​​corresponding to N facial motion units, and the label vector may include N true values ​​corresponding to the N facial motion units, and N is a positive integer.

[0118] Step S27: input the source domain feature to the third classifier to obtain a third classification result, input the target domain feature to the third classifier to obtain a fourth classification result, and determine a third loss value based on the third classification result and the fourth classification result. The third classification result is used to represent the predicted probability value of the source domain feature corresponding to the source domain or the target domain, and the fourth classification result is used to represent the predicted probability value of the target domain feature corresponding to the source domain or the target domain.

[0119] Step S28, adjusting the network parameters of the feature extraction network based on the first loss value, the second loss value and the third loss value; adjusting the network parameters of the global pooling network based on the first loss value, the second loss value and the third loss value; adjusting the network parameters of the first classifier based on the first loss value and the second loss value; adjusting the network parameters of the second classifier based on the first loss value; adjusting the network parameters of the third classifier based on the third loss value.

[0120] Step S29, determine whether the adjusted network model has converged, if not, use the adjusted network model as the initial network model and return to step S22, if yes, execute step S30.

[0121] Step S30, generating a target network model based on the adjusted feature extraction network, the global pooling network and the first classifier, that is, the adjusted feature extraction network, the adjusted global pooling network and the adjusted first classifier form a target network model (i.e., a facial motion unit detection model).

[0122] See also Figure 5 As shown, another structural diagram of the initial network model is shown, and the initial network model includes a feature extraction network, a global pooling network, a first classifier, a second classifier, and a fourth classifier. Exemplarily, the initial network model can be trained to obtain a target network model, and the training process can include:

[0123] Step S31: Acquire a source domain dataset and a target domain dataset.

[0124] Step S32: input the source domain image (such as multiple source domain images) in the source domain data set into the detection network to obtain source domain features, and input the source domain features into the first classifier to obtain a source domain prediction vector.

[0125] Step S33: input the target domain image (such as multiple target domain images) in the target domain data set into the detection network to obtain the target domain features, and input the target domain features into the first classifier to obtain the target domain prediction vector.

[0126] Step S34: determining a source domain distribution vector based on the source domain prediction vector, and determining a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes the co-occurrence probability of any two facial motion units, and the target domain distribution vector includes the co-occurrence probability of any two facial motion units.

[0127] Step S35: input the source domain distribution vector to the second classifier to obtain a first classification result, input the target domain distribution vector to the second classifier to obtain a second classification result, and determine a first loss value based on the first classification result and the second classification result. Exemplarily, the first classification result can be used to represent the predicted probability value of the source domain distribution vector corresponding to the source domain or the target domain, and the second classification result can be used to represent the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

[0128] Step S36, determining a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein the source domain prediction vector may include N prediction probability values ​​corresponding to N facial motion units, and the label vector may include N true values ​​corresponding to the N facial motion units, and N is a positive integer.

[0129] Step S37, input the stitched image into the feature extraction network to obtain the stitched feature, and input the stitched feature into the fourth classifier to obtain the fifth classification result corresponding to each image area in the stitched image, and determine the fourth loss value based on the fifth classification results corresponding to the K image areas.

[0130] Step S38, adjusting the network parameters of the feature extraction network based on the first loss value, the second loss value and the fourth loss value, adjusting the network parameters of the global pooling network based on the first loss value and the second loss value, adjusting the network parameters of the first classifier based on the first loss value and the second loss value, adjusting the network parameters of the second classifier based on the first loss value, and adjusting the network parameters of the fourth classifier based on the fourth loss value.

[0131] Step S39, determine whether the adjusted network model has converged, if not, use the adjusted network model as the initial network model and return to step S32, if yes, execute step S40.

[0132] Step S40, generating a target network model based on the adjusted feature extraction network, the global pooling network and the first classifier, that is, the adjusted feature extraction network, the adjusted global pooling network and the adjusted first classifier form a target network model (i.e., a facial motion unit detection model).

[0133] In a possible implementation, after the target network model is obtained through training, the target facial motion unit corresponding to the image to be detected can be determined based on the target network model. For example, the image to be detected can be input into the detection network (such as feature extraction network and global pooling network) in the target network model to obtain the features to be detected, and then the features to be detected can be input into the first classifier in the target network model to obtain the prediction vector to be detected. For the processing of the detection network and the first classifier, refer to the above embodiment.

[0134] Exemplarily, the target network model can be applied to fields involving facial expression analysis, such as polygraphs, car assisted driving, human-computer interaction, virtual reality, etc., without limitation, and the images to be detected in these scenarios can be used and input into the target network model to obtain the prediction vector to be detected. It should be noted that all types of data involved in this specification are obtained and used with the knowledge and authorization of the relevant users.

[0135] Exemplarily, the prediction vector to be detected may include N prediction probability values ​​corresponding to N facial motion units. Based on the N prediction probability values ​​corresponding to the N facial motion units, the facial motion units whose prediction probability values ​​are greater than a preset threshold value may be determined as target facial motion units corresponding to the image to be detected. For example, if the prediction probability value corresponding to facial motion unit 1 is greater than the preset threshold value, facial motion unit 1 is determined as the target facial motion unit corresponding to the image to be detected; if the prediction probability value corresponding to facial motion unit 2 is not greater than the preset threshold value, facial motion unit 2 is not used as the target facial motion unit, and so on.

[0136] It can be seen from the above technical solutions that in the embodiments of the present application, a facial motion unit detection model can be obtained based on source domain distribution vectors and target domain distribution vectors, and the source domain distribution vectors and target domain distribution vectors can reflect the relational invariance characteristics of facial motion units. The facial motion unit detection model is obtained based on the relational invariance characteristics of facial motion units, thereby improving the accuracy of facial motion unit detection and improving the performance of facial motion unit recognition in the target domain. In this embodiment, a multi-scale unsupervised domain adaptive facial motion unit detection method is proposed, which constrains feature learning through three scales, namely, facial motion unit relational invariance, global feature invariance, and local feature invariance, thereby improving the performance of the facial motion unit detection model in the target domain without labeled samples, and can save a lot of labeling costs.

[0137] Based on the same application concept as the above method, a facial motion unit detection device is proposed in the embodiment of the present application, see Figure 6 FIG. 1 is a schematic diagram of the structure of the device, and the device may include:

[0138] The acquisition module 61 is used to input the source domain image into the detection network to obtain source domain features, and input the source domain features into the first classifier to obtain a source domain prediction vector; input the target domain image into the detection network to obtain target domain features, and input the target domain features into the first classifier to obtain a target domain prediction vector;

[0139] A determination module 62, configured to determine a source domain distribution vector based on the source domain prediction vector, and determine a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes a probability of co-occurrence of any two facial motion units, and the target domain distribution vector includes a probability of co-occurrence of any two facial motion units;

[0140] The acquisition module 61 is further used to input the source domain distribution vector to the second classifier to obtain a first classification result, and input the target domain distribution vector to the second classifier to obtain a second classification result;

[0141] The determination module 62 is further configured to determine a first loss value based on the first classification result and the second classification result, adjust network parameters of the detection network and the first classifier based on the first loss value, and generate a target network model based on the adjusted detection network and the first classifier;

[0142] The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

[0143] Exemplarily, the determination module 62 determines the source domain distribution vector based on the source domain prediction vector, and determines the target domain distribution vector based on the target domain prediction vector by: using the following formula to determine the source domain distribution vector and the target domain distribution vector: like is the source domain prediction vector, then is the matrix transpose of the source domain prediction vector, diag represents normalization of the diagonal of the matrix, is the source domain distribution vector; if is the target domain prediction vector, then is the matrix transpose of the target domain prediction vector, diag represents the normalization of the diagonal of the matrix, is the target domain distribution vector.

[0144] Exemplarily, when the determination module 62 determines the first loss value based on the first classification result and the second classification result, it is specifically configured to: determine the first loss value using the following formula: is the first loss value, is the source domain distribution vector, is the first classification result, which indicates the predicted probability value of the source domain distribution vector corresponding to the source domain or the target domain. is the target domain distribution vector, is the second classification result, and the second classification result represents the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

[0145] Exemplarily, the determination module 62 is further used to determine a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein the source domain prediction vector includes N prediction probability values ​​corresponding to N facial motion units, and the label vector includes N real values ​​corresponding to N facial motion units, and N is a positive integer; when the determination module 62 adjusts the network parameters of the detection network and the first classifier based on the first loss value, it is specifically used to: adjust the network parameters of the detection network and the first classifier based on the first loss value and the second loss value.

[0146] Exemplarily, the acquisition module 61 is further used to input the source domain feature to a third classifier to obtain a third classification result, and input the target domain feature to a third classifier to obtain a fourth classification result; the determination module 62 is further used to determine a third loss value based on the third classification result and the fourth classification result; when the determination module 62 adjusts the network parameters of the detection network and the first classifier based on the first loss value and the second loss value, it is specifically used to: adjust the network parameters of the detection network based on the first loss value, the second loss value and the third loss value, and adjust the network parameters of the first classifier based on the first loss value and the second loss value.

[0147] Exemplarily, the detection network includes a feature extraction network and a global pooling network, and the acquisition module 61 is further used to input the stitched image into the feature extraction network to obtain a stitched feature, the stitched image includes K image regions, K is a positive integer greater than 1, and for each image region, the image region is derived from a source domain image or a target domain image; the stitched feature is input into a fourth classifier to obtain a fifth classification result corresponding to each image region in the stitched image; the determination module 62 is further used to determine a fourth loss value based on the fifth classification results corresponding to the K image regions; when the determination module 62 adjusts the network parameters of the detection network based on the first loss value, the second loss value, and the third loss value, it is specifically used to: adjust the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value, and adjust the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value.

[0148] Exemplarily, the determination module 62 adjusts the network parameters of the first classifier based on the first loss value and the second loss value, adjusts the network parameters of the global pooling network based on the first loss value, the second loss value and the third loss value, and adjusts the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value and the fourth loss value, and is specifically used to: calculate a first gradient value corresponding to the first loss value, a second gradient value corresponding to the second loss value, a third gradient value corresponding to the third loss value, and a fourth gradient value corresponding to the fourth loss value; adjust the network parameters of the first classifier based on the reverse gradient value of the first gradient value and the second gradient value; adjust the network parameters of the global pooling network based on the reverse gradient value of the first gradient value, the second gradient value, and the reverse gradient value of the third gradient value; adjust the network parameters of the feature extraction network based on the reverse gradient value of the first gradient value, the second gradient value, the reverse gradient value of the third gradient value, and the reverse gradient value of the fourth gradient value.

[0149] The determination module 62 is further configured to adjust the network parameters of the second classifier based on the first gradient value; adjust the network parameters of the third classifier based on the third gradient value; and adjust the network parameters of the fourth classifier based on the fourth gradient value.

[0150] Among them, the facial motion unit detection device also includes: a detection module, used to input the image to be detected into the detection network in the target network model to obtain the features to be detected, and input the features to be detected into the first classifier in the target network model to obtain the prediction vector to be detected, wherein the prediction vector to be detected includes N prediction probability values ​​corresponding to N facial motion units; based on the N prediction probability values ​​corresponding to the N facial motion units, the facial motion unit whose prediction probability value is greater than a preset threshold is determined as the target facial motion unit corresponding to the image to be detected.

[0151] Based on the same application concept as the above method, an electronic device is proposed in the embodiment of the present application, see Figure 7 As shown, the electronic device includes: a processor 71 and a machine-readable storage medium 72, the machine-readable storage medium 72 stores machine-executable instructions that can be executed by the processor 71; the processor 71 is used to execute the machine-executable instructions to implement the facial motion unit detection method disclosed in the above example of this application.

[0152] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the facial motion unit detection method disclosed in the above example of the present application can be implemented.

[0153] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0154] The systems, devices, modules or units described in the above embodiments may be implemented by a computer entity or by a product having a certain function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0155] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] Moreover, these computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0160] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A facial motion unit detection method, characterized in that: The method comprises: Inputting the source domain image into the detection network to obtain source domain features, and inputting the source domain features into the first classifier to obtain a source domain prediction vector; inputting the target domain image into the detection network to obtain target domain features, and inputting the target domain features into the first classifier to obtain a target domain prediction vector; Determine a source domain distribution vector based on the source domain prediction vector, and determine a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes a common occurrence probability of any two facial motion units, and the target domain distribution vector includes a common occurrence probability of any two facial motion units; Inputting the source domain distribution vector to a second classifier to obtain a first classification result, inputting the target domain distribution vector to a second classifier to obtain a second classification result, determining a first loss value based on the first classification result and the second classification result, adjusting network parameters of a detection network and a first classifier based on the first loss value, and generating a target network model based on the adjusted detection network and the first classifier; The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

2. The method according to claim 1, characterized in that The determining of a source domain distribution vector based on the source domain prediction vector and determining a target domain distribution vector based on the target domain prediction vector includes: The source domain distribution vector and the target domain distribution vector are determined using the following formula: Among them, if is the source domain prediction vector, then is the matrix transpose of the source domain prediction vector, diag represents normalization of the diagonal of the matrix, is the source domain distribution vector; like is the target domain prediction vector, then is the matrix transpose of the target domain prediction vector, diag represents the normalization of the diagonal of the matrix, is the target domain distribution vector.

3. The method according to claim 1, characterized in that Determining the first loss value based on the first classification result and the second classification result includes: determining the first loss value using the following formula: is the first loss value, is the source domain distribution vector, is the first classification result, which indicates the predicted probability value of the source domain distribution vector corresponding to the source domain or the target domain. is the target domain distribution vector, is the second classification result, and the second classification result represents the predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain.

4. The method according to any one of claims 1 to 3, characterized in that: After the source domain feature is input to the first classifier to obtain the source domain prediction vector, the method further includes: determining a second loss value based on the source domain prediction vector and a label vector corresponding to the source domain image; wherein the source domain prediction vector includes N prediction probability values ​​corresponding to N facial motion units, and the label vector includes N true values ​​corresponding to the N facial motion units, where N is a positive integer; The adjusting the network parameters of the detection network and the first classifier based on the first loss value includes: adjusting the network parameters of the detection network and the first classifier based on the first loss value and the second loss value.

5. The method according to claim 4, characterized in that After the source domain image is input into the detection network to obtain the source domain feature, and the target domain image is input into the detection network to obtain the target domain feature, the method further includes: inputting the source domain feature into a third classifier to obtain a third classification result, inputting the target domain feature into a third classifier to obtain a fourth classification result, and determining a third loss value based on the third classification result and the fourth classification result; The network parameters of the detection network and the first classifier are adjusted based on the first loss value and the second loss value, including: adjusting the network parameters of the detection network based on the first loss value, the second loss value and the third loss value, and adjusting the network parameters of the first classifier based on the first loss value and the second loss value.

6. The method according to claim 5, characterized in that The detection network includes a feature extraction network and a global pooling network, and the method further includes: inputting a spliced ​​image into the feature extraction network to obtain a spliced ​​feature, wherein the spliced ​​image includes K image regions, and for each image region, the image region is derived from a source domain image or a target domain image; inputting the spliced ​​feature into a fourth classifier to obtain a fifth classification result corresponding to each image region in the spliced ​​image, and determining a fourth loss value based on the fifth classification results corresponding to the K image regions; The adjusting of the network parameters of the detection network based on the first loss value, the second loss value and the third loss value includes: adjusting the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value and the fourth loss value, and adjusting the network parameters of the global pooling network based on the first loss value, the second loss value and the third loss value.

7. The method according to claim 6, characterized in that The method further comprises: adjusting the network parameters of the first classifier based on the first loss value and the second loss value, adjusting the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value, and adjusting the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value, including: Calculate a first gradient value corresponding to the first loss value, a second gradient value corresponding to the second loss value, a third gradient value corresponding to the third loss value, and a fourth gradient value corresponding to the fourth loss value; Based on the reverse gradient value of the first gradient value and the second gradient value, the network parameters of the first classifier are adjusted; based on the reverse gradient value of the first gradient value, the second gradient value, and the reverse gradient value of the third gradient value, the network parameters of the global pooling network are adjusted; based on the reverse gradient value of the first gradient value, the second gradient value, the reverse gradient value of the third gradient value, and the reverse gradient value of the fourth gradient value, the network parameters of the feature extraction network are adjusted.

8. The method according to claim 7, characterized in that After calculating the first gradient value corresponding to the first loss value, the second gradient value corresponding to the second loss value, the third gradient value corresponding to the third loss value, and the fourth gradient value corresponding to the fourth loss value, the method further includes: Adjusting a network parameter of the second classifier based on the first gradient value; Adjusting a network parameter of the third classifier based on the third gradient value; A network parameter of the fourth classifier is adjusted based on the fourth gradient value.

9. The method according to any one of claims 1 to 3, characterized in that: After generating the target network model based on the adjusted detection network and the first classifier, the method further includes: Inputting the image to be detected into the detection network in the target network model to obtain the features to be detected; Inputting the feature to be detected into the first classifier in the target network model to obtain a prediction vector to be detected, wherein the prediction vector to be detected includes N prediction probability values ​​corresponding to N facial motion units; Based on the N predicted probability values ​​corresponding to the N facial motion units, a facial motion unit whose predicted probability value is greater than a preset threshold is determined as a target facial motion unit corresponding to the image to be detected.

10. A facial motion unit detection device, characterized in that: The device comprises: An acquisition module is used to input a source domain image into a detection network to obtain source domain features, and input the source domain features into a first classifier to obtain a source domain prediction vector; input a target domain image into a detection network to obtain target domain features, and input the target domain features into a first classifier to obtain a target domain prediction vector; A determination module, configured to determine a source domain distribution vector based on the source domain prediction vector, and determine a target domain distribution vector based on the target domain prediction vector, wherein the source domain distribution vector includes a probability of co-occurrence of any two facial motion units, and the target domain distribution vector includes a probability of co-occurrence of any two facial motion units; The acquisition module is further used to input the source domain distribution vector to the second classifier to obtain a first classification result, and input the target domain distribution vector to the second classifier to obtain a second classification result; The determination module is further used to determine a first loss value based on the first classification result and the second classification result, adjust network parameters of the detection network and the first classifier based on the first loss value, and generate a target network model based on the adjusted detection network and the first classifier; The target network model is used to determine the target facial motion unit corresponding to the image to be detected.

11. The device according to claim 10, It is characterized in that in, The determination module determines the source domain distribution vector based on the source domain prediction vector, and determines the target domain distribution vector based on the target domain prediction vector, and is specifically used to: determine the source domain distribution vector and the target domain distribution vector using the following formula: like is the source domain prediction vector, then is the matrix transpose of the source domain prediction vector, diag represents normalization of the diagonal of the matrix, is the source domain distribution vector; if is the target domain prediction vector, then is the matrix transpose of the target domain prediction vector, diag represents the normalization of the diagonal of the matrix, A distribution vector for the target domain; Wherein, when the determination module determines the first loss value based on the first classification result and the second classification result, it is specifically used to: determine the first loss value using the following formula: is the first loss value, is the source domain distribution vector, is the first classification result, which indicates the predicted probability value of the source domain distribution vector corresponding to the source domain or the target domain. is the target domain distribution vector, is the second classification result, where the second classification result represents a predicted probability value of the target domain distribution vector corresponding to the source domain or the target domain; Wherein, the determination module is further used to determine a second loss value based on the source domain prediction vector and the label vector corresponding to the source domain image; wherein the source domain prediction vector includes N prediction probability values ​​corresponding to N facial motion units, and the label vector includes N real values ​​corresponding to N facial motion units, and N is a positive integer; when the determination module adjusts the network parameters of the detection network and the first classifier based on the first loss value, it is specifically used to: adjust the network parameters of the detection network and the first classifier based on the first loss value and the second loss value; Wherein, the acquisition module is further used to input the source domain feature into the third classifier to obtain a third classification result, and input the target domain feature into the third classifier to obtain a fourth classification result; the determination module is further used to determine a third loss value based on the third classification result and the fourth classification result; when the determination module adjusts the network parameters of the detection network and the first classifier based on the first loss value and the second loss value, it is specifically used to: adjust the network parameters of the detection network based on the first loss value, the second loss value and the third loss value, and adjust the network parameters of the first classifier based on the first loss value and the second loss value; Wherein, the detection network includes a feature extraction network and a global pooling network, and the acquisition module is further used to input the spliced ​​image into the feature extraction network to obtain a spliced ​​feature, the spliced ​​image includes K image regions, K is a positive integer greater than 1, and for each image region, the image region is derived from a source domain image or a target domain image; the spliced ​​feature is input into a fourth classifier to obtain a fifth classification result corresponding to each image region in the spliced ​​image; the determination module is further used to determine a fourth loss value based on the fifth classification results corresponding to the K image regions; when the determination module adjusts the network parameters of the detection network based on the first loss value, the second loss value, and the third loss value, it is specifically used to: adjust the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value, and the fourth loss value, and adjust the network parameters of the global pooling network based on the first loss value, the second loss value, and the third loss value; Wherein, the determination module adjusts the network parameters of the first classifier based on the first loss value and the second loss value, adjusts the network parameters of the global pooling network based on the first loss value, the second loss value and the third loss value, and adjusts the network parameters of the feature extraction network based on the first loss value, the second loss value, the third loss value and the fourth loss value, and is specifically used to: calculate a first gradient value corresponding to the first loss value, a second gradient value corresponding to the second loss value, a third gradient value corresponding to the third loss value, and a fourth gradient value corresponding to the fourth loss value; adjusts the network parameters of the first classifier based on the reverse gradient value of the first gradient value and the second gradient value; adjusts the network parameters of the global pooling network based on the reverse gradient value of the first gradient value, the second gradient value and the reverse gradient value of the third gradient value; adjusts the network parameters of the feature extraction network based on the reverse gradient value of the first gradient value, the second gradient value, the reverse gradient value of the third gradient value and the reverse gradient value of the fourth gradient value; The determining module is further configured to adjust the network parameters of the second classifier based on the first gradient value; adjust the network parameters of the third classifier based on the third gradient value; and adjust the network parameters of the fourth classifier based on the fourth gradient value; Among them, the facial motion unit detection device also includes: a detection module, used to input the image to be detected into the detection network in the target network model to obtain the features to be detected, and input the features to be detected into the first classifier in the target network model to obtain the prediction vector to be detected, wherein the prediction vector to be detected includes N prediction probability values ​​corresponding to N facial motion units; based on the N prediction probability values ​​corresponding to the N facial motion units, the facial motion unit whose prediction probability value is greater than a preset threshold is determined as the target facial motion unit corresponding to the image to be detected.

12. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-9.

Images