Facial action recognition and model training method, device, equipment and storage medium

Abstract

The present application provides a method, apparatus, device and storage medium for facial action recognition and model training, wherein the model training method includes: obtaining multiple video segment samples; constructing at least one homologous positive sample pair and at least one homologous negative sample pair for each video segment sample; constructing at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for multiple video segment samples; based on the face image samples, homologous positive sample pairs, homologous negative sample pairs, non-homologous positive sample pairs and non-homologous negative sample pairs in the multiple video segment samples, a facial action feature model is trained using a self-supervised learning method to obtain an initially trained facial action feature model; using multiple facial image samples labeled with facial action categories, the facial action feature model and the facial action classifier are trained to obtain a facial action recognition model. The scheme of the present application can train a facial action recognition model that can accurately recognize facial actions of human faces.

Description

Technical Field

[0001] The present application relates to the technical field of facial expression recognition, and more specifically, to a method, device, equipment and storage medium for facial action recognition and model training. Background Art

[0002] With the continuous development of computer technology and artificial intelligence technology, the application areas of facial expression recognition technology are increasing.

[0003] Facial expression recognition technology can analyze the facial expression state of a face from a face image. However, due to the subjectivity of emotions themselves and the similarities between different emotions, it is difficult to accurately identify facial expressions in face images. In order to improve the accuracy of facial expression recognition, facial movements in face images can be combined to assist facial expression recognition.

[0004] It can be seen that accurately identifying facial movements in face images is the key to improving facial expression recognition. Therefore, how to construct a facial movement recognition model that can accurately identify facial movements is a technical problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of the above problems, this application provides a method, apparatus, device and storage medium for facial action recognition and model training to build a facial action recognition model that can accurately recognize facial actions of human faces. The specific scheme is as follows:

[0006] In a first aspect of the present application, a method for training a facial action recognition model is provided, comprising:

[0007] Obtaining a plurality of video segment samples, each of the video segment samples comprising a plurality of face image samples belonging to the same user, and the face image samples in the plurality of video segment samples do not all belong to the same user;

[0008] For each video segment sample, construct at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample, wherein the homologous positive sample pair includes two human face image samples belonging to the same video segment sample and having similar expression intensities, and the homologous negative sample pair includes two human face image samples belonging to the same video segment sample and having an expression intensity difference exceeding a first intensity difference;

[0009] Constructing at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples, the non-homologous positive sample pair comprising two human face image samples belonging to different video segment samples and having similar expression intensities, and the non-homologous negative sample pair comprising two human face image samples belonging to different video segment samples and having expression intensity difference exceeding a second intensity difference;

[0010] Based on the face image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs, a facial action feature model is trained using a self-supervised learning method to obtain an initially trained facial action feature model;

[0011] The initially trained facial action feature model and the facial action classifier to be trained are trained using a plurality of facial image samples labeled with facial action categories to obtain a facial action recognition model including the trained facial action feature model and the facial action classifier.

[0012] In a possible implementation, before adopting the self-supervised learning method to train the facial action feature model, the method further includes:

[0013] For each face image sample in each video segment sample, perform key point detection on the face image sample to obtain a key point distribution of at least one face part in the face image sample;

[0014] For each video segment sample, determining the face image sample with the lowest expression intensity in the video segment sample as the face template image in the video segment sample;

[0015] For each face part in each video segment sample, based on the key point distribution of the face part in each face image sample in the video segment sample, at least one local positive sample pair and at least one local negative sample pair for the face part in the video segment sample are constructed, wherein the local positive sample pair includes: a face template image in the video segment sample, and a face image sample in which the key point distribution of the face part in the video segment sample is similar to the key point distribution of the face template image; the local negative sample pair includes: the face template image in the video segment sample, and a face image sample in at least one face image sample in which the key point distribution of the face part in the video segment sample is significantly different from the key point distribution of the face template image;

[0016] The method of training a facial action feature model using a self-supervised learning method based on the face image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs comprises:

[0017] Based on the face image samples in the video segment samples, the homologous positive sample pairs, homologous negative sample pairs, non-homologous positive sample pairs, non-homologous negative sample pairs, and local positive sample pairs and local negative sample pairs corresponding to each face part in the video segment samples, a facial action feature model is trained using a self-supervised learning method.

[0018] On the other hand, the present application also provides a method for recognizing facial movements of a human face, comprising:

[0019] Obtain a face image to be recognized;

[0020] Facial actions in the face image are identified using a pre-trained facial action recognition model, wherein the facial action recognition model is obtained using the training method for the facial action recognition model described in any one of the embodiments of the present application.

[0021] On the other hand, the present application also provides a training device for a facial action recognition model, comprising:

[0022] A video segment obtaining unit, configured to obtain a plurality of video segment samples, each of the video segment samples comprising a plurality of face image samples belonging to the same user, and the face image samples in the plurality of video segment samples do not completely belong to the same user;

[0023] A first sample construction unit is used to construct, for each video segment sample, at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample, wherein the homologous positive sample pair includes two human face image samples belonging to the same video segment sample and having similar expression intensities, and the homologous negative sample pair includes two human face image samples belonging to the same video segment sample and having an expression intensity difference exceeding a first intensity difference;

[0024] A second sample construction unit is used to construct at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples, wherein the non-homologous positive sample pair includes two human face image samples belonging to different video segment samples and having similar expression intensities, and the non-homologous negative sample pair includes two human face image samples belonging to different video segment samples and having an expression intensity difference exceeding a second intensity difference;

[0025] A self-supervised learning unit is used to train a facial action feature model using a self-supervised learning method based on the facial image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs, to obtain an initially trained facial action feature model;

[0026] The transfer learning unit is used to train the initially trained facial action feature model and the facial action classifier to be trained using a plurality of facial image samples labeled with facial action categories, so as to obtain a facial action recognition model including the trained facial action feature model and the facial action classifier.

[0027] On the other hand, the present application also provides a facial action recognition device, comprising:

[0028] An image acquisition unit, used for acquiring a face image to be recognized;

[0029] A facial action recognition unit is used to recognize facial actions in the face image using a pre-trained facial action recognition model, wherein the facial action recognition model is obtained using the training method for the facial action recognition model described in any one of the present applications.

[0030] In another aspect, the present application also provides a computer device, comprising: a memory and a processor;

[0031] The memory is used to store programs;

[0032] The processor is used to execute the program to implement each step of the training method of the facial action recognition model as described in any one of the embodiments of the present application.

[0033] On the other hand, the present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the various steps of the training method of the facial action recognition model described in any one of the embodiments of the present application are implemented.

[0034] By means of the above technical solution, after obtaining multiple video segment samples, the present application combines the expression intensity of each face image sample in each video segment sample, not only will select face images from the same video segment sample to construct positive sample pairs and negative sample pairs, but also will construct positive sample pairs and negative sample pairs containing face images from different video segment samples. Since the face image samples in the same video segment sample belong to the same user, while the face image samples in different video segment samples may belong to different users, the positive sample pairs and negative sample pairs derived from the same video segment sample reflect the relationship between the face actions of the same user, while the positive sample pairs and negative samples derived from two different video segment samples may reflect the relationship between the face actions of different users, so that in the process of training the facial action feature model, the present application not only considers the facial action relationship between different face images of the same user, but also considers the facial action relationship between face images of different users, so that the trained facial action feature model can more accurately extract the facial action features in the face image, and then the facial action recognition model finally trained can accurately recognize the facial actions of the face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0036] Figure 1 A flowchart of a method for training a facial action recognition model provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of another flow chart of a method for training a facial action recognition model provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a process for obtaining a video segment sample provided in an embodiment of the present application;

[0039] Figure 4 A flowchart of a training method for a facial action recognition model provided in an embodiment of the present application in an application example;

[0040] Figure 5 A schematic diagram of a flow chart of a method for recognizing facial movements of a human face provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a training device for a facial action recognition model provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of the structure of a facial action recognition device provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of the composition architecture of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] In order to train a facial action recognition model that can accurately identify facial action features, the inventors of this application have found through research that since there is less labeled data for facial actions, if a supervised training method is used to train the facial action recognition model, the accuracy of the trained action recognition model may be low. If facial actions of face images are manually labeled on a large scale, it will be extremely time-consuming and laborious. Based on this, in order to avoid the need for manual large-scale facial action labeling of face images, the inventors thought of using a self-supervised method to train the facial action recognition model.

[0046] However, the inventors further studied that the similarity between facial image frames with a close time distance in a video segment of the same user is higher than the similarity between facial image frames with a long time difference. Based on this, facial images with adjacent timestamps can be selected from the same video segment to construct positive sample pairs, and facial images with long timestamps can be selected to construct negative sample pairs, and self-supervised learning of facial action recognition models can be performed based on the positive and negative sample pairs. However, this self-supervised learning method only considers the facial action relationship between different facial images of the same user in the same video segment, which has great limitations and is not conducive to training a facial action recognition model with high accuracy.

[0047] Based on the above research, the present application combines the expression intensity of facial images in different video segments of different users to construct positive and negative sample pairs that can come from the same video segment and different video segments, respectively, so that the constructed multiple positive and negative sample pairs can reflect the facial action relationship between different facial images of the same user, and the facial action relationship between facial images of different users, thereby training a more sophisticated facial action recognition model.

[0048] The following is an introduction to the training method of the facial action recognition model of the present application in conjunction with a flowchart.

[0049] like Figure 1 As shown, it shows a flow chart of the training method of the facial action recognition model provided in an embodiment of the present application. The method of this embodiment can be applied to various computer devices, such as personal computers or servers, etc., without limitation.

[0050] The method of this embodiment may include:

[0051] S101, obtaining multiple video segment samples.

[0052] Each video segment sample includes multiple face image samples belonging to the same user, and the face image samples in the multiple video segment samples do not completely belong to the same user.

[0053] S102: For each video segment sample, construct at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample.

[0054] Among them, the homologous positive sample pair includes two face image samples belonging to the same video segment sample and having similar expression intensity, and the homologous negative sample pair includes two face image samples belonging to the same video segment sample and having an expression intensity difference exceeding the first intensity difference.

[0055] Among them, for the sake of easy distinction, this application will refer to the positive sample pairs consisting of face image samples from the same video segment sample as homologous positive sample pairs, and the negative sample pairs consisting of face image samples from the same video segment sample as homologous negative sample pairs; similarly, the positive sample pairs and negative sample pairs consisting of face image samples from different video segment samples will be referred to as non-homologous positive sample pairs and non-homologous negative sample pairs, respectively.

[0056] It is understandable that facial movements are associated with facial expression intensity. Based on this, in the embodiment of the present application, for each video segment sample, the expression intensity (also referred to as the intensity of facial expression) of each face image sample in the video segment sample can be determined. For example, the expression intensity of the face image sample can be identified using a pre-trained expression intensity recognition model. Accordingly, the expression intensity of each face image sample in the video segment sample can be combined to construct a positive sample pair with similar expression intensity and a negative sample pair with a large difference in expression intensity.

[0057] The similarity of the expression intensity of the two face image samples can be: the difference of the expression intensity of the two face image samples is less than a set threshold, or the expression intensity of the two face image samples belongs to the same intensity range, etc., which can be set as needed. Similarly, the difference of the expression intensity of the two face image samples exceeding the first intensity difference can be that the difference of the expression intensity of the two face image samples exceeds the set difference, or the expression intensity of the two face image samples belongs to two different expression intensity ranges, and the difference of the two expression intensity ranges exceeds the set difference, etc.

[0058] For example, in one possible implementation, for each video segment sample, multiple first face image samples with expression intensity lower than a first intensity threshold and multiple second face image samples with expression intensity higher than a second intensity threshold in the video sample can be determined.

[0059] Accordingly, for each video segment sample, at least one homologous positive sample pair and at least one homologous negative sample pair can be constructed based on multiple first face image samples and multiple second face image samples. The homologous positive sample pair includes: two first face image samples or two second face image samples belonging to the same video segment sample. The homologous negative sample pair includes: a first face image sample and a second face image sample belonging to the same video segment sample.

[0060] The second intensity threshold is greater than the first intensity threshold. The first intensity threshold and the second intensity threshold can be set according to the value range of the expression intensity and actual needs, and there is no restriction on this.

[0061] For example, expression intensity can be divided into four categories: no expression, slight expression, strong expression and severe expression. The facial image samples with expression intensity lower than the first intensity threshold can be facial image samples belonging to the slight expression category, and the facial image samples with expression intensity higher than the second intensity threshold can be facial image samples belonging to the strong expression category. For example, taking the expression intensity value as a value in the interval [0,3] as an example, the first intensity threshold can be 1, and the second intensity threshold can be 2.

[0062] In the present application, for a video segment sample, in order to avoid facial image samples with similar expression intensity being constructed as negative sample pairs, the present application actually selects facial image samples of paired negative sample pairs from facial image samples of the slight expression category and facial image samples of the relatively expressive category respectively.

[0063] It can be understood that since the same video segment sample contains face image samples belonging to the same user, the positive sample pairs and negative sample pairs consisting of face image samples from the same video segment sample can reflect the facial action relationship between different face image samples of the same user.

[0064] S103: construct at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples.

[0065] The non-homologous positive sample pair includes two face image samples belonging to different video segment samples and having similar expression intensities. The non-homologous negative sample pair includes two face image samples belonging to different video segment samples and having expression intensity differences exceeding the second intensity difference.

[0066] Here, the meaning of similar expression intensity may be similar to that of the previous step S102. Similarly, the meaning of the expression intensity difference exceeding the second intensity difference is also similar to the meaning of the expression intensity difference exceeding the first intensity difference. The second intensity difference may be different from the first intensity difference or may be the same as the first intensity difference, without limitation.

[0067] For example, in one possible implementation, for each video segment sample, multiple third face image samples having expression intensity lower than a third intensity threshold and multiple fourth face image samples having expression intensity higher than a fourth intensity threshold in the video segment sample are determined, and the fourth intensity threshold is greater than the third intensity threshold.

[0068] Accordingly, the present application can construct at least one non-homologous positive sample pair and at least one non-homologous negative sample pair based on the third face image sample and the fourth face image sample in each video segment sample. Each non-homologous positive sample pair includes: two third face image samples belonging to different video segment samples, and each non-homologous negative sample pair includes: one third face image sample and one fourth face image sample belonging to different video segment samples.

[0069] Among them, the third intensity threshold and the fourth intensity threshold can also be set in combination with the value range of expression intensity and actual needs. For example, in the present application, expression intensity can be divided into four categories: no expression, slight expression, strong expression and severe expression. Then the facial image samples with expression intensity lower than the third intensity threshold can be facial image samples belonging to the expressionless category, and the facial image samples with expression intensity higher than the third intensity threshold can be facial image samples belonging to the strong expression category. For example, taking the value of expression intensity in the interval [0,3] as an example, the third intensity threshold can be 0.5, and the second intensity threshold can be 2.

[0070] For facial image samples from different video segment samples, in order to avoid constructing different types of expressions into positive sample pairs or constructing the same type of expressions into negative sample pairs, the present application only selects pairs from facial image samples whose expression intensity is lower than the third intensity threshold when constructing non-homologous positive sample pairs.

[0071] It can be understood that in this step, the two face image samples in the non-homologous positive sample pair and the non-homologous negative sample pair are used to select different video segment samples. Since the face image samples in different video segment samples may belong to different users, the facial action relationship between the face image samples of different users can be reflected through the non-homologous positive sample pair and the non-homologous negative sample pair.

[0072] In the present application, positive and negative sample pairs are constructed based on the similarity and difference of the expression intensity of face image samples. The feature of similar expression intensity of the same type of expressions can be effectively used to construct positive and negative sample pairs within the same video segment samples and different video segment samples.

[0073] S104, based on the face image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs and the non-homologous negative sample pairs, a facial action feature model is trained using a self-supervised learning method to obtain an initially trained facial action feature model.

[0074] It is understandable that after constructing positive and negative sample pairs, self-supervised learning can be achieved by measuring the distance between the positive and negative sample pairs. Self-supervised learning does not require sample labeling, thus avoiding the time-consuming and laborious problems caused by manual large-scale labeling of facial action features of face image samples.

[0075] In the present application, the facial expression features between the positive sample pairs extracted by the facial action feature model can be made closer through self-supervised learning, while the difference between the facial action features between the extracted negative sample pairs gradually increases. Of course, the present application does not impose any restrictions on the specific implementation of self-supervised learning.

[0076] In one implementation, the present application can construct a homologous face overall loss function for homologous positive sample pairs and homologous negative sample pairs constructed from the same video segment sample, and a non-homologous face loss function can be constructed for non-homologous positive sample pairs and non-homologous negative sample pairs constructed based on different video segment samples. On this basis, through continuous training and learning in the above steps S101 to S104, the homologous face overall loss function value and the homologous face overall loss function value are eventually stabilized, thereby completing self-supervised learning and obtaining a trained facial action feature model for extracting facial action features in face images.

[0077] It should be noted that the facial action feature model required to be trained in self-supervised learning can be any neural network model. By training the neural network model through self-supervised learning, the neural network model finally trained is the facial action feature model.

[0078] S105, using a plurality of facial image samples labeled with facial action categories, the initially trained facial action feature model and the facial action classifier to be trained are trained to obtain a facial action recognition model including the trained facial action feature model and the facial action classifier.

[0079] It is understandable that the facial action recognition model trained by self-supervised learning can only extract facial action features of face images, but cannot be used directly to recognize facial actions. In order to eventually train a facial action recognition model that can be applied to facial action feature recognition, this application actually uses labeled facial image samples to perform transfer learning on the previously trained facial action feature model to further adjust the facial action feature model and obtain a facial action classifier, thereby ultimately constructing a facial action recognition model that includes a facial action feature model and a facial action classifier.

[0080] In the present application, step S105 is actually to perform secondary training on the facial action feature model through transfer learning, and finally train the facial action feature model. Compared with directly using a large amount of labeled data to train the facial action feature model, the amount of labeled sample data required for transfer learning based on the facial action feature model learned by self-supervision is relatively small, so there is no need to manually label the face images on a large scale.

[0081] With the help of the above technical solution, after obtaining multiple video segment samples, the present application combines the expression intensity of each face image sample in each video segment sample, and not only selects face images from the same video segment sample to construct positive sample pairs and negative sample pairs, but also constructs positive sample pairs and negative sample pairs containing face images from different video segment samples. Since the facial image samples in the same video segment sample belong to the same user, while the facial image samples in different video segment samples may belong to different users, the positive sample pairs and negative sample pairs derived from the same video segment sample reflect the relationship between the facial actions of the same user, while the positive sample pairs and negative samples derived from two different video segment samples may reflect the relationship between the facial actions of different users. This allows the present application to consider both the facial action relationship between different facial images of the same user and the facial action relationship between facial images of different users during the training of the facial action feature model, so that the self-supervised learning process is not limited to comparative learning of facial images of a single user, but also performs comparative learning between facial images of different users, thereby enabling the trained facial action feature model to more accurately extract facial action features in facial images, and enabling the facial action recognition model finally trained to accurately recognize facial actions in facial images.

[0082] It is understandable that the above embodiments, when training the facial action feature model, are mainly based on comparative learning of the entire face in the face image, without considering the changes in local facial actions. The changes in the muscle actions of each part of the face will affect the changes in the facial actions of the face. Therefore, in order to further improve the accuracy of the facial action recognition model in recognizing facial actions, the present application also needs to construct positive and negative sample pairs that reflect the changes in facial actions of each part of the face during the training of the facial action recognition model, so that the facial action recognition model can learn more detailed facial action information.

[0083] like Figure 2 As shown, it shows another flow chart of the training method of the facial action recognition model of the present application. The method of this embodiment may include:

[0084] S201, obtaining multiple video segment samples.

[0085] Each video segment sample includes multiple face image samples belonging to the same user, and the face image samples in the multiple video segment samples do not completely belong to the same user.

[0086] S202: For each video segment sample, construct at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample.

[0087] Among them, the homologous positive sample pair includes two face image samples belonging to the same video segment sample and having similar expression intensity; the homologous negative sample pair includes two face image samples belonging to the same video segment sample and having an expression intensity difference exceeding the first intensity difference.

[0088] S203: construct at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples.

[0089] Among them, the non-homologous positive sample pair includes two face image samples belonging to different video segment samples and having similar expression intensities, and the non-homologous negative sample pair includes two face image samples belonging to different video segment samples and having expression intensity difference exceeding the second intensity difference.

[0090] The above steps can be referred to the related introduction of the previous embodiment, which will not be repeated here.

[0091] S204: For each face image sample in each video segment sample, perform key point detection on the face image sample to obtain a key point distribution of at least one face part in the face image sample.

[0092] Among them, the key point detection in the face image sample can determine the key points contained in the face image sample and the face part to which each key point belongs, so as to obtain the key point distribution of different face parts. The key point distribution of the face part includes each key point of the face part and its coordinate position.

[0093] In this application, the face parts that need to learn local motion changes can be set as needed. For example, this application can detect the key point distribution of some or all face parts such as the mouth, nose, left eyebrow, right eyebrow, left eye and right eye in the face.

[0094] S205 , for each video segment sample, determining the face image sample with the lowest expression intensity in the video segment sample as the face template image in the video segment sample.

[0095] It can be understood that the lowest expression intensity of a certain face image sample in the video segment sample indicates that the face image sample is the face image with the slightest facial movement of the user in the video segment sample. After the face image sample with the lowest expression intensity is used as the face template image, the local movement changes of each face part are determined by comparing the changes of other face image samples in the video segment sample with the changes of each face part in the face template image.

[0096] S206, for each face part in each video segment sample, based on the key point distribution of the face part in each face image sample in the video segment sample, construct at least one local positive sample pair and at least one local negative sample pair for the face part in the video segment sample.

[0097] The local positive sample pair includes: a face template image in the video segment sample, and a face image sample whose key point distribution of the face part in the video segment sample is similar to that of the face template image.

[0098] The local negative sample pair includes: a face template image in the video segment sample, and a face image sample in at least one face image sample whose key point distribution of the face part in the video segment sample is significantly different from that of the face template image.

[0099] For example, in a possible implementation, for each face part in a video segment sample, based on the key point distribution of the face part in each face image sample in the video segment sample, the reference length and reference width of the face part in the face template image of the video segment sample can be determined first. Accordingly, a face image sample that meets the first condition can be selected from the video segment sample to form a local positive sample pair with the face template image, and a face image sample that meets the second condition can be selected to form a local negative sample pair with the face template image.

[0100] Among them, for each face part in each video segment sample:

[0101] The first condition is that the sum of the first length difference and the second width difference corresponding to the face image sample is the smallest. The first length difference is the length difference between the length of the face part of the face image sample and the reference length, and the second width difference is the width difference between the width of the face part of the face image sample and the reference width.

[0102] The second condition includes: the difference between the length of the face part of the face image sample and the second length of the reference length is the largest; or the difference between the width of the face part of the face image sample and the second width of the reference width is the largest.

[0103] For the sake of distinction, the length of the face part in the face template image is referred to as the reference length, and the width of the face part is referred to as the reference width. The reference length of the face part in the face template image may be the distance between the two farthest key points on the horizontal coordinate of the face part in the face template image; and the reference width may be the distance between the two farthest key points on the vertical coordinate of the face part in the face template image.

[0104] Similarly, for face image samples other than face template images, the distance between the two key points with the farthest horizontal coordinates on the face part in the face image sample can be used as the length of the face part in the face image sample; and the distance between the two key points with the farthest vertical coordinates on the face part can be used as the width of the face part in the face image sample.

[0105] For example, take the mouth as an example:

[0106] Assume that the video segment sample has M face image samples in addition to the face template image, where M is a natural number greater than 1. And the reference length of the mouth in the face template image is x mould_mouth (width), the base width of the mouth is x mould_mouth (height), then for the mouth part, the face image sample x that can form a positive sample pair with the face template image mould_mouth_positive The total difference between the length and width of the mouth and the face template image is the smallest, which can be expressed by the following formula 1:

[0107]

[0108] Where p is any natural number from 1 to M. p_mouth (width) represents the mouth width of the pth face image sample in the video segment sample, x p_mouth (height) represents the mouth length of the pth face image sample in the video segment sample.

[0109] Similarly, the face image samples that can form a negative sample pair with the face template image may include: the face image sample x with the largest difference in mouth width mould_mouth_negative_width And the face image sample x with the largest difference in mouth length mould_mouth_negative_height , as shown in the following formula 2 and formula 3:

[0110]

[0111]

[0112] Of course, the above takes the mouth as an example. The process of constructing local positive or negative sample pairs for other facial parts is similar, so I will not go into details here.

[0113] It should be noted that the construction of local positive sample pairs and local negative sample pairs can be after or before the construction of sample pairs such as homologous positive sample pairs and homologous negative sample pairs, or it can be while constructing homologous or non-homologous positive (negative) sample pairs, and local positive sample pairs and negative sample pairs for different facial parts in each video segment sample are constructed. There is no restriction on this.

[0114] S207, based on the face image samples in the video segment samples, homologous positive sample pairs, homologous negative sample pairs, non-homologous positive sample pairs, non-homologous negative sample pairs, and local positive sample pairs and local negative sample pairs corresponding to each face part in the video segment samples, a facial action feature model is trained using a self-supervised learning method.

[0115] In this embodiment, the purpose of self-supervised learning is: through continuous learning, the gap in facial action features between homologous positive sample pairs, non-homologous positive sample pairs and face image samples within local positive sample pairs predicted by the facial action feature model is gradually narrowed, while the gap in facial action features between homologous negative sample pairs, non-homologous negative sample pairs and face image samples within each local negative sample pair is gradually increased.

[0116] For example, in one possible implementation, based on the face image samples in the video segment samples, the overall loss function of homologous faces, the overall loss function of non-homologous faces, and the local face loss functions corresponding to different types of face parts can be set, and a self-supervised learning method can be used to jointly train the facial action feature model with multiple loss functions.

[0117] Among them, the overall loss function of homologous faces is the loss function applicable to homologous positive sample pairs and homologous negative sample pairs; the overall loss function of non-homologous faces is the loss function applicable to non-homologous positive sample pairs and non-homologous negative sample pairs; and the local face loss function of each face part is the loss function applicable to the local positive sample pairs and local negative sample pairs corresponding to the face part.

[0118] S208, using a plurality of facial image samples labeled with facial action categories, the initially trained facial action feature model and the facial action classifier to be trained are trained to obtain a facial action recognition model including the trained facial action feature model and the facial action classifier.

[0119] For step S208, reference may be made to the related introduction of the previous embodiment, which will not be described again here.

[0120] In this embodiment, in the process of self-supervised learning of the facial action feature model, not only the overall facial actions of the same user and different users are learned, but also the facial actions of local parts of the face are used for learning, so that more surprising facial action features in the face can be learned, and the accuracy of facial action recognition by the finally trained facial action recognition model can be further improved.

[0121] In this application, the video segment samples used for training can be obtained in many ways.

[0122] In a possible implementation, the present application may first obtain multiple video segments, each of which includes multiple face image samples belonging to the same user, and the face image samples in the multiple video segments do not completely belong to the same user. On this basis, for each video segment, a pre-trained expression intensity recognition model may be used to determine the expression intensity of each face image sample in the video segment. Accordingly, multiple video segment samples that meet the set conditions may be screened out from the multiple video segments.

[0123] Among them, the video segment samples that meet the set conditions include: a set number of fifth face image samples whose expression intensity meets the low expression intensity condition, and a set number of sixth face image samples whose expression intensity meets the high expression intensity condition, and the proportion of face image samples whose expression intensity meets the expressionless condition among the set number of fifth face image samples is a set ratio.

[0124] The low expression intensity condition is a condition that makes the expression intensity of the face image sample belong to a set low expression intensity range. For example, the low expression intensity condition may be that the expression intensity is lower than a set low intensity threshold.

[0125] Similarly, the high expression intensity condition is a condition that makes the expression intensity of the face image sample belong to a set strong expression intensity range. For example, the high expression intensity condition can be that the expression intensity is lower than a set high intensity threshold.

[0126] In this implementation, the screened video segment samples need to contain both low expression intensity facial images and high expression intensity facial images, and the number of low expression intensity facial images and high expression intensity facial images are the same, to ensure that relatively rich positive sample pairs and negative sample pairs can be reasonably constructed.

[0127] It is understandable that the multiple video segments used to screen the video segment samples can be video segments containing faces extracted from movies, TV series and short videos obtained from the network platform. In order to obtain video segments of different users and each video segment contains only the face image of one user, after obtaining the video segments from the network platform or other channels, the application needs to clean the video segments to obtain multiple video segments that can be used to screen the video segment samples.

[0128] like Figure 3 As shown, it shows a schematic diagram of an implementation process of obtaining video segment samples in the present application. The process of this embodiment may include the following steps:

[0129] S301: Obtain multiple candidate video segments.

[0130] The candidate video segment includes multiple frames of face images.

[0131] The candidate video segments may be video segments containing human faces obtained from different channels, but each candidate video segment may contain facial images of multiple users, and there may also be some facial images in the candidate video segments that are not suitable for learning facial action features.

[0132] For example, after obtaining the initial video segments through a network platform or other means, image frames can be extracted for each initial video segment, and then face detection is performed on the image frames. If the image frame does not contain an image of a face, the image frame is discarded. Finally, the image frames extracted from the initial video segments and containing faces are combined into a candidate video segment in order.

[0133] S302, for each frame of the candidate video segment, use a pre-trained head posture estimation model to determine the head posture of each face image in the candidate video segment frame, and based on the head posture of the face image, filter out at least one face image sample in the candidate video segment whose offset angle relative to the set face normal direction is less than a set angle value, to obtain a filtered candidate video segment.

[0134] The filtered candidate video segment contains the at least one selected face image sample.

[0135] In the present application, the head posture estimation model can be any existing model for estimating head posture in face images, without limitation. For example, the head posture estimation model can be obtained by training a neural network model using multiple face images labeled with head postures.

[0136] The set face orientation is also called face positive direction, which represents the direction in which the face faces the camera.

[0137] The set angle value can be set as needed, for example, the set angle value can be 20 degrees.

[0138] It is understandable that the head posture of a face image can characterize the direction of the face in the face image, and can also determine information such as the offset angle of the face in the face image relative to the normal face direction. Based on this, if it is determined based on the head posture of the face image in the candidate video segment that the angle at which the face in the face image deviates from the normal face direction exceeds a set angle value (such as 20 degrees), the face image needs to be discarded, so that only the face image whose angle of deviation from the normal face direction is less than the set angle value is retained in the candidate video segment, and the retained face image is the face image sample.

[0139] S303, for each filtered candidate video segment, extract the facial features of each facial image sample in the candidate video segment respectively, split the candidate video segment into at least one video segment according to the facial features of each facial image sample in the candidate video segment and the sequence of each facial image sample, and obtain multiple video segments split from the multiple candidate video segments.

[0140] Each video segment includes at least one face image sample of the same user.

[0141] For example, according to the facial features of each facial image sample in the candidate video segment, the facial similarity of any two adjacent frames of facial image samples can be calculated in sequence. If the facial similarity of the two frames of facial image samples exceeds the set similarity threshold, it is confirmed that the two frames of facial image samples belong to the facial images of the same user.

[0142] On this basis, according to the order of the facial image samples of each frame in the candidate video segment, multiple frames of facial image samples belonging to the same user and continuous with each other can be divided into one video segment, while facial image samples of different users will be divided into different video segments, thereby dividing the candidate video segment into video segments corresponding to multiple different users.

[0143] As an optional method, considering that the change of facial emotions is a continuous process, if the duration of the video segment is short, it is difficult to reflect the change process of facial emotions. Therefore, the present application can also discard video segments whose duration is less than the set duration (such as two seconds).

[0144] S304: for each video segment, using a pre-trained expression intensity recognition model, respectively determine the expression intensity of each face image sample in the video segment.

[0145] Among them, the expression intensity recognition model can adopt an existing recognition model for identifying expression intensity; it can also be pre-trained using multiple face image samples labeled with expression intensity, such as training a network model by regression based on face image sample data labeled with expression intensity to obtain a trained expression intensity recognition model.

[0146] S305: Filter out a plurality of video segment samples satisfying a set condition from the plurality of video segments.

[0147] Among them, the video segment samples that meet the set conditions include: a set number of fifth face image samples whose expression intensity meets the low expression intensity condition, and a set number of sixth face image samples whose expression intensity meets the high expression intensity condition, and the proportion of face image samples whose expression intensity meets the expressionless condition among the set number of fifth face image samples is a set ratio.

[0148] For step S305, please refer to the above related introduction, which will not be repeated here.

[0149] It should be noted that Figure 3 The following is an example of an implementation method for obtaining video segment samples. In practical applications, video segment samples may also be obtained in other ways, which are not limited.

[0150] In order to facilitate understanding of the solution of the present application, an example is provided below for explanation. The expression intensity of a facial image is divided into four intensity levels as an example. The four intensity levels are: no expression, slight expression, strong expression, and severe expression. Accordingly, the expression intensity value of the facial image sample of the present application is between the interval [0, 3], where the expression intensity value of 0 represents no expression, and the expression intensity value of 3 represents severe expression.

[0151] like Figure 4 As shown, it shows a flow chart of a training method for a facial action recognition model provided by an embodiment of the present application in an application example. The method of this embodiment may include:

[0152] S401, obtaining multiple video segments.

[0153] Each video segment includes at least one face image sample of the same user, and the face image samples in multiple video segments do not completely belong to the same user.

[0154] S402, for each video segment, using a pre-trained expression intensity recognition model, respectively determine the expression intensity of each face image sample in the video segment, and determine the face image sample with the lowest expression intensity in the video segment sample as the face template image in the video segment sample.

[0155] S403: Select a target number of video segment samples that meet a set condition from the multiple video segments.

[0156] In this embodiment, the target number is K, which is the number of selected video segment samples and is a natural number greater than 1.

[0157] Among them, the video segment samples that meet the set conditions include: a set number N of face image samples with expression intensity less than 1, and N face image samples with expression intensity greater than 2, and among the set number N face image samples with expression intensity less than 1, the proportion of face image samples with expression intensity less than 0.5 is 1 / 2.

[0158] S404: For each video segment sample, construct at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample.

[0159] Among them, the homologous positive sample pair includes: two face image samples x that belong to the same video segment sample and whose expression intensity is less than 1 strength＜1 ; or, two face image samples x that belong to the same video segment sample and whose expression intensity is greater than 2 strength＞2 .

[0160] The homologous negative sample pair includes: a face image sample x that belongs to the same video segment and has an expression intensity less than 1 strength＜1 and a face image sample x with expression intensity greater than 2 strength＞2 .

[0161] S405 : construct at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples.

[0162] Among them, non-homologous positive sample pairs include: face image samples x belonging to different video segments and with expression strength less than 0.5 strength＜0.5 .

[0163] Non-homologous negative sample pairs include: samples belonging to different video segments, and a face image sample x with expression intensity less than 0.5 strength＜0.5 And a face image sample x with expression intensity greater than 2 strength＞2 .

[0164] It can be seen that for frame images from different video segments, in order to avoid different types of expressions being used as positive sample pairs or the same type of expressions being used as negative sample pairs, the present application only performs non-homologous positive sample pair pairing between facial image samples that are close to expressionless and have expression intensity lower than 0.5 in samples from different video segments, and performs negative sample pair pairing between facial images that are close to expressionless and facial image samples with stronger expressions in samples from different video segments.

[0165] S406: For each face image sample in each video segment sample, perform key point detection on the face image sample to obtain a key point distribution of at least one face part in the face image sample.

[0166] S407, for each face part in each video segment sample, based on the key point distribution of the face part in each face image sample in the video segment sample, select a face image sample that meets the first condition from the video segment sample and the face template image in the video segment sample to form a local positive sample pair, and select a face image sample that meets the second condition and the face template image to form a local negative sample pair.

[0167] Among them, for each face part in a video segment sample, the first condition is that the sum of the first length difference and the second width difference corresponding to the face image sample is the smallest. The first length difference is the length difference between the length of the face part of the face image sample and the reference length of the face part in the face template image, and the second width difference is the width difference between the width of the face part of the face image sample and the reference width of the face part in the face template image.

[0168] The second condition includes: the difference between the length of the face part of the face image sample and the second length of the reference length is the largest; or the difference between the width of the face part of the face image sample and the second width of the reference width is the largest.

[0169] Among them, for each face part, the width of the face part in the face image sample (also applicable to the face template image) is: the distance between the two farthest key points on the horizontal coordinate corresponding to the face part in the face image sample; and the length of the face part is: the distance between the two farthest key points on the vertical coordinate corresponding to the face part in the face image sample.

[0170] S408, based on the face image samples in each video segment sample, the overall loss function of the homologous face, the overall loss function of the non-homologous face and the local face loss function corresponding to different types of face parts are set, and the facial action feature model is jointly trained with multiple loss functions using a self-supervised learning method.

[0171] In this embodiment, the overall loss function of homologous faces is a loss function applicable to homologous positive sample pairs and homologous negative sample pairs. The overall loss function value L same_video See the following formula 4:

[0172]

[0173] Among them, K is the number of selected video segment samples (i.e., the target number); N is the number of face image samples with expression intensity less than 1 contained in the video segment samples, or the number of face image samples with expression intensity greater than 2; the values ​​of p1 and p2 are both natural numbers from 1 to 2N.

[0174] l same_video(p1, p2) represents the loss function value between the p1th face image sample and the p2th face image sample among the 2N face image samples with expression intensity less than 1 and expression intensity greater than 2 in the same video segment sample;

[0175] l same_video (p2, p1) represents the loss function value between the p2th face image sample and the p1th face image sample among the 2N face image samples with expression intensity less than 1 and expression intensity greater than 2 in the same video segment sample;

[0176] l same_video (p1,p2) and l same_video (p2, p1) can be calculated by the following formula 5.

[0177] Formula 5 represents the loss function value L between the i-th face image sample and the j-th face image sample among the 2N face image samples with expression intensity less than 1 and expression intensity greater than 2 in the same video segment sample. same_video (i,j):

[0178]

[0179] Where i and j are natural numbers from 1 to 2N, k is a natural number from 1 to K, and m is a natural number from 1 to N.

[0180] represents the face image sample with the i-th expression intensity less than 1 in the video segment sample k;

[0181] represents the face image sample with the jth expression intensity less than 1 in the video segment sample k;

[0182] represents the face image sample with the i-th expression intensity greater than 2 in the video segment sample k;

[0183] Indicates the face image sample with the jth expression intensity greater than 2 in the video segment sample k;

[0184] Indicates the face image sample with the mth expression intensity greater than 2 in the video segment sample k;

[0185] f() is the function representation of the facial action feature model, such as, Indicates inputting face image samples into the facial action feature model After that, the output facial action features.

[0186] Among them, the overall loss function of non-homologous faces is a loss function applicable to non-homologous positive sample pairs and non-homologous negative sample pairs. The overall loss function value of non-homologous faces is L different_video It can be expressed as the following formula 6:

[0187]

[0188] Among them, l different_video (p1, p2) represents the loss function value between the p1th face image sample among 2N face image samples in a video segment sample and the p2th face image sample among 2N face image samples in another video segment sample; the 2N face image samples in each video image sample are the 2N face image samples with expression intensity less than 1 and expression intensity greater than 2 in the video image sample.

[0189] l different_video (p2, p1) represents the loss function value between the p2th face image sample among 2N face image samples in a video segment sample and the p1th face image sample among 2N face image samples in another video segment sample.

[0190] Similarly, l different_video (p1,p2) and l different_video (p2, p1) can be calculated by the following formula 7.

[0191]

[0192] Formula 7 represents the loss function value between the i-th face image sample among the 2N face image samples in the same video segment sample and the j-th face image sample among the 2N face image samples in another video segment sample.

[0193] Among them, the value of q is a natural number from 1 to K.

[0194] Indicates the face image sample with the i-th expression intensity less than 0.5 in the video segment sample k;

[0195] Indicates the face image sample with the jth expression intensity less than 0.5 in the video segment sample k;

[0196] Represents the mth face image sample with expression intensity greater than 2 in the video segment sample q.

[0197] Similarly, the local face loss function for each face part is a loss function applicable to the local positive sample pairs and local negative sample pairs corresponding to the face part.

[0198] For example, the function value L of the local loss function of the mouth in the face mouth Please refer to the following formula 8:

[0199]

[0200] in, Represents the face template image in the video segment sample k.

[0201] On this basis, the loss function value L corresponding to the total loss function of self-supervised learning can be expressed as the following formula 9:

[0202] L=L same_video +λ1L different_video +λ2L mouth +λ3L left_eyebrow +λ4L right_eyebrow +λ5L left_eye +λ6L right_eye , (Formula 9);

[0203] Among them, λ1, λ2, λ3, λ4, λ5, and λ6 are all pre-set different parameter values, which can be set or adjusted according to actual needs. left_eyebrow , L right_eyebrow , L left_eye , L right_eye Represent the loss function values ​​of self-supervised learning for the left eyebrow, right eyebrow, left eye, and right eye, respectively.

[0204] S409, using a plurality of facial image samples labeled with facial action categories, the initially trained facial action feature model and the facial action classifier to be trained are trained to obtain a facial action recognition model including the trained facial action feature model and the facial action classifier.

[0205] For step S409, reference may be made to the related introduction of the previous embodiment, which will not be described again here.

[0206] On the other hand, the present application also provides a method for recognizing facial movements of a human face.

[0207] like Figure 5 As shown, it shows a flow chart of a method for face and facial action recognition provided by the present application. The method of this embodiment may include:

[0208] S501, obtaining a face image to be recognized.

[0209] The face image is a face image for which a facial action category needs to be identified.

[0210] S502: Recognize facial actions in the face image using a pre-trained facial action recognition model.

[0211] The facial action recognition model may be obtained by using the training method of the facial action recognition model described in any of the previous embodiments.

[0212] On the other hand, corresponding to the training method of the facial action recognition model of the present application, the present application also provides a training method of the facial action recognition model.

[0213] like Figure 6 As shown, it shows a schematic diagram of the composition structure of a training device for a facial action recognition model of the present application. The device of this embodiment may include:

[0214] The video segment obtaining unit 601 is used to obtain a plurality of video segment samples, each of which includes a plurality of face image samples belonging to the same user, and the face image samples in the plurality of video segment samples do not all belong to the same user;

[0215] A first sample construction unit 602 is used to construct, for each video segment sample, at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample, wherein the homologous positive sample pair includes two human face image samples belonging to the same video segment sample and having similar expression intensities, and the homologous negative sample pair includes two human face image samples belonging to the same video segment sample and having an expression intensity difference exceeding a first intensity difference;

[0216] A second sample construction unit 603 is configured to construct at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples, wherein the non-homologous positive sample pair includes two human face image samples belonging to different video segment samples and having similar expression intensities, and the non-homologous negative sample pair includes two human face image samples belonging to different video segment samples and having an expression intensity difference exceeding a second intensity difference;

[0217] A self-supervised learning unit 604 is used to train a facial action feature model using a self-supervised learning method based on the facial image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs to obtain an initially trained facial action feature model;

[0218] The transfer learning unit 605 is used to train the initially trained facial action feature model and the facial action classifier to be trained using multiple facial image samples labeled with facial action categories to obtain a facial action recognition model including the trained facial action feature model and the facial action classifier.

[0219] In a possible implementation, the device further includes:

[0220] A key point detection unit is used to perform key point detection on each face image sample in each video segment sample before the self-supervised learning unit trains the facial action feature model using the self-supervised learning method, so as to obtain a key point distribution of at least one face part in the face image sample;

[0221] A template determination unit, configured to determine, for each video segment sample, a face image sample with the lowest expression intensity in the video segment sample as a face template image in the video segment sample;

[0222] A third sample construction unit is used to construct, for each face part in each video segment sample, at least one local positive sample pair and at least one local negative sample pair for the face part in the video segment sample based on the key point distribution of the face part in each face image sample in the video segment sample, wherein the local positive sample pair includes: a face template image in the video segment sample, and a face image sample whose key point distribution of the face part in the video segment sample is similar to that of the face template image; the local negative sample pair includes: the face template image in the video segment sample, and a face image sample in at least one face image sample whose key point distribution of the face part in the video segment sample is significantly different from that of the face template image;

[0223] The self-supervised learning unit is specifically used to train the facial action feature model using a self-supervised learning method based on the face image samples in the video segment samples, the homologous positive sample pairs, homologous negative sample pairs, non-homologous positive sample pairs, non-homologous negative sample pairs, and local positive sample pairs and local negative sample pairs corresponding to each face part in the video segment samples.

[0224] In yet another possible implementation, the self-supervised learning unit includes:

[0225] A self-supervised learning subunit is used to perform multi-loss function joint training on the facial action feature model using a self-supervised learning method based on the face image samples in the video segment samples, the set homologous face overall loss function, the non-homologous face overall loss function, and the face local loss function corresponding to different types of face parts;

[0226] The homologous face overall loss function is a loss function applicable to homologous positive sample pairs and homologous negative sample pairs;

[0227] The non-homologous face overall loss function is a loss function applicable to non-homologous positive sample pairs and non-homologous negative sample pairs;

[0228] The local face loss function of each face part is a loss function applicable to the local positive sample pairs and the local negative sample pairs corresponding to the face part.

[0229] In yet another possible implementation, the first sample construction unit includes:

[0230] A first sample preliminary selection subunit is used to determine, for each video segment sample, a plurality of first face image samples whose expression intensity is lower than a first intensity threshold and a plurality of second face image samples whose expression intensity is higher than a second intensity threshold in the video sample, wherein the second intensity threshold is greater than the first intensity threshold;

[0231] The first sample construction subunit is used to construct at least one homologous positive sample pair and at least one homologous negative sample pair for each video segment sample based on the multiple first face image samples and the multiple second face image samples, wherein the homologous positive sample pair includes: two first face image samples or two second face image samples belonging to the same video segment sample, and the homologous negative sample pair includes: a first face image sample and a second face image sample belonging to the same video segment sample.

[0232] In yet another possible implementation, the second sample construction unit includes:

[0233] A second sample preliminary selection subunit is used to determine, for each video segment sample, a plurality of third face image samples whose expression intensity is lower than a third intensity threshold, and a plurality of fourth face image samples whose expression intensity is higher than a fourth intensity threshold in the video segment sample, wherein the fourth intensity threshold is greater than the third intensity threshold;

[0234] The second sample construction subunit is used to construct at least one non-homologous positive sample pair and at least one non-homologous negative sample pair based on the third face image sample and the fourth face image sample in each video segment sample, wherein the non-homologous positive sample pair includes: two third face image samples belonging to different video segment samples, and the non-homologous negative sample pair includes a third face image sample and a fourth face image sample belonging to different video segment samples.

[0235] In yet another possible implementation, the third sample construction unit includes:

[0236] The third sample construction subunit is used to determine, for each face part in each video segment sample, a reference length and a reference width of the face part in the face template image of the video segment sample based on the key point distribution of the face part in each face image sample in the video segment sample, select a face image sample that meets a first condition from the video segment sample and the face template image to form a local positive sample pair, and select a face image sample that meets a second condition and the face template image to form a local negative sample pair;

[0237] The first condition is that the sum of the first length difference and the second width difference corresponding to the face image sample is the smallest, the first length difference is the length difference between the length of the face part of the face image sample and the reference length, and the second width difference is the width difference between the width of the face part of the face image sample and the reference width;

[0238] The second condition includes: the difference between the length of the face part of the face image sample and the second length of the reference length is the largest; or the difference between the width of the face part of the face image sample and the second width of the reference width is the largest.

[0239] In yet another possible implementation, the video segment obtaining unit includes:

[0240] An initial video obtaining unit, used to obtain a plurality of video segments, each of which includes a plurality of face image samples belonging to the same user, and the face image samples in the plurality of video segments do not completely belong to the same user;

[0241] An expression recognition unit, for determining the expression intensity of each face image sample in each video segment using a pre-trained expression intensity recognition model;

[0242] A video segment screening unit is used to screen out multiple video segment samples that meet set conditions from the multiple video segments, wherein the video segment samples that meet the set conditions include: a set number of fifth face image samples whose expression intensity meets the low expression intensity condition, and a set number of sixth face image samples whose expression intensity meets the high expression intensity condition, and the proportion of face image samples whose expression intensity meets the expressionless condition in the set number of fifth face image samples is a set ratio.

[0243] In an optional manner, the initial video obtaining unit includes:

[0244] A candidate video obtaining unit, used to obtain a plurality of candidate video segments, wherein the candidate video segments include a plurality of frames of face images;

[0245] a candidate video screening unit, configured to determine, for each candidate video segment, the head posture of each face image in each frame of the candidate video segment using a pre-trained head posture estimation model, and screen out, based on the head posture of the face image, at least one face image sample in the candidate video segment whose offset angle relative to a set face normal orientation is less than a set angle value, to obtain a filtered candidate video segment, wherein the filtered candidate video segment contains the screened at least one face image sample;

[0246] The candidate video splitting unit is used to extract the facial features of each facial image sample in each filtered candidate video segment, and split the candidate video segment into at least one video segment according to the facial features of each facial image sample in the candidate video segment and the sequence of each facial image sample, each video segment including at least one facial image sample of the same user.

[0247] On the other hand, corresponding to a human face facial action recognition method of the present application, the present application also provides a human face facial action recognition device. Figure 7 As shown, it shows a schematic diagram of the composition structure of a face and facial action recognition device provided by the present application. The device of this embodiment includes:

[0248] An image acquisition unit 701 is used to acquire a face image to be recognized;

[0249] The facial action recognition unit 702 is used to recognize facial actions in the face image using a pre-trained facial action recognition model, wherein the facial action recognition model is obtained using the training method for the facial action recognition model described in any one of claims 1 to 8.

[0250] The training method of the facial action recognition model provided in the embodiment of the present application can be applied to a computer device, a personal computer or a server, etc. Optionally, Figure 8 A hardware structure block diagram of a computer device provided by the present application is shown, referring to Figure 8 , the hardware structure of the computer device may include: at least one processor 801, at least one communication interface 802, at least one memory 803 and at least one communication bus 804;

[0251] In the embodiment of the present application, the number of the processor 801, the communication interface 802, the memory 803, and the communication bus 804 is at least one, and the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804;

[0252] The processor 801 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0253] The memory 803 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0254] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0255] Obtaining a plurality of video segment samples, each of which includes a plurality of face image samples belonging to the same user, and the face image samples in the plurality of video segment samples do not all belong to the same user;

[0256] For each video segment sample, construct at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample, wherein the homologous positive sample pair includes two human face image samples belonging to the same video segment sample and having similar expression intensities, and the homologous negative sample pair includes two human face image samples belonging to the same video segment sample and having an expression intensity difference exceeding a first intensity difference;

[0257] Constructing at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples, the non-homologous positive sample pair comprising two human face image samples belonging to different video segment samples and having similar expression intensities, and the non-homologous negative sample pair comprising two human face image samples belonging to different video segment samples and having expression intensity difference exceeding a second intensity difference;

[0258] Based on the face image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs, a facial action feature model is trained using a self-supervised learning method to obtain an initially trained facial action feature model;

[0259] The initially trained facial action feature model and the facial action classifier to be trained are trained using a plurality of facial image samples labeled with facial action categories, thereby obtaining a facial action recognition model including the trained facial action feature model and the facial action classifier.

[0260] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0261] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0262] Obtaining a plurality of video segment samples, each of which includes a plurality of face image samples belonging to the same user, and the face image samples in the plurality of video segment samples do not all belong to the same user;

[0263] For each video segment sample, construct at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample, wherein the homologous positive sample pair includes two human face image samples belonging to the same video segment sample and having similar expression intensities, and the homologous negative sample pair includes two human face image samples belonging to the same video segment sample and having an expression intensity difference exceeding a first intensity difference;

[0264] Constructing at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples, the non-homologous positive sample pair comprising two human face image samples belonging to different video segment samples and having similar expression intensities, and the non-homologous negative sample pair comprising two human face image samples belonging to different video segment samples and having expression intensity difference exceeding a second intensity difference;

[0265] Based on the face image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs, a facial action feature model is trained using a self-supervised learning method to obtain an initially trained facial action feature model;

[0266] The initially trained facial action feature model and the facial action classifier to be trained are trained using a plurality of facial image samples labeled with facial action categories, thereby obtaining a facial action recognition model including the trained facial action feature model and the facial action classifier.

[0267] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0268] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0269] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0270] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a facial action recognition model, characterized in that: include: Obtaining a plurality of video segment samples, each of the video segment samples comprising a plurality of face image samples belonging to the same user, and the face image samples in the plurality of video segment samples do not all belong to the same user; For each video segment sample, construct at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample, wherein the homologous positive sample pair includes two human face image samples belonging to the same video segment sample and having similar expression intensities, and the homologous negative sample pair includes two human face image samples belonging to the same video segment sample and having an expression intensity difference exceeding a first intensity difference; Constructing at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples, the non-homologous positive sample pair comprising two human face image samples belonging to different video segment samples and having similar expression intensities, and the non-homologous negative sample pair comprising two human face image samples belonging to different video segment samples and having expression intensity difference exceeding a second intensity difference; Based on the face image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs, a facial action feature model is trained using a self-supervised learning method to obtain an initially trained facial action feature model; The initially trained facial action feature model and the facial action classifier to be trained are trained using a plurality of facial image samples labeled with facial action categories to obtain a facial action recognition model including the trained facial action feature model and the facial action classifier.

2. The method according to claim 1, characterized in that Before the self-supervised learning method is used to train the facial action feature model, the method further includes: For each face image sample in each video segment sample, perform key point detection on the face image sample to obtain a key point distribution of at least one face part in the face image sample; For each video segment sample, determining the face image sample with the lowest expression intensity in the video segment sample as the face template image in the video segment sample; For each face part in each video segment sample, based on the key point distribution of the face part in each face image sample in the video segment sample, at least one local positive sample pair and at least one local negative sample pair for the face part in the video segment sample are constructed, wherein the local positive sample pair includes: a face template image in the video segment sample, and a face image sample in which the key point distribution of the face part in the video segment sample is similar to the key point distribution of the face template image; the local negative sample pair includes: the face template image in the video segment sample, and a face image sample in at least one face image sample in which the key point distribution of the face part in the video segment sample is significantly different from the key point distribution of the face template image; The method of training a facial action feature model using a self-supervised learning method based on the face image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs comprises: Based on the face image samples in the video segment samples, the homologous positive sample pairs, homologous negative sample pairs, non-homologous positive sample pairs, non-homologous negative sample pairs, and local positive sample pairs and local negative sample pairs corresponding to each face part in the video segment samples, a facial action feature model is trained using a self-supervised learning method.

3. The method according to claim 2, characterized in that The facial action feature model is trained using a self-supervised learning method based on the face image samples in the video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, the non-homologous negative sample pairs, and the local positive sample pairs and local negative sample pairs corresponding to each face part in the video segment samples, including: Based on the face image samples in the video segment samples, a homologous face overall loss function, a non-homologous face overall loss function, and a face local loss function corresponding to different face parts are set, and a facial action feature model is jointly trained with multiple loss functions using a self-supervised learning method; The homologous face overall loss function is a loss function applicable to homologous positive sample pairs and homologous negative sample pairs; The non-homologous face overall loss function is a loss function applicable to non-homologous positive sample pairs and non-homologous negative sample pairs; The local face loss function of each face part is a loss function applicable to the local positive sample pairs and the local negative sample pairs corresponding to the face part.

4. The method according to claim 1, characterized in that The step of constructing, for each video segment sample, at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample comprises: For each video segment sample, determining a plurality of first face image samples having expression intensity lower than a first intensity threshold and a plurality of second face image samples having expression intensity higher than a second intensity threshold in the video segment sample, wherein the second intensity threshold is greater than the first intensity threshold; For each video segment sample, based on the multiple first face image samples and the multiple second face image samples, at least one homologous positive sample pair and at least one homologous negative sample pair are constructed, the homologous positive sample pair includes: two first face image samples or two second face image samples belonging to the same video segment sample, and the homologous negative sample pair includes: a first face image sample and a second face image sample belonging to the same video segment sample.

5. The method according to claim 1 or 4, characterized in that: The constructing at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples includes: For each video segment sample, determining a plurality of third face image samples having expression strength lower than a third intensity threshold and a plurality of fourth face image samples having expression strength higher than a fourth intensity threshold in the video segment sample, wherein the fourth intensity threshold is greater than the third intensity threshold; Based on the third face image sample and the fourth face image sample in each video segment sample, at least one non-homologous positive sample pair and at least one non-homologous negative sample pair are constructed, wherein the non-homologous positive sample pair includes: two third face image samples belonging to different video segment samples, and the non-homologous negative sample pair includes a third face image sample and a fourth face image sample belonging to different video segment samples.

6. The method according to claim 2, characterized in that The method of constructing, for each face part in each video segment sample, at least one local positive sample pair and at least one local negative sample pair for the face part in the video segment sample based on the key point distribution of the face part in each face image sample in the video segment sample, comprises: For each face part in each video segment sample, based on the distribution of key points of the face part in each face image sample in the video segment sample, determine the reference length and reference width of the face part in the face template image of the video segment sample, select a face image sample that meets a first condition from the video segment sample and the face template image to form a local positive sample pair, and select a face image sample that meets a second condition and the face template image to form a local negative sample pair; The first condition is that the sum of the first length difference and the second width difference corresponding to the face image sample is the smallest, the first length difference is the length difference between the length of the face part of the face image sample and the reference length, and the second width difference is the width difference between the width of the face part of the face image sample and the reference width; The second condition includes: the difference between the length of the face part of the face image sample and the second length of the reference length is the largest; or the difference between the width of the face part of the face image sample and the second width of the reference width is the largest.

7. The method according to claim 1, characterized in that The obtaining of multiple video segment samples comprises: Obtaining multiple video segments, each video segment includes multiple face image samples belonging to the same user, and the face image samples in the multiple video segments do not completely belong to the same user; For each video segment, using a pre-trained expression intensity recognition model, respectively determine the expression intensity of each face image sample in the video segment; A plurality of video segment samples satisfying set conditions are screened out from the plurality of video segments, wherein the video segment samples satisfying the set conditions include: a set number of fifth face image samples whose expression intensity meets a low expression intensity condition, and a set number of sixth face image samples whose expression intensity meets a high expression intensity condition, and a proportion of face image samples whose expression intensity meets a no expression condition among the set number of fifth face image samples is a set ratio.

8. The method according to claim 7, characterized in that The obtaining of multiple video segments comprises: Obtaining a plurality of candidate video segments, wherein the candidate video segments include a plurality of frames of face images; For each candidate video segment, use a pre-trained head posture estimation model to respectively determine the head posture of each face image in the candidate video segment frame, and based on the head posture of the face image, filter out at least one face image sample in the candidate video segment whose offset angle relative to a set face normal direction is less than a set angle value, to obtain a filtered candidate video segment, wherein the filtered candidate video segment contains at least one filtered face image sample; For each filtered candidate video segment, facial features of each facial image sample in the candidate video segment are extracted respectively, and according to the facial features of each facial image sample in the candidate video segment and the sequence of each facial image sample, the candidate video segment is split into at least one video segment, and each video segment includes at least one facial image sample of the same user.

9. A method for recognizing facial movements, characterized in that: include: Obtain a face image to be recognized; Facial actions in the face image are identified using a pre-trained facial action recognition model, wherein the facial action recognition model is obtained using the training method for the facial action recognition model described in any one of claims 1 to 8.

10. A training device for a facial action recognition model, characterized in that: include: A video segment obtaining unit, configured to obtain a plurality of video segment samples, each of the video segment samples comprising a plurality of face image samples belonging to the same user, and the face image samples in the plurality of video segment samples do not completely belong to the same user; A first sample construction unit is used to construct, for each video segment sample, at least one homologous positive sample pair and at least one homologous negative sample pair of the video segment sample, wherein the homologous positive sample pair includes two human face image samples belonging to the same video segment sample and having similar expression intensities, and the homologous negative sample pair includes two human face image samples belonging to the same video segment sample and having an expression intensity difference exceeding a first intensity difference; A second sample construction unit is used to construct at least one non-homologous positive sample pair and at least one non-homologous negative sample pair for the multiple video segment samples, wherein the non-homologous positive sample pair includes two human face image samples belonging to different video segment samples and having similar expression intensities, and the non-homologous negative sample pair includes two human face image samples belonging to different video segment samples and having an expression intensity difference exceeding a second intensity difference; A self-supervised learning unit is used to train a facial action feature model using a self-supervised learning method based on the facial image samples in the multiple video segment samples, the homologous positive sample pairs, the homologous negative sample pairs, the non-homologous positive sample pairs, and the non-homologous negative sample pairs, to obtain an initially trained facial action feature model; The transfer learning unit is used to train the initially trained facial action feature model and the facial action classifier to be trained using a plurality of facial image samples labeled with facial action categories, so as to obtain a facial action recognition model including the trained facial action feature model and the facial action classifier.

11. A facial action recognition device, characterized in that: include: An image acquisition unit, used for acquiring a face image to be recognized; A facial action recognition unit is used to recognize facial actions in the face image using a pre-trained facial action recognition model, wherein the facial action recognition model is obtained using the training method for the facial action recognition model described in any one of claims 1 to 8.

12. A computer device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the training method for the facial action recognition model as described in any one of claims 1 to 8.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the training method for a facial action recognition model according to any one of claims 1 to 8 is implemented.

Images