Facial expression recognition model training method, facial expression recognition method, related device and medium

Abstract

The embodiment of the specification discloses an expression recognition model training method, an expression recognition method, related devices and media. The expression recognition model training method comprises: obtaining a first sample data set and a second sample data set, the first sample data set comprising a plurality of first sample images of a source domain with expression labels, and the second sample data set comprising a plurality of second sample images of a target domain without expression labels; determining a first paired sample data set based on the first sample data set, the first paired sample data set comprising at least one group of first paired sample data, and the first paired sample data comprising at least two first sample images in the first sample data set; and training an expression recognition model based on the first sample data set, the first paired sample data set and the second sample data set.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method for training an expression recognition model, an expression recognition method, related devices and media. Background Technology

[0002] Facial expression recognition is a technology that uses static images or video sequences to identify various facial expressions such as happiness, surprise, sadness, anger, disgust, fear, and neutrality. It enables computers to better understand human emotions and thus become better intelligent assistants, achieving better human-computer interaction.

[0003] Currently, the main approach is to first train the expression recognition model using a set of manually labeled facial images, and then use the trained expression recognition model to achieve expression recognition based on the user's facial images. Summary of the Invention

[0004] This specification provides an embodiment of a method for training an expression recognition model, an expression recognition method, related devices, and a medium. By using a first paired sample dataset, the expression recognition model undergoes comparative learning training, improving its robustness in recognizing ambiguous expression categories. The above technical solution is as follows:

[0005] Firstly, the embodiments of this specification provide a method for training an expression recognition model, including:

[0006] Obtain a first sample dataset and a second sample dataset; the first sample dataset includes multiple first sample images with expression labels in the source domain; the second sample dataset includes multiple second sample images without expression labels in the target domain;

[0007] A first paired sample dataset is determined based on the aforementioned first sample dataset; the aforementioned first paired sample dataset includes at least one set of first paired sample data; the aforementioned first paired sample data includes at least two first sample images from the aforementioned first sample dataset;

[0008] The facial expression recognition model was trained based on the first sample dataset, the first paired sample dataset, and the second sample dataset mentioned above.

[0009] In one possible implementation, training the facial expression recognition model based on the first sample dataset, the first paired sample dataset, and the second sample dataset includes:

[0010] The first loss of the facial expression recognition model is determined based on the first paired sample dataset mentioned above.

[0011] Based on the first sample dataset and the second sample dataset mentioned above, the second loss of the above expression recognition model is determined;

[0012] The above-mentioned facial expression recognition model is trained based on the first loss and the second loss mentioned above.

[0013] In one possible implementation, the above-mentioned facial expression recognition model includes a backbone network, an uncertainty extraction network, and an facial expression feature generation network;

[0014] The first loss of the expression recognition model, determined based on the first paired sample dataset, includes:

[0015] The first feature of each first sample image in the first paired sample data is extracted using the aforementioned backbone network.

[0016] The first feature of each first sample image in the first paired sample data is input into the uncertainty extraction network, and the uncertainty vector corresponding to each first sample image in the first paired sample data is output.

[0017] The first feature of each first sample image in the first paired sample data is input into the expression feature generation network, and the first expression feature vector corresponding to each first sample image in the first paired sample data is output.

[0018] The uncertainty vector is fused with the first facial expression feature vector to obtain the mixed feature corresponding to the first paired sample data.

[0019] The first loss of the above expression recognition model is determined based on the mixed features corresponding to each first paired sample data in the first paired sample dataset.

[0020] In one possible implementation, determining the first loss of the expression recognition model based on the mixed features corresponding to each first paired sample data in the first paired sample dataset includes:

[0021] Based on the mixed features corresponding to each first paired sample data in the first paired sample dataset, predict the mixed expression probability corresponding to each first paired sample data in the first paired sample dataset.

[0022] The first loss of the above expression recognition model is determined based on the above mixed expression probability and the expression labels corresponding to each first sample image in the above first paired sample data.

[0023] In one possible implementation, determining the second loss of the expression recognition model based on the first sample dataset and the second sample dataset includes:

[0024] Based on the first sample dataset mentioned above, the target threshold set for the above expression recognition model is determined;

[0025] The second loss of the above expression recognition model is determined based on the above target threshold set and the above second sample dataset.

[0026] In one possible implementation, determining the target threshold set for the expression recognition model based on the first sample dataset includes:

[0027] The first feature of each first sample image in the first sample dataset is extracted using the aforementioned backbone network.

[0028] The first feature of each first sample image in the first sample dataset is input into the expression feature generation network, and the first expression feature vector corresponding to each first sample image in the first sample dataset is output.

[0029] Based on the first expression feature vector corresponding to each first sample image in the first sample dataset, predict the first expression recognition result corresponding to each first sample image in the first sample dataset.

[0030] Based on the first expression recognition results corresponding to each first sample image in the first sample dataset and the expression labels, the target threshold set of the expression recognition model is determined.

[0031] In one possible implementation, determining the second loss of the expression recognition model based on the target threshold set and the second sample dataset includes:

[0032] The second feature of each second sample image in the second sample dataset is extracted using the aforementioned backbone network.

[0033] The second features of each second sample image in the second sample dataset are input into the expression feature generation network, and the second expression feature vector corresponding to each second sample image in the second sample dataset is output.

[0034] Based on the second expression feature vector corresponding to each second sample image in the second sample dataset, predict the second expression recognition result corresponding to each second sample image in the second sample dataset.

[0035] Based on the target threshold set of the above expression recognition model and the second expression recognition results corresponding to each second sample image in the above second sample dataset, the second sample images in the above second sample dataset are labeled to obtain the sample training dataset of the above target domain.

[0036] The second loss of the above expression recognition model is determined based on the above sample training dataset.

[0037] In one possible implementation, the second expression recognition result includes the classification probability of each expression category corresponding to the second sample image; the target threshold set includes at least one target threshold corresponding to an expression category.

[0038] The above-mentioned second sample images in the second sample dataset are labeled based on the target threshold set of the above-mentioned expression recognition model and the second expression recognition results corresponding to each second sample image in the above-mentioned second sample dataset, to obtain the sample training dataset of the above-mentioned target domain, including:

[0039] Based on the second expression recognition results corresponding to each second sample image in the second sample dataset and the target threshold set, the target second sample image in the second sample dataset is determined; the target second sample image is used to represent the second sample image corresponding to the second expression recognition result whose maximum classification probability is greater than the target threshold of the expression category corresponding to the maximum classification probability.

[0040] Based on the expression category corresponding to the maximum classification probability of the target second sample image in the aforementioned second sample dataset, the target second sample image in the aforementioned second sample dataset is labeled to obtain the sample training dataset of the aforementioned target domain.

[0041] In one possible implementation, determining the second loss of the expression recognition model based on the target threshold set and the second sample dataset includes:

[0042] The second sample images in the second sample dataset above are subjected to a first transformation to obtain a first transformed sample set;

[0043] The second sample image in the second sample dataset is subjected to a second transformation to obtain a second transformed sample set; the change of the second sample image by the second transformation is greater than the change of the second sample image by the first transformation.

[0044] Based on the target threshold set of the above expression recognition model and the first transformation sample set, the second transformed second sample image in the second transformation sample set is labeled to obtain the sample training dataset of the above target domain.

[0045] The second loss of the above expression recognition model is determined based on the above sample training dataset.

[0046] In one possible implementation, the first transformation includes at least one of rotation transformation, translation transformation, image flipping transformation, and affine transformation; the second transformation includes random erasure transformation of image regions.

[0047] Secondly, embodiments of this specification provide an expression recognition method, including:

[0048] Obtain the target user's facial image;

[0049] The target facial image is input into the expression recognition model, and the target expression recognition result corresponding to the target user is output. The expression recognition model is trained based on the expression recognition model training method provided by the first aspect or any possible implementation of the first aspect of the embodiments of this specification.

[0050] Thirdly, embodiments of this specification provide an expression recognition model training device, which includes:

[0051] The acquisition module is used to acquire a first sample dataset and a second sample dataset; the first sample dataset includes multiple first sample images with expression labels in the source domain; the second sample dataset includes multiple second sample images without expression labels in the target domain.

[0052] The determining module is used to determine a first paired sample dataset based on the aforementioned first sample dataset; the aforementioned first paired sample dataset includes at least one set of first paired sample data; the aforementioned first paired sample data includes at least two first sample images from the aforementioned first sample dataset;

[0053] The training module is used to train the facial expression recognition model based on the first sample dataset, the first paired sample dataset, and the second sample dataset mentioned above.

[0054] In one possible implementation, the training module described above includes:

[0055] The first determining unit is used to determine the first loss of the expression recognition model based on the aforementioned first paired sample dataset.

[0056] The second determining unit is used to determine the second loss of the above-mentioned expression recognition model based on the above-mentioned first sample dataset and the above-mentioned second sample dataset;

[0057] The training unit is used to train the above-mentioned expression recognition model based on the first loss and the second loss.

[0058] In one possible implementation, the above-mentioned facial expression recognition model includes a backbone network, an uncertainty extraction network, and an facial expression feature generation network;

[0059] The aforementioned first determining unit includes:

[0060] The first feature extraction subunit is used to extract the first feature of each first sample image in the first paired sample data using the backbone network described above.

[0061] The uncertainty extraction subunit is used to input the first features of each first sample image in the first paired sample data into the uncertainty extraction network and output the uncertainty vector corresponding to each first sample image in the first paired sample data.

[0062] The first facial expression feature extraction subunit is used to input the first features of each first sample image in the first paired sample data into the facial expression feature generation network and output the first facial expression feature vector corresponding to each first sample image in the first paired sample data.

[0063] The fusion subunit is used to fuse the uncertainty vector and the first expression feature vector to obtain the mixed features corresponding to the first paired sample data.

[0064] The first determining subunit is used to determine the first loss of the above expression recognition model based on the mixed features corresponding to each first paired sample data in the above first paired sample dataset.

[0065] In one possible implementation, the aforementioned determining sub-unit is specifically used for:

[0066] Based on the mixed features corresponding to each first paired sample data in the first paired sample dataset, predict the mixed expression probability corresponding to each first paired sample data in the first paired sample dataset; based on the mixed expression probability and the expression label corresponding to each first sample image in the first paired sample dataset, determine the first loss of the expression recognition model.

[0067] In one possible implementation, the second determining unit includes:

[0068] The second determining subunit is used to determine the target threshold set of the above-mentioned expression recognition model based on the first sample dataset mentioned above.

[0069] The third determining subunit is used to determine the second loss of the above expression recognition model based on the above target threshold set and the above second sample dataset.

[0070] In one possible implementation, the second determining subunit is specifically used for:

[0071] The first feature of each first sample image in the first sample dataset is extracted using the backbone network described above; the first feature of each first sample image in the first sample dataset is input into the expression feature generation network, and the first expression feature vector corresponding to each first sample image in the first sample dataset is output; the first expression recognition result corresponding to each first sample image in the first sample dataset is predicted based on the first expression feature vector corresponding to each first sample image in the first sample dataset; the target threshold set of the expression recognition model is determined based on the first expression recognition result corresponding to each first sample image in the first sample dataset and the expression label.

[0072] In one possible implementation, the aforementioned third determining subunit is specifically used for:

[0073] The second features of each second sample image in the second sample dataset are extracted using the backbone network described above; the second features of each second sample image in the second sample dataset are input into the expression feature generation network, and the second expression feature vector corresponding to each second sample image in the second sample dataset is output; the second expression recognition result corresponding to each second sample image in the second sample dataset is predicted based on the second expression feature vector corresponding to each second sample image in the second sample dataset; the second sample images in the second sample dataset are labeled based on the target threshold set of the expression recognition model and the second expression recognition result corresponding to each second sample image in the second sample dataset, thereby obtaining the sample training dataset of the target domain; and the second loss of the expression recognition model is determined based on the sample training dataset.

[0074] In one possible implementation, the second expression recognition result includes the classification probability of each expression category corresponding to the second sample image; the target threshold set includes at least one target threshold corresponding to an expression category.

[0075] The third determining subunit, when used to annotate the second sample images in the second sample dataset based on the target threshold set of the expression recognition model and the second expression recognition results corresponding to each second sample image in the second sample dataset, to obtain the sample training dataset for the target domain, is specifically used for:

[0076] Based on the second expression recognition results corresponding to each second sample image in the second sample dataset and the target threshold set, the target second sample image in the second sample dataset is determined; the target second sample image is used to represent the second sample image corresponding to the second expression recognition result whose maximum classification probability is greater than the target threshold of the expression category corresponding to the maximum classification probability; the target second sample image in the second sample dataset is labeled based on the expression category corresponding to the maximum classification probability of the target second sample image in the second sample dataset to obtain the sample training dataset of the target domain.

[0077] In one possible implementation, the aforementioned third determining subunit is specifically used for:

[0078] The second sample images in the second sample dataset are subjected to a first transformation to obtain a first transformed sample set; the second sample images in the second sample dataset are subjected to a second transformation to obtain a second transformed sample set; the change magnitude of the second transformation on the second sample images is greater than the change magnitude of the first transformation on the second sample images; the second transformed second sample images in the second transformed sample set are labeled based on the target threshold set of the expression recognition model and the first transformed sample set to obtain the sample training dataset of the target domain; the second loss of the expression recognition model is determined based on the sample training dataset.

[0079] In one possible implementation, the first transformation includes at least one of rotation transformation, translation transformation, image flipping transformation, and affine transformation; the second transformation includes random erasure transformation of image regions.

[0080] Fourthly, embodiments of this specification provide an expression recognition device, which includes:

[0081] The acquisition module is used to acquire the target user's facial image.

[0082] The expression recognition module is used to input the target face image into the expression recognition model and output the target expression recognition result corresponding to the target user; the expression recognition model is trained based on the expression recognition model training method provided by the first aspect of the embodiments of this specification or any possible implementation of the first aspect.

[0083] Fifthly, embodiments of this specification provide an electronic device, including: a processor and a memory;

[0084] The processor is connected to the memory.

[0085] The aforementioned memory is used to store executable program code;

[0086] The processor reads the executable program code stored in the memory to run the program corresponding to the executable program code, so as to execute the method provided by the first aspect or any possible implementation of the first aspect or the second aspect or any possible implementation of the embodiments of this specification.

[0087] Sixthly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method provided by the first aspect or any possible implementation of the first aspect or the second aspect or any possible implementation of the second aspect of the embodiments of this specification.

[0088] In a seventh aspect, embodiments of this specification provide a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to perform the method provided by the first aspect or any possible implementation of the first aspect or the second aspect or any possible implementation of the second aspect of the embodiments of this specification.

[0089] This embodiment first obtains a first sample dataset and a second sample dataset. The first sample dataset includes multiple first sample images with expression labels in the source domain, and the second sample dataset includes multiple second sample images without expression labels in the target domain. Then, a first paired sample dataset is determined based on the first sample dataset. The first paired sample dataset includes at least one set of first paired sample data, which includes at least two first sample images from the first sample dataset. Finally, an expression recognition model is trained based on the first sample dataset, the first paired sample dataset, and the second sample dataset. On the one hand, the embodiments of this specification train the network in the expression recognition model by using first paired sample data including at least two first sample images for comparative learning. This allows the expression recognition model to intuitively learn the differences between different first sample images, avoiding unclear or incorrect learning due to similar expression features in first sample images with different expression labels. This enhances the training effect of the expression recognition model, thereby improving its robustness in recognizing ambiguous expression categories and increasing its accuracy. On the other hand, the embodiments of this specification also train the expression recognition model by semi-supervised sample mining using a first sample dataset with expression labels in the source domain and a second sample dataset without expression labels in the target domain. This avoids the situation where there are too few samples with expression labels in the target domain or the annotation data takes too long, resulting in the inability to efficiently and accurately achieve expression recognition in the target domain in a short time. This solves the cold start problem in the target domain when training the expression recognition model and enhances the domain generalization ability of the expression recognition model. Attached Figure Description

[0090] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0091] Figure 1 This is a schematic diagram illustrating the implementation process of a facial expression recognition model training method provided in related technologies.

[0092] Figure 2 This specification provides an exemplary embodiment of a facial expression recognition model training method and a schematic diagram of the implementation environment of the facial expression recognition method.

[0093] Figure 3 A flowchart illustrating an exemplary embodiment of this specification for training an expression recognition model;

[0094] Figure 4 A schematic diagram illustrating the specific implementation process of an expression recognition model training method provided in this specification, as an exemplary embodiment.

[0095] Figure 5 A schematic diagram illustrating the implementation process of an expression recognition model training method provided as an exemplary embodiment of this specification;

[0096] Figure 6 A schematic diagram illustrating the implementation process of determining a second loss for an expression recognition model, provided as an exemplary embodiment of this specification;

[0097] Figure 7 This is a schematic diagram illustrating an implementation process for determining a sample training dataset for a target domain, provided as an exemplary embodiment of this specification.

[0098] Figure 8 A schematic diagram illustrating a specific implementation process for determining the second loss of an expression recognition model, provided as an exemplary embodiment of this specification;

[0099] Figure 9 A schematic diagram illustrating another implementation process for determining a sample training dataset for a target domain, provided as an exemplary embodiment of this specification;

[0100] Figure 10 A schematic diagram illustrating the implementation process of another facial expression recognition model training method provided as an exemplary embodiment of this specification;

[0101] Figure 11 A flowchart illustrating an exemplary embodiment of this specification for a facial expression recognition method;

[0102] Figure 12 A schematic diagram of the structure of an expression recognition model training device provided as an exemplary embodiment of this specification;

[0103] Figure 13 A schematic diagram of the structure of an expression recognition device provided for an exemplary embodiment of this specification;

[0104] Figure 14 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0105] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings.

[0106] The terms "first," "second," "third," etc., used in this specification, claims, and the foregoing drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0107] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, facial images, sample images, etc. involved in this specification were obtained with full authorization.

[0108] In related technologies, such as Figure 1As shown, after training an expression recognition model using a first sample set with expression labels in the source domain, if the application scenario of expression recognition changes, to ensure the accuracy of the expression recognition model, it is necessary to manually label the second sample images in the second sample set without expression labels in the target domain corresponding to the changed application scenario. Then, the expression recognition model is trained again using the labeled second sample set. That is, different domains require labeled sample sets corresponding to those domains to train the expression recognition model in order to ensure the accuracy of expression recognition in each domain. Therefore, the domain generalization ability of expression recognition models in related technologies is poor. When expression recognition is required in a target domain with too few or no expression labels, the time consumption of manually labeling data will be very long, which will lead to the inability to efficiently and accurately achieve expression recognition in the target domain in a short period of time. At the same time, the expression recognition model trained by the training method in related technologies also has difficulty accurately identifying ambiguous expression categories in images.

[0109] Please refer to the following. Figure 2 , Figure 2 This diagram illustrates an exemplary embodiment of the facial expression recognition model training method and the implementation environment of the facial expression recognition method. Figure 1 As shown, the implementation environment includes an image acquisition device 210 and a server 220. Wherein:

[0110] The image acquisition device 210 can be a mobile phone, tablet computer, laptop computer or other device with user software and camera installed, or a camera or other device with camera installed. This specification does not limit this.

[0111] Optionally, when performing expression recognition in the target domain, the image acquisition device 210 can first acquire a second sample dataset of the target domain, which includes multiple second sample images of the target domain without expression labels. Simultaneously, the image acquisition device 210 can also acquire a first sample dataset via a network, which includes multiple first sample images of the source domain with expression labels. After acquiring the first and second sample datasets, the image acquisition device 210 can train an expression recognition model using the expression recognition model training method provided in the embodiments of this specification. After training the expression recognition model, the image acquisition device 210 can also acquire a target face image of the target user, and then obtain the target expression recognition result corresponding to the target user using the expression recognition method provided in the embodiments of this specification.

[0112] Optionally, after acquiring data such as the second sample dataset of the target domain and the target face image of the target user through the image acquisition device 210, the image acquisition device 210 can also establish a data relationship with the network and establish a data connection relationship with the server 220 through the network. For example, it can send data such as the second sample dataset of the target domain and the target face image of the target user, and receive the expression recognition model trained by the server 220 based on the second sample dataset and the target expression recognition result obtained after expression recognition based on the target face image of the target user.

[0113] Server 220 can be a server capable of providing training for multiple facial expression recognition models. It can first receive data such as a first sample dataset and a second sample dataset sent by image acquisition device 210 or other devices via a network, and then train the facial expression recognition model using the facial expression recognition model training method provided in the embodiments of this specification. Server 220 can also be a server capable of providing multiple facial expression recognition capabilities. It can receive data such as target facial images of the target user sent by image acquisition device 210 or other devices via a network. After training the facial expression recognition model, it can obtain the target facial expression recognition result corresponding to the target user using the facial expression recognition method provided in the embodiments of this specification.

[0114] Specifically, server 220 can be, but is not limited to, a hardware server, a virtual server, a cloud server, etc.

[0115] The network can be a medium that provides a communication link between server 220 and any image acquisition device 210, or it can be the Internet, which includes network devices and transmission media, and is not limited thereto. The transmission media can be a wired link, such as, but not limited to, coaxial cable, fiber optic cable, and digital subscriber line (DSL), or a wireless link, such as, but not limited to, wireless fidelity (WIFI), Bluetooth, and mobile device networks.

[0116] Understandably, Figure 2 The number of image acquisition devices 210 and servers 220 in the illustrated implementation environment is merely an example. In a specific implementation, this environment can contain any number of image acquisition devices and servers. This specification does not specifically limit this. For example, but not limited to, image acquisition device 210 can be an image acquisition device cluster composed of multiple image acquisition devices, and server 220 can be a server cluster composed of multiple servers.

[0117] To address the problems in the aforementioned related technologies, the following will combine... Figure 2 Taking the training of an facial expression recognition model on server 220 as an example, this document introduces the facial expression recognition model training method provided in the embodiments. Please refer to the following for details. Figure 3 This is a flowchart illustrating an exemplary embodiment of an facial expression recognition model training method provided in this specification. Figure 3 As shown, the training method for this facial expression recognition model includes the following steps:

[0118] S302, obtain the first sample dataset and the second sample dataset. The first sample dataset includes multiple first sample images with expression labels in the source domain, and the second sample dataset includes multiple second sample images without expression labels in the target domain.

[0119] Specifically, the first sample image may include, but is not limited to, facial images of users performing face recognition or expression recognition in the source domain, and the second sample image may include, but is not limited to, facial images of users performing face recognition or expression recognition in the target domain. The source domain is used to represent the domain or application scenario in which the first sample dataset is located, and the target domain can be used to represent the domain or application scenario in which the second sample dataset is located. The first and second sample datasets may be data from different domains, or data from the same domain at different time periods, etc., and this specification does not limit this. The expression labels may include, but are not limited to, expression labels corresponding to expression categories such as happy, surprised, sad, angry, disgusted, fearful, and neutral. For ease of annotation and training, the expression labels may use numbers or letters to represent expression categories, for example, but not limited to, using expression label 1 to represent the expression category "happy", expression label 2 to represent the expression category "surprised", etc.

[0120] S304, determine a first paired sample dataset based on the first sample dataset, the first paired sample dataset including at least one set of first paired sample data, the first paired sample data including at least two first sample images in the first sample dataset.

[0121] Specifically, after obtaining the first sample dataset, multiple first sample images in the first sample dataset can be randomly paired to obtain a first paired sample dataset including at least one set of first paired sample data. The random pairing can be pairing the multiple first sample images in pairs, or pairing every three first sample images, etc., and this specification does not limit the specific pairing.

[0122] S306, an expression recognition model is trained based on the first sample dataset, the first paired sample dataset, and the second sample dataset.

[0123] This embodiment first obtains a first sample dataset and a second sample dataset. The first sample dataset includes multiple first sample images with expression labels in the source domain, and the second sample dataset includes multiple second sample images without expression labels in the target domain. Then, a first paired sample dataset is determined based on the first sample dataset. The first paired sample dataset includes at least one set of first paired sample data, which includes at least two first sample images from the first sample dataset. Finally, an expression recognition model is trained based on the first sample dataset, the first paired sample dataset, and the second sample dataset. On the one hand, the embodiments of this specification train the network in the expression recognition model by using first paired sample data including at least two first sample images for comparative learning. This allows the expression recognition model to intuitively learn the differences between different first sample images, avoiding unclear or incorrect learning due to similar expression features in first sample images with different expression labels. This enhances the training effect of the expression recognition model, thereby improving its robustness in recognizing ambiguous expression categories and increasing its accuracy. On the other hand, the embodiments of this specification also train the expression recognition model by semi-supervised sample mining using a first sample dataset with expression labels in the source domain and a second sample dataset without expression labels in the target domain. This avoids the situation where there are too few samples with expression labels in the target domain or the annotation data takes too long, resulting in the inability to efficiently and accurately achieve expression recognition in the target domain in a short time. This solves the cold start problem in the target domain when training the expression recognition model and enhances the domain generalization ability of the expression recognition model.

[0124] Specifically, such as Figure 4 As shown, the specific implementation process of training the facial expression recognition model in S306 above may include the following steps:

[0125] S402, determine the first loss of the facial expression recognition model based on the first paired sample dataset.

[0126] Specifically, such as Figure 5As shown, the facial expression recognition model includes a backbone network, an uncertainty extraction network, and an expression feature generation network. After determining the first paired sample dataset, the backbone network of the facial expression recognition model can be used to extract the first features of each first sample image in the first paired sample data. Then, the first features of each first sample image in the first paired sample data are input into the uncertainty extraction network, which outputs the uncertainty vector corresponding to each first sample image in the first paired sample data. The first features of each first sample image in the first paired sample data are also input into the expression feature generation network, which outputs the first expression feature vector corresponding to each first sample image in the first paired sample data. The backbone network described above can be a commonly used recognition network, such as ResNet or ShuffleNetV2, and this embodiment does not limit it. After obtaining the uncertainty vector and the first expression feature vector of each first sample image in the first paired sample data, the uncertainty vector and the first expression feature vector can be fused to obtain the mixed features corresponding to the first paired sample data. Finally, the first loss of the expression recognition model is determined based on the mixed features corresponding to each first paired sample data in the first paired sample dataset. Thus, the uncertainty extraction network of the expression recognition model can be used to perform robust comparative learning and judgment on the fuzzy expression categories in the first paired sample data to ensure the robustness of the expression recognition model. At the same time, the expression feature generation network of the expression recognition model can be used to learn the expression features in each first sample image to ensure the accuracy of the expression recognition model.

[0127] Furthermore, when determining the first loss of the expression recognition model based on the mixed features corresponding to each first paired sample data in the first paired sample dataset, such as... Figure 5 As shown, the probability of mixed expression corresponding to each first paired sample data in the first paired sample dataset can be predicted first based on the mixed features corresponding to each first paired sample data in the first paired sample dataset. Then, the first loss of the expression recognition model can be determined based on the above mixed expression probability and the expression label corresponding to each first sample image in the first paired sample dataset.

[0128] For example, such as Figure 5 As shown, if the uncertainty vector corresponding to the first sample image i in the first paired sample data is δ i Its corresponding first expression feature vector is The uncertainty vector corresponding to the first sample image j paired with the first sample image i in the first paired sample data is δ. j Its corresponding first expression feature vector is Then the mixed features corresponding to the first paired sample data can be calculated. in, 1≤i≤N, 1≤j≤N and j≠i, Mean is used to scalarize the vector in parentheses, that is, to scalarize the vector. and The various dimensions are averaged to form a scalar. The mixture feature U corresponding to each paired sample data in the first paired sample dataset is then obtained. ij Then, based on the above-mentioned hybrid characteristics, Calculate the first loss of the facial expression recognition model Where N is an integer greater than or equal to 2, representing the total number of the first sample images; y i The expression label (expression category) corresponding to the first sample image i; For emoji tag y i The corresponding weight vector, For the expression label y predicted based on mixed features i The corresponding probability of mixed expressions; For emoji tag y j The corresponding weight vector, For the expression label y predicted based on mixed features j The corresponding mixed expression probability; C is the set of all expression categories recognized by the expression recognition model; c is the expression category predicted by the expression recognition model; W c This refers to the weight vector used by the facial expression recognition model to predict the classification probability of facial expression category c. The weight vector in the first loss L1 can be continuously updated as the facial expression recognition model is trained.

[0129] Please continue to refer to the following. Figure 4 ,like Figure 4 As shown, the implementation process of training the facial expression recognition model in S306 above also includes:

[0130] S404, determine the second loss of the facial expression recognition model based on the first sample dataset and the second sample dataset.

[0131] Understandably, S402 and S404 can be executed simultaneously or sequentially, and the embodiments in this specification do not limit this.

[0132] Specifically, such as Figure 6 As shown, the implementation process of determining the second loss of the facial expression recognition model in S404 above may include the following steps:

[0133] S602, determine the target threshold set for the facial expression recognition model based on the first sample dataset.

[0134] Specifically, the aforementioned target threshold set includes target thresholds corresponding to at least one expression category. The target thresholds corresponding to each expression category within the at least one expression category may be the same or different; this specification does not limit this. For example... Figure 5 As shown, after obtaining the first sample dataset, the backbone network can be used to extract the first features of each first sample image in the first sample dataset. Then, the first features of each first sample image in the first sample dataset are input into the expression feature generation network, which outputs the first expression feature vector corresponding to each first sample image in the first sample dataset. Based on the first expression feature vector corresponding to each first sample image in the first sample dataset, the first expression recognition result corresponding to each first sample image in the first sample dataset is predicted. Finally, based on the first expression recognition result corresponding to each first sample image in the first sample dataset and the expression label, the target threshold set of the expression recognition model is determined.

[0135] Furthermore, the number of target thresholds in the aforementioned target threshold set, i.e., the types of expression categories, corresponds to the types of expression labels in the first sample dataset. The first expression recognition result corresponding to each first sample image in the aforementioned first sample dataset includes the classification probability of at least one expression category corresponding to the user in each first sample image. The number of expression categories in the aforementioned first expression recognition result corresponds to the types of expression labels in the first sample dataset. After obtaining the first expression recognition result corresponding to each first sample image in the first sample dataset, a target classification probability mean set can be calculated based on the first expression recognition result corresponding to each first sample image in the first sample dataset, and then a target threshold set can be determined based on the target classification probability mean set. The aforementioned target classification probability mean set includes the target classification probability mean among the first sample images corresponding to various expression labels in the first sample dataset, and the aforementioned target classification probability mean corresponds to the expression category corresponding to the expression label of the first sample image. That is, after obtaining the first expression recognition result corresponding to each first sample image in the first sample dataset, the mean of the classification probability of the expression category corresponding to the expression label in the first expression recognition result of the first sample image with the same expression label in the first sample dataset can be calculated first, thereby obtaining the target classification probability mean set. Then, the mean target classification probability of each expression category in the above target classification probability mean set can be directly determined as the target threshold for each expression category when the expression recognition model performs expression recognition in the next time. Thus, the second sample image in the second sample dataset of the target domain can be labeled by determining the target threshold of each expression category based on the first sample dataset of the source domain. This allows the expression recognition model to automatically obtain the sample dataset with expression labels in the target domain, which not only solves the cold start problem of data and improves the efficiency of the expression recognition model in being applied to the target domain, but also enhances the domain generalization ability of the expression recognition model and realizes the autonomous cross-domain learning of the expression recognition model.

[0136] Optionally, to avoid the problem that the expression recognition model's performance is poor in the early stages of training, resulting in low classification probability values ​​for the actual expression labels of the first sample images during recognition, i.e., low accuracy of the expression recognition model, and consequently, excessively low target thresholds for each expression category, thus affecting the accuracy of the second sample image annotations in the target domain, after obtaining the first expression recognition results corresponding to each first sample image in the first sample dataset, a target threshold set can be determined based on the first expression recognition results corresponding to each first sample image in the first sample dataset and a preset threshold set. That is, firstly, the target classification probability mean set is calculated based on the first expression recognition results corresponding to each first sample image in the first sample dataset, and then the target threshold set is determined based on the target classification probability mean set and the aforementioned preset threshold set. The aforementioned preset threshold set includes preset thresholds corresponding to at least one expression category. In the embodiments of this specification, different preset thresholds may be set for each of the at least one expression category, or the same preset threshold may be set for each expression category; this embodiment of the specification does not limit this. The aforementioned determination of the target threshold set based on the mean set of target classification probabilities and the aforementioned preset threshold set can be achieved by comparing the mean target classification probability of each expression category with the preset threshold. If the mean target classification probability of the expression category is greater than or equal to the preset threshold, then the mean target classification probability is determined as the target threshold corresponding to that expression category; if the mean target classification probability of the expression category is less than the preset threshold, then the preset threshold is determined as the target threshold corresponding to that expression category. In this way, by pre-setting the preset thresholds corresponding to each expression category, the annotation quality of the second sample images without expression labels in the target domain can be controlled to a certain extent. Only when the mean target classification probability is greater than or equal to the preset threshold of its corresponding expression category will the mean target classification probability be determined as the corresponding target threshold to mine the second sample images in the second sample dataset that can be annotated. This avoids the situation where the annotation accuracy of the second sample images in the target domain is low, which affects the training effect of the expression recognition model and ensures the effect of cross-domain learning of the expression recognition model.

[0137] S604, determine the second loss of the facial expression recognition model based on the target threshold set and the second sample dataset.

[0138] Optionally, such as Figure 5As shown, after obtaining the second sample dataset and determining the target threshold set of the expression recognition model, the second features of each second sample image in the second sample dataset can be extracted using the backbone network. The second features of each second sample image in the second sample dataset are then input into the expression feature generation network, which outputs the second expression feature vector corresponding to each second sample image in the second sample dataset. Based on the second expression feature vector corresponding to each second sample image in the second sample dataset, the second expression recognition result corresponding to each second sample image in the second sample dataset is predicted. Finally, based on the target threshold set of the expression recognition model and the second expression recognition result corresponding to each second sample image in the second sample dataset, the second sample images in the second sample dataset are labeled to obtain the sample training dataset of the target domain. Thus, the second loss of the expression recognition model can be determined based on the sample training dataset.

[0139] Furthermore, the aforementioned second expression recognition result includes the classification probability of each expression category corresponding to the second sample image, and the aforementioned target threshold set includes a target threshold corresponding to at least one expression category. When labeling the second sample images in the second sample dataset based on the target threshold set of the expression recognition model and the second expression recognition results corresponding to each second sample image in the second sample dataset to obtain the sample training dataset for the target domain, the target second sample images in the second sample dataset can be determined first based on the second expression recognition results corresponding to each second sample image in the second sample dataset and the target threshold set. Then, the target second sample images in the second sample dataset can be labeled based on the expression category corresponding to the maximum classification probability of the target second sample images in the second sample dataset to obtain the sample training dataset for the target domain. The aforementioned target second sample images are used to characterize the second sample images corresponding to the second expression recognition results whose maximum classification probability is greater than the target threshold of the expression category corresponding to the maximum classification probability.

[0140] In other words, after determining the target thresholds (i.e., the target threshold set) for each expression category and the second expression recognition results for each second sample image in the second sample dataset, we can first determine the maximum classification probability in the second expression recognition results corresponding to the second sample image. Then, we compare this maximum classification probability with the target threshold corresponding to its expression category. Finally, we determine the second sample images in the second sample dataset whose maximum classification probability is greater than the target threshold of their corresponding expression category as target second sample images. In this way, we can select target second sample images in the second sample dataset that can guarantee the labeling quality through the above process. This avoids the problem that directly labeling all second sample images with the expression category corresponding to the maximum classification probability in the second expression recognition results would result in poor labeling quality of the training dataset of samples with expression labels in the target domain, thus affecting the cross-domain learning and training effect of the expression recognition model.

[0141] Furthermore, after identifying the target second sample image that can be labeled in the second sample dataset, the expression label corresponding to the expression category with the highest classification probability in the second expression recognition result of the target second sample image can be directly labeled as the expression label of the target second sample image. Thus, there is no need to manually label the second sample image in the target domain. The training dataset of samples with expression labels in the target domain can be automatically obtained through domain adaptive semi-supervised learning, which reduces the consumption of human resources in the training process of the expression recognition model, solves the cold start problem of the expression recognition model in the target domain, and improves the training efficiency of the expression recognition model.

[0142] For example, such as Figure 7 As shown, if the second sample dataset includes second sample image A, second sample image B, and second sample image C without expression labels in the target domain, and the second expression recognition results corresponding to each second sample image in the second sample dataset include as follows: Figure 7 The classification probabilities of each expression category (corresponding to expression label) shown can be compared with the target threshold 710 corresponding to expression category 1 (corresponding to expression label 1) of each second sample image, and the classification probabilities of each second sample image corresponding to expression category 2 (corresponding to expression label 2) can be compared with the target threshold 720 corresponding to expression category 2 (corresponding to expression label 2). Then, the second sample image B with a classification probability of expression category 1 (corresponding to expression label 1) greater than the target threshold 710 is labeled with expression label 1, and the second sample image C with a classification probability of expression category 2 (corresponding to expression label 2) greater than the target threshold 720 is labeled with expression label 2, thereby obtaining the sample training dataset of the target domain. The above sample training dataset includes the second sample image B with expression label 1 and the second sample image C with expression label 2 in the target domain.

[0143] For example, after obtaining the sample training dataset for the target domain, the aforementioned sample training dataset can be input into the facial expression recognition model, and then... Figure 5 The backbone network and expression feature generation network of the expression recognition model shown above obtain the expression features of the second sample images k, each with expression labels, from the above training dataset. Then, based directly on the facial expression features of the second sample image k mentioned above... and the expression label corresponding to the second sample image k. Calculate the second loss of the facial expression recognition model in, emoji tags The corresponding weight vector, For facial expression features based on the second sample image k Predicted emoji tags The corresponding expression classification probability; K is an integer greater than or equal to 1 and less than or equal to N, used to represent the total number of second sample images with expression labels included in the sample training dataset.

[0144] Optionally, after determining the target threshold set, in addition to... Figure 7 The method shown involves labeling the second sample images in the second sample dataset based on the target threshold set of the expression recognition model and the second expression recognition results corresponding to each second sample image in the second sample dataset, thus obtaining a sample training dataset for the target domain. In addition to determining the second loss of the expression recognition model based on this sample training dataset, to enhance the diversity of the sample training dataset in the target domain, the second sample images in the second sample dataset can be transformed first. Then, the transformed second sample images in the second sample dataset are labeled based on the target threshold set of the expression recognition model, obtaining a sample training dataset. Finally, the second loss of the expression recognition model is determined based on the aforementioned sample training dataset. Specifically, as follows... Figure 8 As shown, the implementation process of determining the second loss of the expression recognition model based on the target threshold set and the second sample dataset in S604 above may include the following steps:

[0145] S802, perform a first transformation on the second sample image in the second sample dataset to obtain a first transformed sample set.

[0146] Specifically, the aforementioned first transformation can be called a weak transformation, and may include, but is not limited to, at least one of rotation transformation, translation transformation, image flipping transformation, and affine transformation. That is, the aforementioned first transformation sample set includes multiple first transformation sample images of the target domain expressionless labels obtained by applying the first transformation to multiple second sample images of the target domain expressionless labels in the second sample dataset. The number of the aforementioned first transformation sample sets may include one or more, meaning that multiple first transformations can be applied to the second sample images in the second sample dataset to obtain multiple corresponding first transformation sample sets.

[0147] For example, after obtaining the second sample dataset, each second sample image in the second sample dataset can be rotated, and then an affine transformation can be performed on each rotated second sample image in the second sample dataset to obtain the first transformed sample set. Simultaneously, after obtaining the second sample dataset, each second sample image in the second sample dataset can also be translated, and then an image flip transformation can be performed on each translated second sample image in the second sample dataset to obtain the second transformed sample set.

[0148] S804, perform a second transformation on the second sample image in the second sample dataset to obtain the second transformed sample set.

[0149] Specifically, the aforementioned second transformation can be called a strong transformation, which may include, but is not limited to, image region random erasure transformation, or a combination of image region random erasure transformation and at least one of weak transformations such as rotation transformation, translation transformation, image flipping transformation, and affine transformation. That is, the aforementioned second transformation sample set includes multiple second-transformed sample images of the target domain without expression labels obtained by applying the second transformation to multiple second-transformed sample images of the target domain without expression labels in the second sample dataset. The magnitude of change of the second sample image by the aforementioned second transformation is greater than the magnitude of change of the second sample image by the first transformation.

[0150] Understandably, S802 and S804 can be executed simultaneously or sequentially, and the embodiments in this specification do not limit this.

[0151] Understandably, in order to enhance the diversity of training samples in the target domain, the number of the second transformation sample sets can be multiple. That is, the second sample images in the second sample dataset can be subjected to various different second transformations to obtain multiple corresponding second transformation sample sets.

[0152] S806, based on the target threshold set of the expression recognition model and the first transformation sample set, the second transformed second sample image in the second transformation sample set is labeled to obtain the sample training dataset of the target domain.

[0153] Specifically, after obtaining the first transformed sample set and the second sample dataset, the first transformed sample set can be input into the expression recognition model to output the expression recognition results corresponding to each first transformed sample image in the first transformed sample set. Then, based on the expression recognition results corresponding to each first transformed sample image in the first transformed sample set, the second transformed sample images in the second transformed sample set are labeled. That is, the target first transformed sample images in the first transformed sample set with the maximum classification probability greater than the target threshold of their corresponding expression category are first selected, and then the target second transformed sample images corresponding to the above target first transformed sample images in the second transformed sample set are selected, that is, the target second transformed sample images in the second transformed sample set after the second transformation. Finally, the expression label corresponding to the expression category with the maximum classification probability in the expression recognition results corresponding to the target first transformed sample image is labeled as the expression label of the target second transformed sample image corresponding to the target first transformed sample image, thus obtaining the sample training dataset.

[0154] This embodiment first performs strong and weak transformations on the second sample dataset of the target domain. Then, based on the target threshold set, it labels the strong transformation samples (second transformation sample set) with weak transformation samples (first transformation sample set). This not only selects the target second sample images with guaranteed labeling quality from the second transformation sample set, ensuring the labeling quality of the sample training dataset used for training in the target domain, but also allows the target second transformation sample images with expression labels to be used as the sample training dataset of the target domain, improving the diversity of training samples and training effect of the expression recognition model, making the expression recognition model more robust.

[0155] For example, such as Figure 9 As shown, the second sample dataset includes a second sample image B without a target domain expression label. The second sample image B undergoes two different first transformations to obtain first transformed sample a and first transformed sample b, respectively. The second sample image B undergoes a second transformation to obtain second transformed sample c. The second expression recognition results corresponding to each of the first transformed sample a and first transformed sample b include, as shown... Figure 9 The classification probabilities of each expression category (corresponding to expression labels) can be calculated by first calculating the average classification probability of expression category 1 corresponding to the first transformed sample a and the first transformed sample b, which is the sum of the classification probability of expression category 1 (0.9) of the first transformed sample a and the classification probability of expression category 1 (0.7) of the first transformed sample b, and then dividing by 2, thus obtaining the average classification probability of expression category 1 as 0.8; and then calculating the average classification probability of expression category 2 corresponding to the first transformed sample a and the first transformed sample b, which is the sum of the classification probability of expression category 2 (0.1) of the first transformed sample a and the classification probability of expression category 2 (0.3) of the first transformed sample b, and then dividing by 2, thus obtaining the average classification probability of expression category 2 as 0.2. Then, the average classification probability of expression category 1 (0.8) is compared with the target threshold of expression category 1 (0.75), and the average classification probability of expression category 2 (0.2) is compared with the target threshold of expression category 2 (0.85). The expression label 1 corresponding to expression category 1 when the average classification probability is greater than the target threshold of the corresponding expression category is labeled as the expression label of the second transformed sample c. Thus, the second sample image B with expression label in the target domain of the sample training dataset after the second transformation, i.e., the second transformed sample c with expression label, can be obtained.

[0156] Understandably, Figure 9 The classification probabilities included in the second expression recognition result are not limited to two expression categories. The second expression recognition result may also include classification probabilities corresponding to multiple expression categories such as happy, surprised, sad, angry, disgusted, fearful, and neutral. This specification does not limit this.

[0157] Please continue to refer to the following. Figure 8 ,like Figure 8 As shown in S806, after annotating the second transformed second sample images in the second transformed sample set based on the target threshold set of the expression recognition model and the first transformed sample set to obtain the sample training dataset of the target domain, the implementation process of determining the second loss of the expression recognition model also includes:

[0158] S808 determines the second loss of the facial expression recognition model based on the sample training dataset.

[0159] Specifically, after obtaining the sample training dataset of the target domain, the sample training dataset can be input into the expression recognition model to output the expression recognition results corresponding to each second sample image or each second transformed sample with expression labels in the sample training dataset. Then, the second loss of the expression recognition model is determined based on the expression recognition results corresponding to each second sample image or each second transformed sample with expression labels in the sample training dataset and their corresponding expression labels.

[0160] Please continue to refer to the following. Figure 4 ,like Figure 4 As shown, after determining the first loss and the second loss of the facial expression recognition model, the implementation process of training the facial expression recognition model in S306 above also includes:

[0161] S406, an expression recognition model is trained based on the first loss and the second loss.

[0162] Optionally, after determining the first loss and the second loss of the expression recognition model, the first loss and the second loss can be integrated to obtain the target loss of the expression recognition model. Then, the expression recognition model can be trained by continuously optimizing the target loss using the gradient descent method based on the target loss.

[0163] For example, but not limited to, the first loss L1 and the second loss L2 can be summed to obtain the target loss L = L1 + L2 of the expression recognition model.

[0164] Optionally, after determining the first loss and the second loss of the expression recognition model, the expression recognition model can be trained first based on the first loss and then based on the second loss, or the expression recognition model can be trained first based on the second loss and then based on the first loss. This specification does not limit this.

[0165] Optionally, in the embodiments of this specification, a first loss of the expression recognition model may be determined first, and after training the expression recognition model based on the first loss, a second loss of the expression recognition model may be determined based on the trained expression recognition model and the second sample dataset, and the expression recognition model may be trained again based on the second loss. This process may be repeated until both the first loss and the second loss of the expression recognition model meet the training requirements, at which point the training ends.

[0166] The embodiments in this specification can determine the first loss of the expression recognition model using a first paired sample dataset with expression labels in the source domain, allowing the expression recognition model to intuitively learn the differences between different first sample images, thereby enhancing the robustness of the expression recognition model in recognizing ambiguous expression categories and improving the accuracy of the expression recognition model; and can also determine the second loss of the expression recognition model based on a first sample dataset with expression labels in the source domain and a second sample dataset without expression labels in the target domain, that is, to perform semi-supervised sample mining training on the expression recognition model using a first sample dataset with expression labels in the source domain and a second sample dataset without expression labels in the target domain, thereby enhancing the domain generalization ability of the expression recognition model.

[0167] Please refer to the following. Figure 10 This is a schematic diagram illustrating the specific implementation process of an expression recognition model training method provided in an exemplary embodiment of this specification. Figure 10 As shown, after obtaining the first sample dataset with expression labels in the source domain, the first sample images in the first sample dataset can be paired to obtain the first paired sample dataset. Then, according to... Figure 10 The target threshold set and the first loss L1 of the facial expression recognition model are determined in the manner shown. After obtaining the second sample dataset of expressionless labels in the target domain, in addition to following... Figure 5 In addition to directly determining the second loss L2 of the facial expression recognition model as shown, it can be done according to... Figure 10The method described above involves first performing a first transformation and a second transformation on the second sample images in the second sample dataset, resulting in corresponding first-transformed sample sets and second-transformed sample sets. Then, the first-transformed sample sets are input into the backbone network of the expression recognition model, and the expression feature generation network of the expression recognition model extracts the second expression feature vectors of each first-transformed sample in the first-transformed sample set. Based on the second expression features of each first-transformed sample in the first-transformed sample set, the second expression recognition result of each first-transformed sample is determined. The second-transformed samples in the second-transformed sample set are then labeled based on the second expression recognition results of each first-transformed sample and the target threshold set. Next, the labeled second-transformed samples in the second-transformed sample set, i.e., the sample training dataset, are input into the backbone network of the expression recognition model, and the third expression feature vectors of each second-transformed sample in the sample training dataset are extracted from the first-transformed sample set of the expression recognition model. Based on the third expression feature vectors of each second-transformed sample in the sample training dataset, the third expression recognition result of each second-transformed sample is determined. Based on the third expression recognition results of each second-transformed sample and the expression labels corresponding to each second-transformed sample, the second loss of the expression recognition model corresponding to the target domain is determined. Finally, the facial expression recognition model can be trained based on the first and second losses as described in S406 above.

[0168] Next, combine Figures 2-10 Taking server 220 performing facial expression recognition as an example, this specification describes the facial expression recognition method provided in the embodiments. Please refer to [link / reference needed] for details. Figure 11 This is a flowchart illustrating an exemplary embodiment of an expression recognition method provided in this specification. Figure 11 As shown, this facial expression recognition method includes the following steps:

[0169] S1102, Obtain the target user's face image.

[0170] Specifically, when it is desired to understand the facial expressions of a target user, an image acquisition device 210 can be used to acquire a target facial image of the target user, and then the target facial image can be sent to a server 220 via a network. The server 220 can receive target facial images sent by the image acquisition device 210, other terminals, or other servers via the network.

[0171] S1104: Input the target face image into the expression recognition model and output the target expression recognition result corresponding to the target user.

[0172] Specifically, after obtaining the target user's facial image, the target facial image can be input into the facial expression recognition model trained using the facial expression recognition model training method provided in the embodiments of this specification, thereby obtaining the target facial expression recognition result corresponding to the target user. The target facial expression recognition result includes the target facial expression recognition result corresponding to the target user in the target facial image, that is, the classification probability corresponding to each expression category.

[0173] Optionally, after obtaining the target expression recognition result corresponding to the target user, the expression category corresponding to the highest classification probability in the above target expression recognition result can be directly determined as the expression category corresponding to the target user in the target face image.

[0174] Optionally, to ensure the accuracy and effectiveness of expression recognition, after obtaining the target expression recognition result for the target user, the expression category corresponding to the maximum classification probability in the target expression recognition result will only be determined as the expression category corresponding to the target user in the target face image if the maximum classification probability in the target expression recognition result is greater than the target threshold of the expression category corresponding to the maximum classification probability in the trained expression recognition model. If the maximum classification probability in the target expression recognition result is less than or equal to the target threshold of the expression category corresponding to the maximum classification probability in the trained expression recognition model, to ensure the accuracy of expression recognition, the expression recognition can be directly confirmed as a failure, and the target user's target face image needs to be re-acquired for expression recognition.

[0175] This specification describes an embodiment of an expression recognition model trained using contrastive learning based on first paired sample data and domain-adaptive semi-supervised learning training on a first sample dataset with expression labels in the source domain and a second sample dataset without expression labels in the target domain. This model identifies the expression categories of target users in target face images. It leverages the uncertainty of the target face image during expression recognition to enhance the robustness of identifying ambiguous expression categories, improving the accuracy of expression recognition. Furthermore, it avoids the problem of insufficient labeled samples in the target domain or excessively time-consuming manual data annotation, which hinders efficient and accurate expression recognition in the target domain within a short period. This solves the cold start problem of expression recognition in the target domain.

[0176] Please refer to the following. Figure 12 , Figure 12 This specification provides an exemplary embodiment of an facial expression recognition model training device. For example... Figure 12 As shown, the facial expression recognition model training device 1200 includes:

[0177] The acquisition module 1210 is used to acquire a first sample dataset and a second sample dataset; the first sample dataset includes multiple first sample images with expression labels in the source domain; the second sample dataset includes multiple second sample images without expression labels in the target domain.

[0178] The determining module 1220 is used to determine a first paired sample dataset based on the first sample dataset mentioned above; the first paired sample dataset includes at least one set of first paired sample data; the first paired sample data includes at least two first sample images from the first sample dataset.

[0179] Training module 1230 is used to train an expression recognition model based on the first sample dataset, the first paired sample dataset, and the second sample dataset.

[0180] In one possible implementation, the training module 1230 includes:

[0181] The first determining unit is used to determine the first loss of the expression recognition model based on the aforementioned first paired sample dataset.

[0182] The second determining unit is used to determine the second loss of the above-mentioned expression recognition model based on the above-mentioned first sample dataset and the above-mentioned second sample dataset;

[0183] The training unit is used to train the above-mentioned expression recognition model based on the first loss and the second loss.

[0184] In one possible implementation, the above-mentioned facial expression recognition model includes a backbone network, an uncertainty extraction network, and an facial expression feature generation network;

[0185] The aforementioned first determining unit includes:

[0186] The first feature extraction subunit is used to extract the first feature of each first sample image in the first paired sample data using the backbone network described above.

[0187] The uncertainty extraction subunit is used to input the first features of each first sample image in the first paired sample data into the uncertainty extraction network and output the uncertainty vector corresponding to each first sample image in the first paired sample data.

[0188] The first facial expression feature extraction subunit is used to input the first features of each first sample image in the first paired sample data into the facial expression feature generation network and output the first facial expression feature vector corresponding to each first sample image in the first paired sample data.

[0189] The fusion subunit is used to fuse the uncertainty vector and the first expression feature vector to obtain the mixed features corresponding to the first paired sample data.

[0190] The first determining subunit is used to determine the first loss of the above expression recognition model based on the mixed features corresponding to each first paired sample data in the above first paired sample dataset.

[0191] In one possible implementation, the aforementioned determining sub-unit is specifically used for:

[0192] Based on the mixed features corresponding to each first paired sample data in the first paired sample dataset, predict the mixed expression probability corresponding to each first paired sample data in the first paired sample dataset; based on the mixed expression probability and the expression label corresponding to each first sample image in the first paired sample dataset, determine the first loss of the expression recognition model.

[0193] In one possible implementation, the second determining unit includes:

[0194] The second determining subunit is used to determine the target threshold set of the above-mentioned expression recognition model based on the first sample dataset mentioned above.

[0195] The third determining subunit is used to determine the second loss of the above expression recognition model based on the above target threshold set and the above second sample dataset.

[0196] In one possible implementation, the second determining subunit is specifically used for:

[0197] The first feature of each first sample image in the first sample dataset is extracted using the backbone network described above; the first feature of each first sample image in the first sample dataset is input into the expression feature generation network, and the first expression feature vector corresponding to each first sample image in the first sample dataset is output; the first expression recognition result corresponding to each first sample image in the first sample dataset is predicted based on the first expression feature vector corresponding to each first sample image in the first sample dataset; the target threshold set of the expression recognition model is determined based on the first expression recognition result corresponding to each first sample image in the first sample dataset and the expression label.

[0198] In one possible implementation, the aforementioned third determining subunit is specifically used for:

[0199] The second features of each second sample image in the second sample dataset are extracted using the backbone network described above; the second features of each second sample image in the second sample dataset are input into the expression feature generation network, and the second expression feature vector corresponding to each second sample image in the second sample dataset is output; the second expression recognition result corresponding to each second sample image in the second sample dataset is predicted based on the second expression feature vector corresponding to each second sample image in the second sample dataset; the second sample images in the second sample dataset are labeled based on the target threshold set of the expression recognition model and the second expression recognition result corresponding to each second sample image in the second sample dataset, thereby obtaining the sample training dataset of the target domain; and the second loss of the expression recognition model is determined based on the sample training dataset.

[0200] In one possible implementation, the second expression recognition result includes the classification probability of each expression category corresponding to the second sample image; the target threshold set includes at least one target threshold corresponding to an expression category.

[0201] The third determining subunit, when used to annotate the second sample images in the second sample dataset based on the target threshold set of the expression recognition model and the second expression recognition results corresponding to each second sample image in the second sample dataset, to obtain the sample training dataset for the target domain, is specifically used for:

[0202] Based on the second expression recognition results corresponding to each second sample image in the second sample dataset and the target threshold set, the target second sample image in the second sample dataset is determined; the target second sample image is used to represent the second sample image corresponding to the second expression recognition result whose maximum classification probability is greater than the target threshold of the expression category corresponding to the maximum classification probability; the target second sample image in the second sample dataset is labeled based on the expression category corresponding to the maximum classification probability of the target second sample image in the second sample dataset to obtain the sample training dataset of the target domain.

[0203] In one possible implementation, the aforementioned third determining subunit is specifically used for:

[0204] The second sample images in the second sample dataset are subjected to a first transformation to obtain a first transformed sample set; the second sample images in the second sample dataset are subjected to a second transformation to obtain a second transformed sample set; the change magnitude of the second transformation on the second sample images is greater than the change magnitude of the first transformation on the second sample images; the second transformed second sample images in the second transformed sample set are labeled based on the target threshold set of the expression recognition model and the first transformed sample set to obtain the sample training dataset of the target domain; the second loss of the expression recognition model is determined based on the sample training dataset.

[0205] In one possible implementation, the first transformation includes at least one of rotation transformation, translation transformation, image flipping transformation, and affine transformation; the second transformation includes random erasure transformation of image regions.

[0206] The division of modules in the above-described facial expression recognition model training device is for illustrative purposes only. In other embodiments, the facial expression recognition model training device can be divided into different modules as needed to complete all or part of the functions of the above-described facial expression recognition model training device. The implementation of each module in the facial expression recognition model training device provided in the embodiments of this specification can be in the form of a computer program. This computer program can run on a terminal or server. The program modules constituted by this computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements all or part of the steps of the facial expression recognition model training method described in the embodiments of this specification.

[0207] Please refer to the following. Figure 13 , Figure 13 This specification provides an exemplary embodiment of an expression recognition device. For example... Figure 13 As shown, the facial expression recognition device 1300 includes:

[0208] The acquisition module 1310 is used to acquire the target user's face image.

[0209] The expression recognition module 1320 is used to input the target face image into the expression recognition model and output the target expression recognition result corresponding to the target user; the expression recognition model is trained based on the expression recognition model training method provided in the embodiments of this specification.

[0210] The division of modules in the above-described facial expression recognition device is for illustrative purposes only. In other embodiments, the facial expression recognition device can be divided into different modules as needed to complete all or part of the functions of the facial expression recognition device. The implementation of each module in the facial expression recognition device provided in the embodiments of this specification can be in the form of a computer program. This computer program can run on a terminal or server. The program modules constituted by this computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements all or part of the steps of the facial expression recognition method described in the embodiments of this specification.

[0211] Please see Figure 14 , Figure 14 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. For example... Figure 14 As shown, the electronic device 1400 may include: at least one processor 1410, at least one communication bus 1420, a user interface 1430, at least one network interface 1440, and a memory 1450. The communication bus 1420 can be used to enable communication between the aforementioned components.

[0212] The user interface 1430 may include a display screen and a camera. Optionally, the user interface 1430 may also include a standard wired interface and a wireless interface.

[0213] The network interface 1440 may optionally include a Bluetooth module, a Near Field Communication (NFC) module, a Wireless Fidelity (Wi-Fi) module, etc.

[0214] The processor 1410 may include one or more processing cores. The processor 1410 connects to various parts within the electronic device 1400 using various interfaces and lines. It executes various functions and processes data of the routing electronic device 1400 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1450, and by calling data stored in the memory 1450. Optionally, the processor 1410 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1410 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 1410.

[0215] The memory 1450 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1450 may include a non-transitory computer-readable medium. The memory 1450 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1450 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as an acquisition function, an expression recognition model training function, an expression recognition function, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 1450 may also be at least one storage device located remotely from the aforementioned processor 1410. Figure 14 As shown, the memory 1450, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and application programs.

[0216] Specifically, the electronic device 1400 can be the aforementioned facial expression recognition model training device, in Figure 14In the illustrated electronic device 1400, the user interface 1430 is mainly used to provide an input interface for the user, such as buttons or a camera on the aforementioned facial expression recognition model training device, to obtain user-triggered commands; while the processor 1410 can be used to call the application program stored in the memory 1450 and specifically perform the following operations:

[0217] Obtain a first sample dataset and a second sample dataset; the first sample dataset includes multiple first sample images with expression labels in the source domain; the second sample dataset includes multiple second sample images without expression labels in the target domain.

[0218] A first paired sample dataset is determined based on the aforementioned first sample dataset; the aforementioned first paired sample dataset includes at least one set of first paired sample data; the aforementioned first paired sample data includes at least two first sample images from the aforementioned first sample dataset.

[0219] The facial expression recognition model was trained based on the first sample dataset, the first paired sample dataset, and the second sample dataset mentioned above.

[0220] In some possible embodiments, when the processor 1410 trains the expression recognition model based on the first sample dataset, the first paired sample dataset, and the second sample dataset, it is specifically used to perform:

[0221] The first loss of the facial expression recognition model is determined based on the first paired sample dataset mentioned above.

[0222] The second loss of the above expression recognition model is determined based on the first sample dataset and the second sample dataset.

[0223] The above-mentioned facial expression recognition model is trained based on the first loss and the second loss mentioned above.

[0224] In some possible embodiments, the above-described facial expression recognition model includes a backbone network, an uncertainty extraction network, and an facial expression feature generation network;

[0225] When the processor 1410 determines the first loss of the expression recognition model based on the first paired sample dataset, it specifically performs the following:

[0226] The first feature of each first sample image in the first paired sample data is extracted using the backbone network described above.

[0227] The first feature of each first sample image in the first paired sample data is input into the uncertainty extraction network, and the uncertainty vector corresponding to each first sample image in the first paired sample data is output.

[0228] The first feature of each first sample image in the first paired sample data is input into the expression feature generation network, and the first expression feature vector corresponding to each first sample image in the first paired sample data is output.

[0229] The uncertainty vector is fused with the first facial expression feature vector to obtain the mixed feature corresponding to the first paired sample data.

[0230] The first loss of the above expression recognition model is determined based on the mixed features corresponding to each first paired sample data in the first paired sample dataset.

[0231] In some possible embodiments, when the processor 1410 determines the first loss of the expression recognition model based on the mixed features corresponding to each first paired sample data in the first paired sample dataset, it specifically performs the following:

[0232] Based on the mixed features corresponding to each first paired sample data in the aforementioned first paired sample dataset, predict the probability of mixed expressions corresponding to each first paired sample data in the aforementioned first paired sample dataset.

[0233] The first loss of the above expression recognition model is determined based on the above mixed expression probability and the expression labels corresponding to each first sample image in the above first paired sample data.

[0234] In some possible embodiments, when the processor 1410 determines the second loss of the expression recognition model based on the first sample dataset and the second sample dataset, it specifically performs the following:

[0235] Based on the first sample dataset mentioned above, the target threshold set of the above expression recognition model is determined.

[0236] The second loss of the above expression recognition model is determined based on the above target threshold set and the above second sample dataset.

[0237] In some possible embodiments, when the processor 1410 determines the target threshold set of the expression recognition model based on the first sample dataset, it specifically performs the following:

[0238] The first feature of each first sample image in the first sample dataset is extracted using the backbone network described above.

[0239] The first feature of each first sample image in the first sample dataset is input into the expression feature generation network, and the first expression feature vector corresponding to each first sample image in the first sample dataset is output.

[0240] Based on the first expression feature vector corresponding to each first sample image in the first sample dataset, predict the first expression recognition result corresponding to each first sample image in the first sample dataset.

[0241] Based on the first expression recognition results corresponding to each first sample image in the first sample dataset and the expression labels, the target threshold set of the expression recognition model is determined.

[0242] In some possible embodiments, when the processor 1410 determines the second loss of the expression recognition model based on the target threshold set and the second sample dataset, it specifically performs the following:

[0243] The second feature of each second sample image in the second sample dataset is extracted using the backbone network described above.

[0244] The second features of each second sample image in the second sample dataset are input into the expression feature generation network, and the second expression feature vector corresponding to each second sample image in the second sample dataset is output.

[0245] Based on the second expression feature vector corresponding to each second sample image in the aforementioned second sample dataset, predict the second expression recognition result corresponding to each second sample image in the aforementioned second sample dataset.

[0246] Based on the target threshold set of the above expression recognition model and the second expression recognition results corresponding to each second sample image in the above second sample dataset, the second sample images in the above second sample dataset are labeled to obtain the sample training dataset of the above target domain.

[0247] The second loss of the above expression recognition model is determined based on the above sample training dataset.

[0248] In some possible embodiments, the second expression recognition result includes the classification probability of each expression category corresponding to the second sample image; the target threshold set includes at least one target threshold corresponding to an expression category.

[0249] When the processor 1410 annotates the second sample images in the second sample dataset based on the target threshold set of the expression recognition model and the second expression recognition results corresponding to each second sample image in the second sample dataset to obtain the sample training dataset of the target domain, it is specifically used to perform the following:

[0250] Based on the second expression recognition results corresponding to each second sample image in the second sample dataset and the target threshold set, the target second sample image in the second sample dataset is determined; the target second sample image is used to represent the second sample image corresponding to the second expression recognition result whose maximum classification probability is greater than the target threshold of the expression category corresponding to the maximum classification probability.

[0251] Based on the expression category corresponding to the maximum classification probability of the target second sample image in the aforementioned second sample dataset, the target second sample image in the aforementioned second sample dataset is labeled to obtain the sample training dataset of the aforementioned target domain.

[0252] In some possible embodiments, when the processor 1410 determines the second loss of the expression recognition model based on the target threshold set and the second sample dataset, it specifically performs the following:

[0253] The second sample images in the second sample dataset are subjected to a first transformation to obtain the first transformed sample set.

[0254] The second sample image in the second sample dataset is subjected to a second transformation to obtain a second transformed sample set; the change of the second sample image by the second transformation is greater than the change of the second sample image by the first transformation.

[0255] Based on the target threshold set of the above expression recognition model and the first transformation sample set, the second transformed second sample image in the second transformation sample set is labeled to obtain the sample training dataset of the above target domain.

[0256] The second loss of the above expression recognition model is determined based on the above sample training dataset.

[0257] In one possible implementation, the first transformation includes at least one of rotation transformation, translation transformation, image flipping transformation, and affine transformation; the second transformation includes random erasure transformation of image regions.

[0258] In some possible embodiments, the electronic device 1400 may be the aforementioned facial expression recognition device, and the processor 1410 may further perform the following:

[0259] Obtain the target user's facial image.

[0260] The target facial image is input into the expression recognition model, and the target expression recognition result corresponding to the target user is output. The expression recognition model is trained based on the expression recognition model training method provided in the embodiments of this specification.

[0261] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described facial expression recognition model training device or facial expression recognition device are implemented as software functional units and sold or used as independent products, they can be stored in the above-described computer-readable storage medium.

[0262] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0263] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0264] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims.

[0265] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims and specification may be performed in a different order than in the embodiments described in the specification and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for training an facial expression recognition model, the method comprising: Obtain the first sample dataset and the second sample dataset; The first sample dataset includes multiple first sample images with facial expression labels in the source domain; The second sample dataset includes multiple second sample images of the target domain without expression labels; A first paired sample dataset is determined based on the first sample dataset; The first paired sample dataset includes at least one set of first paired sample data; The first paired sample data includes at least two first sample images from the first sample dataset; The facial expression recognition model is trained based on the first sample dataset, the first paired sample dataset, and the second sample dataset. The process of training an expression recognition model based on the first sample dataset, the first paired sample dataset, and the second sample dataset includes: Determine the first loss of the facial expression recognition model based on the first paired sample dataset; The second loss of the facial expression recognition model is determined based on the first sample dataset and the second sample dataset; The facial expression recognition model is trained based on the first loss and the second loss; The facial expression recognition model includes a backbone network, an uncertainty extraction network, and a facial expression feature generation network; The determination of the first loss of the facial expression recognition model based on the first paired sample dataset includes: The backbone network is used to extract the first features of each first sample image in the first paired sample data; The first feature of each first sample image in the first paired sample data is input into the uncertainty extraction network, and the uncertainty vector corresponding to each first sample image in the first paired sample data is output. The first feature of each first sample image in the first paired sample data is input into the expression feature generation network, and the first expression feature vector corresponding to each first sample image in the first paired sample data is output. The uncertainty vector is fused with the first facial expression feature vector to obtain the mixed feature corresponding to the first paired sample data; The first loss of the expression recognition model is determined based on the mixed features corresponding to each first paired sample data in the first paired sample dataset.

2. The method as described in claim 1, wherein determining the first loss of the expression recognition model based on the mixed features corresponding to each first paired sample data in the first paired sample dataset includes: Based on the mixed features corresponding to each first paired sample data in the first paired sample dataset, predict the mixed expression probability corresponding to each first paired sample data in the first paired sample dataset; The first loss of the expression recognition model is determined based on the mixed expression probability and the expression labels corresponding to each first sample image in the first paired sample data.

3. The method of claim 1, wherein determining the second loss of the expression recognition model based on the first sample dataset and the second sample dataset comprises: Determine the target threshold set for the facial expression recognition model based on the first sample dataset; The second loss of the facial expression recognition model is determined based on the target threshold set and the second sample dataset.

4. The method of claim 3, wherein determining the target threshold set of the expression recognition model based on the first sample dataset includes: The backbone network is used to extract the first feature of each first sample image in the first sample dataset; The first feature of each first sample image in the first sample dataset is input into the expression feature generation network, and the first expression feature vector corresponding to each first sample image in the first sample dataset is output. Based on the first expression feature vector corresponding to each first sample image in the first sample dataset, predict the first expression recognition result corresponding to each first sample image in the first sample dataset; The target threshold set of the expression recognition model is determined based on the first expression recognition result corresponding to each first sample image in the first sample dataset and the expression label.

5. The method of claim 3, wherein determining the second loss of the expression recognition model based on the target threshold set and the second sample dataset comprises: The backbone network is used to extract the second features of each second sample image in the second sample dataset; The second features of each second sample image in the second sample dataset are input into the expression feature generation network, and the second expression feature vector corresponding to each second sample image in the second sample dataset is output. Based on the second expression feature vector corresponding to each second sample image in the second sample dataset, predict the second expression recognition result corresponding to each second sample image in the second sample dataset; Based on the target threshold set of the expression recognition model and the second expression recognition results corresponding to each second sample image in the second sample dataset, the second sample images in the second sample dataset are labeled to obtain the sample training dataset of the target domain. The second loss of the facial expression recognition model is determined based on the sample training dataset.

6. The method of claim 5, wherein the second expression recognition result includes the classification probability of each expression category corresponding to the second sample image; the target threshold set includes at least one target threshold corresponding to an expression category; The second sample images in the second sample dataset are labeled based on the target threshold set of the expression recognition model and the second expression recognition results corresponding to each second sample image in the second sample dataset to obtain the sample training dataset of the target domain, including: The target second sample image in the second sample dataset is determined based on the second expression recognition result corresponding to each second sample image in the second sample dataset and the target threshold set. The target second sample image is used to characterize the second sample image corresponding to the second expression recognition result where the maximum classification probability is greater than the target threshold of the expression category corresponding to the maximum classification probability. The target second sample images in the second sample dataset are labeled based on the expression category corresponding to the maximum classification probability of the target second sample images in the second sample dataset, thus obtaining the sample training dataset of the target domain.

7. The method of claim 3, wherein determining the second loss of the expression recognition model based on the target threshold set and the second sample dataset comprises: The second sample image in the second sample dataset is subjected to a first transformation to obtain a first transformed sample set; The second sample image in the second sample dataset is subjected to a second transformation to obtain a second transformed sample set; the change of the second sample image by the second transformation is greater than the change of the second sample image by the first transformation. Based on the target threshold set of the expression recognition model and the first transformation sample set, the second transformed second sample image in the second transformation sample set is labeled to obtain the sample training dataset of the target domain. The second loss of the facial expression recognition model is determined based on the sample training dataset.

8. The method of claim 7, wherein the first transformation includes at least one of rotation transformation, translation transformation, image flip transformation, and affine transformation; and the second transformation includes random image region erasure transformation.

9. An expression recognition method, the method comprising: Obtain the target user's facial image; The target facial image is input into the expression recognition model, and the target expression recognition result corresponding to the target user is output. The facial expression recognition model is trained based on the facial expression recognition model training method described in any one of claims 1-8.

10. An expression recognition model training device, the device comprising: The acquisition module is used to acquire the first sample dataset and the second sample dataset; The first sample dataset includes multiple first sample images with facial expression labels in the source domain; The second sample dataset includes multiple second sample images of the target domain without expression labels; The determining module is used to determine the first paired sample dataset based on the first sample dataset; The first paired sample dataset includes at least one set of first paired sample data; The first paired sample data includes at least two first sample images from the first sample dataset; The training module is used to train an expression recognition model based on the first sample dataset, the first paired sample dataset, and the second sample dataset. The training module includes: The first determining unit is configured to determine the first loss of the expression recognition model based on the first paired sample dataset. The second determining unit is configured to determine a second loss of the expression recognition model based on the first sample dataset and the second sample dataset; Training unit, used to train the facial expression recognition model based on the first loss and the second loss; The facial expression recognition model includes a backbone network, an uncertainty extraction network, and a facial expression feature generation network; The first determining unit includes: The first feature extraction subunit is used to extract the first feature of each first sample image in the first paired sample data using the backbone network; An uncertainty extraction subunit is used to input the first features of each first sample image in the first paired sample data into the uncertainty extraction network and output the uncertainty vector corresponding to each first sample image in the first paired sample data. The first facial expression feature extraction subunit is used to input the first features of each first sample image in the first paired sample data into the facial expression feature generation network and output the first facial expression feature vector corresponding to each first sample image in the first paired sample data. A fusion subunit is used to fuse the uncertainty vector with the first expression feature vector to obtain the mixed features corresponding to the first paired sample data; The first determining subunit is used to determine the first loss of the expression recognition model based on the mixed features corresponding to each first paired sample data in the first paired sample dataset.

11. An expression recognition device, the device comprising: The acquisition module is used to acquire the target user's facial image; The expression recognition module is used to input the target facial image into the expression recognition model and output the target expression recognition result corresponding to the target user. The facial expression recognition model is trained based on the facial expression recognition model training method described in any one of claims 1-8.

12. An electronic device, comprising: Processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-9.

13. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method as claimed in any one of claims 1-9.

14. A computer program product comprising instructions that, when run on a computer or processor, cause the computer or processor to perform the method as described in any one of claims 1-9.

Images