A feature separation-based dual-branch loss adversarial expression recognition method and system

Abstract

The application discloses a double-branch loss-based feature separation expression recognition method and system, obtains training data, and performs data preprocessing and data enhancement on the training data; a deep learning network model is built, and a network model comprising a feature extractor, a weight distribution module and an expression classification module is built; a confusion loss and a domain adversarial loss are used to guide a non-expression feature branch; a feature separation loss between non-expression features and expression features is constructed; and after model training is completed, a result is predicted by using the feature extractor and the classifier. The application can separate expression features and improve the prediction accuracy of the model.

Description

Technical Field

[0001] This invention belongs to the field of facial recognition technology, specifically relating to a dual-branch loss adversarial expression recognition method and system based on feature separation, applicable to nighttime fatigue driving and security monitoring. Background Technology

[0002] Two-branch loss adversarial facial expression recognition constructs two parallel network branches: one for extracting facial expression features and the other for extracting interfering features. The facial expression branch aims to learn and recognize key features related to facial expressions, while the interfering branch aims to identify and separate interfering factors unrelated to facial expressions that may affect the recognition results, such as head pose and lighting conditions. This method is used to identify and classify different facial expressions from face images. It combines deep learning and adversarial training, using two parallel branch networks to simultaneously extract facial expression features and adversarial interfering features.

[0003] Facial expression recognition (FER), as an important component of facial recognition technology, has significant practical implications. In recent years, it has received widespread attention in human-computer interaction, security, robotics, automation, healthcare, communications, and autonomous driving, becoming a research hotspot in both academia and industry. The purpose of expression recognition is to analyze various facial expressions from different people, explore the connections between different expressions, filter out false features caused by interference factors, and identify common information about the same expression. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a dual-branch loss adversarial expression recognition method and system based on feature separation, which addresses the shortcomings of the prior art. This method enables the image features extracted from the two entangled branches to be separable, thereby allowing expression recognition to be performed directly on the expression branch. This solves the technical problem that the presence of other expression-related information in the existing data interferes with the expression recognition task.

[0005] The present invention adopts the following technical solution:

[0006] A two-branch loss-based adversarial facial expression recognition method based on feature separation includes the following steps:

[0007] S1. Obtain the training image dataset I and perform preprocessing and data augmentation on the training image dataset;

[0008] S2. Construct a deep learning network model including feature extractor B, weight allocation module, expression classification module, and interference factor classification module. Use feature extractor B to extract initial features from the training image dataset I obtained in step S1. For initial features Weighting is performed to obtain facial expression features X e X e With initial features After subtraction, the non-expression feature X is obtained. d ;

[0009] S3. Use confusion loss and domain adversarial loss to construct adversarial loss, and guide the non-expression feature branch to extract non-expression features;

[0010] S4. Construct non-expression features X d and facial features X e Feature separation loss between;

[0011] S5. The adversarial loss obtained in step S3 and the feature separation loss obtained in step S4 are weighted and summed. The resulting total loss for facial expression recognition is used to train the deep learning network model obtained in step S2. The test set images are input into the feature extractor B and then subjected to feature separation. Finally, the images are input into the facial expression classifier to obtain the classification result.

[0012] Preferably, in step S1, the training image dataset I is:

[0013]

[0014] in, This represents the i-th facial expression image. express The label is 'n', where 'n' represents the number of images.

[0015] Preferably, in step S2, iResNet50 is used as the backbone network encoder B to process the input facial expression image I. e To extract the mixed initial features, i.e., X = B(I e The feature residual module is defined as follows:

[0016] X*(1-δ(X))=X–X*δ(X)

[0017] Where * denotes element-wise multiplication, δ denotes the weight allocation module, and X*δ(X) denotes the facial expression feature X. e X*(1-δ(X)) represents the identity feature X d .

[0018] Preferably, in step S2, the information related to facial expressions in the feature map is separated by the weight allocation module supervised by the facial expression recognition task, and the branch features regarded as identity-related information are supervised by the adversarial loss. The weight allocation module includes a channel attention module and a spatial attention module.

[0019] Preferably, in step S3, the resistance loss is... for:

[0020]

[0021] in, To obscure the loss, For domain-adaptive adversarial loss.

[0022] More preferably, optimize the loss function for facial expression recognition. Confusion Loss Domain Adaptive Adversarial Loss They are respectively:

[0023]

[0024]

[0025] Where, N ex L represents the number of expressions in this batch. CE C(X) represents the cross-entropy loss of the predicted distribution of the classifier output. d y is the output of classifier C, W is the network weight, and y is the weight of the network. e For emoji tags, GRL(B(X) d )) represents performing gradient inversion on the identity features output by classifier B.

[0026] Preferably, in step S4, the feature separation loss for:

[0027]

[0028] Where |||| denotes the L2 norm of the vector, X e and X d Let N be the feature vector and N be the number of samples.

[0029] Preferably, in step S5, the total loss of expression recognition for:

[0030]

[0031] Where λ1 and λ2 represent equilibrium parameters. Let the loss function be for facial expression recognition. To combat the losses, This is the feature separation loss.

[0032] More preferably, the loss function for facial expression recognition for:

[0033]

[0034] Among them, L CELet A represent the cross-entropy loss, W be the network weights, and y represent the expression recognition network weights. e For emoji tags.

[0035] Secondly, embodiments of the present invention provide a dual-branch loss adversarial facial expression recognition system based on feature separation, comprising:

[0036] The data module acquires the training image dataset I and performs preprocessing and data augmentation on the training image dataset;

[0037] The construction module builds a deep learning network model including a feature extractor B, a weight allocation module, an expression classification module, and a interference factor classification module. The feature extractor B extracts initial features from the training image dataset I obtained from the data module. For initial features Weighting is performed to obtain facial expression features X e X e With initial features After subtraction, the non-expression feature X is obtained. d ;

[0038] The first function module uses confusion loss and domain adversarial loss to construct adversarial loss, which guides the non-expression feature branch to extract non-expression features.

[0039] The second function module constructs the non-expression feature X. d and facial features X e Feature separation loss between;

[0040] The output module performs a weighted sum of the adversarial loss obtained from the first function module and the feature separation loss obtained from the second function module. It then uses the resulting total loss for expression recognition to train the deep learning network model obtained from the construction module. After inputting the test set images into the feature extractor B and performing feature separation, the images are finally input into the expression classifier to obtain the classification results.

[0041] Thirdly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the aforementioned feature-separation-based dual-branch loss adversarial expression recognition method.

[0042] Fourthly, embodiments of the present invention provide an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-described feature-separation-based dual-branch loss adversarial expression recognition method.

[0043] Compared with the prior art, the present invention has at least the following beneficial effects:

[0044] An adversarial facial expression recognition method based on feature separation and bi-branch loss is proposed. This method utilizes a deep learning framework to solve the facial expression recognition problem. It leverages the powerful descriptive capabilities of deep neural networks to model and learn and recognize seven common facial expressions (anger, disgust, surprise, happiness, sadness, neutrality, and fear). A significant problem in facial expression recognition is that personal information can interfere with expression learning. Different people make the same expression differently, severely impacting the model's ability to learn facial information. This invention aims to separate facial expression and facial information as much as possible for facial expression recognition. This invention not only demonstrates excellent results on data collected in a laboratory environment but also exhibits strong robustness to data collected in natural environments. This is because data collected in natural environments typically suffers from objective factors such as occlusion and pose variations, which significantly affect facial expression discrimination.

[0045] Furthermore, collecting a large and diverse training image dataset ensures that the model can learn the broad sample distribution required for facial expression recognition tasks, which helps improve the model's generalization ability. By performing operations such as image standardization, normalization, denoising, and resizing, unnecessary noise is eliminated, and the data format is unified, making it easier for the model to learn the essential features of the images and reducing the impact of data bias on model training. Additional training samples are generated through techniques such as flipping, rotating, scaling, cropping, and color transformation, artificially increasing data diversity, effectively mitigating overfitting, improving the model's ability to recognize various facial expression variations, and enhancing the model's robustness.

[0046] Furthermore, a deep neural network architecture, such as the Convolutional Neural Network (CNN) in this invention, is employed to automatically extract high-dimensional, abstract feature representations from the original images. These features capture visual patterns of facial expressions, such as facial muscle movements and shape changes, providing a foundation for subsequent classification. The weight allocation module assigns weights to the initial features, emphasizing expression-related features while suppressing non-expression features. This helps the model focus on the core elements of the expression recognition task, improving recognition efficiency and accuracy. The expression classification module specifically classifies the extracted expression features, mapping them to predefined expression categories to achieve automatic expression recognition. The interference factor classification module identifies and separates non-expression features (such as changes in illumination, occlusion, and background interference), helping the model ignore the influence of irrelevant factors when recognizing expressions, improving recognition stability and anti-interference capabilities.

[0047] Furthermore, a weight allocation module is constructed using a channel attention module (CA) and a spatial attention module (SA) to highlight expression-related information at both the channel and spatial levels. Note that both attention modules have the same output size, and a linear summation approach is used to combine channel and spatial attention. This balances their respective characteristics to some extent and allows for flexible adjustment of their weights in the final feature representation. Both spatial and channel attention have their advantages; the linear summation approach ensures that the final feature representation considers both the importance of spatial location and the correlation between channels, thus providing a more comprehensive description of the features.

[0048] Furthermore, the confusion loss penalizes the model for misclassifying non-expression features, prompting the model to learn to distinguish between expression and non-expression features during training, avoiding confusion between the two, and enhancing the model's focus on expression recognition in practical applications. The domain adversarial loss, drawing on the idea of ​​adversarial learning, constructs a branch that rivals the expression recognition task, encouraging the model to learn discriminative features that can effectively distinguish between expression and non-expression, further improving the targeting and accuracy of feature extraction.

[0049] Furthermore, by defining and minimizing the distance between non-expression features and expression features (such as cosine similarity, Mahalanobis distance, etc.), the model is forced to learn independent feature representations, ensuring that expression features and non-expression features are separated as much as possible in the feature space, reducing mutual interference between them. Clear feature separation helps the model more accurately distinguish expression categories during the expression recognition stage, avoiding the misleading influence of non-expression factors on the expression recognition results, and improving recognition accuracy.

[0050] Furthermore, adversarial loss and feature separation loss are combined to form a composite loss function, which takes into account both the model's learning objectives in expression recognition and non-expression suppression, making the training process more comprehensive. By optimizing the total loss function, the model is guided to improve its ability in expression feature extraction, expression classification, and non-expression feature separation, ensuring a balanced improvement in overall model performance. This training strategy, which comprehensively considers multiple losses, helps the model maintain good expression recognition performance in various complex scenarios and enhances its generalization ability on unknown test data.

[0051] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0052] In summary, this invention can separate facial and non-facial information using a single tag, and has high promotional value.

[0053] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0054] Figure 1 This is a flowchart of the method of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of a deep learning network model in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a computer device provided in an embodiment of the present invention;

[0057] Figure 4 This is a block diagram of a chip provided according to an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0060] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0061] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0062] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0063] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0064] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0065] This invention provides a two-branch loss-based adversarial facial expression recognition method based on feature separation. It employs a convolutional neural network and loss-based adversarial training. The iResNet50 pre-trained model has wide applications in facial expression recognition tasks due to its simple structure, reasonable design, and significant performance; many high-performance facial expression recognition algorithms use this pre-trained model as the basic feature extraction network. The loss adversarial network uses two different loss functions added together to achieve adversarial results. The key point of this invention is combining adversarial training and loss function methods to constrain the extracted basic image features, thereby separating facial expression information. For specific technical solutions, better results can be obtained by replacing the initial feature extraction network with other networks with stronger feature extraction capabilities, or by designing other feature constraint methods to improve performance.

[0066] Please see Figure 1 This invention discloses a two-branch loss adversarial expression recognition method based on feature separation, comprising the following steps:

[0067] S1. Obtain the training image dataset Preprocessing and data augmentation are performed on the training image dataset;

[0068] in, This represents the i-th facial expression image. express The label is 'n', where 'n' represents the number of images.

[0069] For the input training image dataset, data augmentation is first performed. Different data augmentations are performed for different datasets, generally including but not limited to random horizontal flipping and normalization.

[0070] S2. Construct a deep learning network model including feature extractor B, weight allocation module, expression classification module, and interference factor classification module. Use feature extractor B to extract initial features from the training image dataset obtained in step S1. and initial features Weighting is performed to obtain facial expression features X e X e With initial features After subtraction, the non-expression feature X is obtained. d ;

[0071] Please see Figure 2 This paper proposes a weighted feature decomposition module to decompose mixed initial features in the high-level semantic space. The main difference lies in the fact that operations on feature vectors are more complex than operations on feature maps. An iResNet50 is used as the backbone network encoder B to decompose the input facial expression image I. e To extract the mixed initial features, i.e., X = B(I e ), and the feature residual module is defined as:

[0072] X*(1-δ(X))=X–X*δ(X) (1)

[0073] Where δ represents element-wise multiplication, represents the weight allocation module, and X*δ(X) represents the facial expression feature X. e X*(1-δ(X)) represents the identity feature X d .

[0074] The weight allocation module, supervised by the facial expression recognition task, separates expression-related information from the feature map, and uses adversarial loss to supervise branch features considered as identity-related. Therefore, the weight allocation module constrains the decomposition module, facilitating better separation of expression-related features from the mixed feature map.

[0075] In this invention, a weight allocation module is constructed using a channel attention module (CA) and a spatial attention module (SA) to highlight expression-related information at both the channel and spatial levels.

[0076] S3. Construct adversarial loss using confusion loss and domain adversarial loss. Guide the non-expression feature branch to extract non-expression features;

[0077] To robustly decompose features using the FED module, this invention employs both facial expression recognition tasks and adversarial loss for supervision, further constraining the network's feature decomposition.

[0078] X eThe encoding of facial expression features is supervised through a facial expression recognition task. It's important to clarify that the facial expression recognition network A and the classification network C with the adversarial loss branch have exactly the same structure.

[0079] Define the loss function for optimizing facial expression recognition. for:

[0080]

[0081] Among them, L CE Let A represent the cross-entropy loss, W represent the facial expression recognition network, and y represent the network weights. e For emoji tags.

[0082] The weight allocation module is responsible for allocating weights to expression-related regions. This invention employs a loss-supervised training method, which can guide the non-expression branch to extract non-expression features using only expression labels. Generally, tasks based on the idea of ​​de-entanglement use corresponding labels to guide model training when extracting certain features. Here, it is necessary to suppress the extraction of expression information. This is mainly achieved by using expression labels to guide the non-expression branch to extract non-expression features in a loss adversarial manner.

[0083] Next, adversarial loss is used to supervise the non-expression branch feature X. d The study of X here d The definition of non-expression is to ignore the interference factors of facial expressions, but according to the observation of the data, the biggest interference factor is still the difference in people's appearance. Therefore, the non-expression branch is also defined as the identity feature branch.

[0084] We introduce cross-expression domain adversarial learning, encouraging X through continuous domain adaptation with gradient inversion layer (GRL). d Maintaining facial expression stability (i.e., the extracted information does not change with facial expression). The inhibition loss of the identity feature branch mainly consists of two... and The composition is as follows:

[0085] y p =SoftMax(C(X) d )W+b)

[0086]

[0087]

[0088]

[0089] In equation (5), the first term is the confusion loss, and the second term is the domain adaptive adversarial loss, y e It's an emoji tag.

[0090] Note that both terms in equation (5) use a network structure with shared parameters. A classifier needs to be trained to better extract X. e Features are trained using cross-entropy loss. Additionally, a non-expression branch needs to be trained using confusion loss. Domain Adaptive Adversarial Loss Joint training was conducted. ex This represents the number of expressions in this batch, calculated by dividing the loss by N. ex This yields an average loss function. This balances the confusion loss across all categories, ensuring no bias towards any particular category, and effectively confuses all expressions together.

[0091] S4. Construct non-expression features X d and facial features X e Feature separation loss between

[0092] For FER, X e It should be consistent and must not change with confounding factors in other data; this is crucial, but X e and X d In fact, there is a strong connection between them.

[0093] For example, X e and X d They exhibit a high linear correlation with each other. Therefore, X e Some interference factors can negatively impact facial expression recognition; the aforementioned features, including the Feature Decomposition Network (FED) based on the weight allocation module and the domain adversarial loss based on gradient inversion and confusion loss, are all designed to ensure X... e and X d They are unrelated.

[0094] To further decompose and separate the two parts of features, a feature separation loss is introduced to further determine the similarity between the extracted two parts of features. Formally, given a pair of features X e X d The features are then evaluated using specific information processing units. Clearly, the goal of this invention is to reduce X during the training process. e and X d This reduces the correlation between the two features by increasing their similarity. Therefore, X e and X d Through continuous learning via the network, they achieve low similarity and eventually become significantly unrelated; the feature separation loss formula is defined as follows:

[0095]

[0096] Normalized eigenvectors:

[0097] First, Norm represents the condition for two input feature vectors X. e and X d L2 normalization is performed on each vector, where · represents the dot product of the vectors and |||| represents the L2 norm of the vectors. This step ensures that the feature vectors have the same scale and direction when calculating cosine similarity;

[0098] Next, the cosine similarity between the two normalized feature vectors is calculated to obtain their degree of similarity;

[0099] The cosine similarity score ranges from -1 to 1, where 1 indicates perfect similarity and -1 indicates complete dissimilarity.

[0100] Next, the cosine similarity is converted into the square of the cosine distance. This transforms similarity into distance, making a larger value indicate a greater difference between the feature vectors. The mean of the squares of the cosine distances is then taken as the final loss value. The loss value represents the degree of difference between two feature vectors. Minimizing this loss function helps to separate or differentiate them further. N represents the number of samples and is used to normalize the loss.

[0101] S5. The adversarial loss L obtained in step S3 adv The feature separation loss L obtained in step S4 fl A weighted sum is performed, and the resulting total loss for facial expression recognition is used to train the deep learning network model obtained in step S2. After the test set images are input into the feature extractor B and subjected to feature separation, they are finally input into the facial expression classifier to obtain the classification result.

[0102] The model is trained adversarially in an end-to-end manner, and non-expression features are extracted by using a weight allocation module network to assist in adversarial training.

[0103] Secondly, feature similarity evaluation helps enhance the independence between decomposed components. The difference method ensures that the adversarial branch of the loss can extract non-expression features, while the expression label supervision ensures that the expression branch can extract expression features.

[0104] Finally, the model is also used to remove certain factors in various non-expression branches, such as pose and outdoor environment. Based on the above equations (1) to (6), the total loss of the proposed model is as follows:

[0105]

[0106] Where λ1 and λ2 represent equilibrium parameters.

[0107] By optimizing the total loss network, interference from non-expression factors in facial images can be effectively eliminated, and the required expression features can be extracted. After the model training is completed, the test image is input into the feature extractor B for feature decomposition, and finally input into the expression classifier to obtain the classification result.

[0108] In another embodiment of the present invention, a feature-separation-based two-branch loss adversarial facial expression recognition system is provided. This system can be used to implement the above-mentioned feature-separation-based two-branch loss adversarial facial expression recognition method. Specifically, the feature-separation-based two-branch loss adversarial facial expression recognition system includes a data module, a construction module, a first function module, a second function module, and an output module.

[0109] The data module acquires the training image dataset I and performs preprocessing and data augmentation on the training image dataset.

[0110] The construction module builds a deep learning network model including a feature extractor B, a weight allocation module, an expression classification module, and a interference factor classification module. The feature extractor B extracts initial features from the training image dataset I obtained from the data module. For initial features Weighting is performed to obtain facial expression features X e X e With initial features After subtraction, the non-expression feature X is obtained. d ;

[0111] The first function module uses confusion loss and domain adversarial loss to construct adversarial loss, which guides the non-expression feature branch to extract non-expression features.

[0112] The second function module constructs the non-expression feature X. d and facial features X e Feature separation loss between;

[0113] The output module performs a weighted sum of the adversarial loss obtained from the first function module and the feature separation loss obtained from the second function module. It then uses the resulting total loss for expression recognition to train the deep learning network model obtained from the construction module. After inputting the test set images into the feature extractor B and performing feature separation, the images are finally input into the expression classifier to obtain the classification results.

[0114] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, the computer program including program instructions, and the processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a feature-separated two-branch loss adversarial expression recognition method, including:

[0115] Obtain the training image dataset I and perform preprocessing and data augmentation on it; build a deep learning network model including feature extractor B, weight allocation module, expression classification module, and interference factor classification module, and use feature extractor B to extract initial features from the obtained training image dataset I. For initial features Weighting is performed to obtain facial expression features X e X e With initial features After subtraction, the non-expression feature X is obtained. d Adversarial loss is constructed using confusion loss and domain adversarial loss to guide the non-expression feature branch to extract non-expression features; non-expression feature X is constructed. d and facial features X e The feature separation loss between the two is calculated; the adversarial loss and the feature separation loss are weighted and summed, and the resulting total loss for expression recognition is used to train the deep learning network model. The test set images are input into the feature extractor B, and after feature separation, they are finally input into the expression classifier to obtain the classification result.

[0116] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0117] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the feature-separation-based dual-branch loss adversarial expression recognition method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps:

[0118] Obtain the training image dataset I and perform preprocessing and data augmentation on it; build a deep learning network model including feature extractor B, weight allocation module, expression classification module, and interference factor classification module, and use feature extractor B to extract initial features from the obtained training image dataset I. For initial features Weighting is performed to obtain facial expression features X e X e With initial features After subtraction, the non-expression feature X is obtained. d Adversarial loss is constructed using confusion loss and domain adversarial loss to guide the non-expression feature branch to extract non-expression features; non-expression feature X is constructed. d and facial features X e The feature separation loss between the two is calculated; the adversarial loss and the feature separation loss are weighted and summed, and the resulting total loss for expression recognition is used to train the deep learning network model. The test set images are input into the feature extractor B, and after feature separation, they are finally input into the expression classifier to obtain the classification result.

[0119] Please see Figure 3The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the fluid composition calculation method in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the fluid composition calculation system in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here.

[0120] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 3 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0121] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0122] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0123] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0124] Please see Figure 4 The terminal device is a chip. In this embodiment, the chip 600 includes a processor 622, which may be one or more, and a memory 632 for storing computer programs executable by the processor 622. The computer program stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 622 may be configured to execute the computer program to perform the generalizable monocular absolute depth map estimation method described above.

[0125] Additionally, chip 600 may also include a power supply component 626 and a communication component 650. The power supply component 626 can be configured to perform power management of chip 600, and the communication component 650 can be configured to enable communication of chip 600, such as wired or wireless communication. Furthermore, chip 600 may also include an input / output interface 658. Chip 600 can operate on an operating system stored in memory 632.

[0126] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0127] To verify the effectiveness of this invention, two commonly used facial expression datasets, FERPlus and RAF-DB, were selected for experimental verification.

[0128] Table 1: Accuracy of Experimental Classification (Unit: %)

[0129] EAC 90.35 89.64 PACVT 88.21 88.72 SimFLE 89.41 88.94 FFDNet 88.5 88.75 VTFF 88.14 88.81 IPD-FER 88.89 88.42 Ours 91.85 91.28

[0130] This invention improves performance compared to existing methods, outperforming EAC, PACVT, SimFLE, FFDNet, VTFF, and IPD-FER by 1.5%, 3.64%, 2.44%, 3.35%, 3.71%, and 1.96% respectively on the RAF-DB dataset.

[0131] While existing research has made significant progress, data collected in natural environments still contains some objective factors that are difficult to remove. These include variations in posture, differences in human appearance, changes in lighting, facial blurring, and partial facial occlusion, all of which greatly increase the difficulty of expression recognition. This invention primarily addresses the problem of poor expression recognition performance due to differences in human appearance in data collected in natural environments. It removes the interference of identity information from expression images by separating facial features from facial features.

[0132] In summary, this invention presents a feature-separation-based dual-branch loss adversarial facial expression recognition method and system. It is trained end-to-end, using residuals to construct two branches: an expression feature branch and a non-expression interference factor branch, to separate interference factors. The facial expression recognition method learns how to guide the non-expression branch to learn non-expression information during adversarial training, while simultaneously minimizing the similarity between pairwise decomposed features of expressions and corresponding interference factors through supervised feature separation loss. In testing, only expression features are used for facial expression recognition. The loss adversarial training network is guided by two different loss functions through gradient flipping and confusion loss to achieve loss adversarial goals. The key point of this invention is the combination of feature disentanglement and loss-supervised training to constrain the extracted basic image features, thereby separating expression information. Experimental results on the FERPlus and RAF-DB databases demonstrate that this invention has superior performance in facial expression recognition.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0134] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0136] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0140] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A dual-branch loss adversarial expression recognition method based on feature separation, characterized in that, Includes the following steps: S1. Obtain the training image dataset I and perform preprocessing and data augmentation on the training image dataset; S2. Construct a deep learning network model including feature extractor B, weight allocation module, expression classification module, and interference factor classification module. Use feature extractor B to extract initial features from the training image dataset I obtained in step S1. For initial features Weighting is performed to obtain facial expression features X e X e With initial features After subtraction, the non-expression feature X is obtained. d ; S3. Use confusion loss and domain adversarial loss to construct adversarial loss, and guide the non-expression feature branch to extract non-expression features; S4. Construct non-expression features X d and facial features X e Feature separation loss between; S5. The adversarial loss obtained in step S3 and the feature separation loss obtained in step S4 are weighted and summed. The resulting total loss for facial expression recognition is used to train the deep learning network model obtained in step S2. The test set images are input into the feature extractor B and then subjected to feature separation. Finally, the images are input into the facial expression classifier to obtain the classification result.

2. The adversarial expression recognition method based on feature separation using a dual-branch loss according to claim 1, characterized in that, In step S1, the training image dataset I is: in, This represents the i-th facial expression image. express The label is 'n', where 'n' represents the number of images.

3. The adversarial expression recognition method based on feature separation using a dual-branch loss according to claim 1, characterized in that, In step S2, iResNet50 is used as the backbone network encoder B to process the input facial expression image I. e To extract the mixed initial features, i.e., X = B(I e The feature residual module is defined as follows: X*(1-δ(X))=X–X*δ(X) Where * denotes element-wise multiplication, δ denotes the weight allocation module, and X*δ(X) denotes the facial expression feature X. e X*(1-δ(X)) represents the identity feature X d .

4. The adversarial expression recognition method based on feature separation using a dual-branch loss according to claim 1, characterized in that, In step S2, the information related to facial expressions in the feature map is separated by the weight allocation module supervised by the facial expression recognition task, and the branch features regarded as identity-related information are supervised by the adversarial loss. The weight allocation module includes a channel attention module and a spatial attention module.

5. The adversarial expression recognition method based on feature separation using a dual-branch loss according to claim 1, characterized in that, In step S3, combating losses for: in, To obscure the loss, For domain-adaptive adversarial loss.

6. The adversarial expression recognition method based on feature separation with dual-branch loss according to claim 5, characterized in that, Optimize the loss function for facial expression recognition Confusion Loss Domain Adaptive Adversarial Loss They are respectively: Where, N ex L represents the number of expressions in this batch. CE C(X) represents the cross-entropy loss of the predicted distribution of the classifier output. d y is the output of classifier C, W is the network weight, and y is the weight of the network. e For emoji tags, GRL(B(X) d )) represents performing gradient inversion on the identity features output by classifier B.

7. The adversarial expression recognition method based on feature separation using a dual-branch loss according to claim 1, characterized in that, In step S4, feature separation loss for: Where || represents the L2 norm of the vector, X e and X d Let N be the feature vector and N be the number of samples.

8. The adversarial expression recognition method based on feature separation with dual-branch loss according to claim 1, characterized in that, In step S5, the total loss of expression recognition for: Where λ1 and λ2 represent equilibrium parameters. Let the loss function be for facial expression recognition. To combat the losses, This is the feature separation loss.

9. The adversarial expression recognition method based on feature separation with dual-branch loss according to claim 8, characterized in that, Loss function for facial expression recognition for: Among them, L CE Let A represent the cross-entropy loss, W be the network weights, and y represent the expression recognition network weights. e For emoji tags.

10. A dual-branch loss adversarial facial expression recognition system based on feature separation, characterized in that, include: The data module acquires the training image dataset I and performs preprocessing and data augmentation on the training image dataset; The construction module builds a deep learning network model including a feature extractor B, a weight allocation module, an expression classification module, and a interference factor classification module. The feature extractor B extracts initial features from the training image dataset I obtained from the data module. For initial features Weighting is performed to obtain facial expression features X e X e With initial features After subtraction, the non-expression feature X is obtained. d ; The first function module uses confusion loss and domain adversarial loss to construct adversarial loss, which guides the non-expression feature branch to extract non-expression features. The second function module constructs the non-expression feature X. d and facial features X e Feature separation loss between; The output module performs a weighted sum of the adversarial loss obtained from the first function module and the feature separation loss obtained from the second function module. It then uses the resulting total loss for expression recognition to train the deep learning network model obtained from the construction module. After inputting the test set images into the feature extractor B and performing feature separation, the images are finally input into the expression classifier to obtain the classification results.

Images