Occluded facial expression recognition method based on self-supervised learning technology and application

Abstract

The application discloses a kind of based on self-supervised learning technique's being covered face expression recognition method and application, the steps of this method include:1, the image preprocessing of a database containing untagged face image;2, constructing the network model of pre-task stage of self-supervised learning;3, constructing similarity loss and cover identification loss;4, using gradient descent method to optimize the network model parameter of pre-task stage of self-supervised learning;5, the image preprocessing of N class face expression image database;6, constructing the network model of downstream task stage of self-supervised learning;7, constructing classification loss;8, using gradient descent method to optimize the network model parameter of downstream task stage of self-supervised learning;9, using the trained classifier to carry out face expression classification identification to the face image to be measured.The application can overcome the negative influence caused by image cover to face expression recognition effect, so as to realize the accurate identification of face expression.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of occluded facial expression recognition, affective computing and self-supervised learning, and particularly relates to an occluded facial expression recognition method based on self-supervised learning technology and application. BACKGROUND

[0002] As a complex case of facial expression recognition, occluded facial expression recognition has a wide range of applications in real production and life. The existing occluded facial expression recognition methods can be mainly divided into four categories, namely, robust facial feature-based method, unoccluded facial image reconstruction-based method, sub-region analysis-based method and unoccluded facial image auxiliary-based method. The robust facial feature-based method aims to find a feature representation that is insensitive to occlusion but can distinguish different expressions. However, occlusion can occur at any position in the facial image, so it is difficult to find a facial feature representation that is robust to occlusion. The unoccluded facial image reconstruction-based method aims to reconstruct the unoccluded version of the image to be recognized using a generative model, and then classify the expression according to the reconstructed image. However, the authenticity of the generated image is usually not high, which directly affects the performance of facial expression recognition. The sub-region analysis-based method divides the image into several regions and then performs facial expression recognition from these sub-regions and the entire image. Generally, facial key point detection and attention mechanism technology are used for facial sub-region segmentation, but facial key point detection for occluded facial images is still challenging, which also affects the accuracy of facial expression recognition of this method. The unoccluded facial image auxiliary-based method usually uses unoccluded facial images as privileged information to assist occluded facial expression recognition. During training, these methods generally construct two networks: one for unoccluded facial expression recognition and the other for occluded facial expression recognition. During testing, these methods assume that all facial images are occluded and only use the occluded facial expression recognition network for recognition. However, in real-world scenarios, it is impossible to know whether a facial image is occluded. In addition, all the above methods require complete expression-labeled images for training, but because the type and location of the occluded part are infinite, it is very difficult to collect large-scale image datasets with various expressions and occlusions for these methods. SUMMARY

[0003] The present application is to solve the above-mentioned deficiencies in the prior art, and proposes an occluded facial expression recognition method based on self-supervised learning technology and application, so as to use unoccluded facial and occlusion pattern position information to assist in the recognition of occluded facial expressions, thereby improving the accuracy of occluded facial expression recognition.

[0004] To achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0005] The feature of the self-supervised learning-based occluded face expression recognition method is to perform the following steps:

[0006] Step 1, image preprocessing is performed on a database containing untagged face images:

[0007] According to the face key point information of each image in the database, size cropping, scaling and rotation operations are performed on each image in the database containing untagged face images, so that the face in the image is in the middle position, and a processed image set is obtained, denoted as , wherein represents the total number of training samples, represents the i-th unoccluded face image; represents the height of the image, represents the width of the image;

[0008] a random one is selected from a plurality of occlusion patterns and is randomly added to the i-th unoccluded face image to generate the i-th occluded face image ; and the corresponding occlusion mask is denoted as ;

[0009] The processed image set , the synthesized occluded image set and the corresponding occlusion mask set are denoted as the pre-task stage data set of self-supervised learning ;

[0010] Step 2, a pre-task stage network model of self-supervised learning is constructed, including a pre-feature extractor , an image feature extractor , an occlusion identifier and a feature mapping head ;

[0011] The pre-feature extractor is composed of an input convolutional layer, a batch normalization layer and a ReLU activation function;

[0012] The image feature extractor is composed of a max-pooling layer, a Resnet feature extraction layer and a mean-pooling layer;

[0013] The occlusion identifier is composed of an input convolutional layer, an up-sampling convolutional layer and an output convolutional layer, and the input convolutional layer and the up-sampling convolutional layer are connected to a batch normalization layer and a ReLU activation function, and the output convolutional layer is connected to a batch normalization layer;

[0014] The feature mapping head consists of an input linear layer, a ReLU activation function and an output linear layer;

[0015] occlusion identifier and feature mapping head The parameters in the need to be initialized;

[0016] Step 3, construct similarity loss and occlusion identification loss;

[0017] Step 3.1, similarity loss:

[0018] Step 3.1.1, input the ith unoccluded face image to the pre-feature extractor for processing to obtain the ith unoccluded pre-feature ;

[0019] Step 3.1.2, input the unoccluded pre-feature to the image feature extractor for processing to obtain the unoccluded intermediate feature ;

[0020] Step 3.1.3, input the unoccluded intermediate feature to the feature mapping head for processing to obtain the unoccluded face representation ;

[0021] Step 3.1.4, input the ith occluded face image to the pre-feature extractor for processing to obtain the ith occluded pre-feature ;

[0022] Step 3.1.5, input the occluded pre-feature to the image feature extractor for processing to obtain the occluded intermediate feature ;

[0023] Step 3.1.6, input the occluded intermediate feature to the feature mapping head for processing to obtain the occluded face representation ;

[0024] Step 3.1.4 constructs a similarity loss function using formula (1) :

[0025] (1)

[0026] In formula (1), and respectively represent and the similarity loss, and is obtained by formula (2) and formula (3):

[0027] (2)

[0028] (3)

[0029] In formula (2) and formula (3), denotes a temperature parameter, denotes a cosine similarity function; denotes any one of the set of all facial representations except denotes any one of the set of all facial representations except

[0030] Step 3.2, occlusion identification loss:

[0031] Step 3.2.1, inputting the pre-occlusion feature to the occlusion identifier for processing to obtain an occlusion prediction matrix ;

[0032] Step 3.2.3, constructing an occlusion identification loss function using formula (4):

[0033] (4)

[0034] In formula (4), denotes whether the pixel point in the gth row and the kth column of the occlusion mask belongs to the occlusion pattern, denotes the predicted probability that the pixel point in the gth row and the kth column of the occlusion prediction matrix belongs to the occlusion pattern;

[0035] Step 4, using gradient descent method to optimize the pre-task stage network model parameters of the self-supervised learning;

[0036] Step 4.1, constructing the total loss function of the pre-task stage network model of the self-supervised learning using formula (5): :

[0037] (5)

[0038] In formula (5), and are weight factors for controlling the weight of the similarity loss and the occlusion identification loss, respectively;

[0039] ​​Step 4.2: Set the total number of external training steps to... The current training steps outside are Set the total number of training steps internally to The current training steps inside are Set the number of samples to be used in each training iteration as follows: ;initialization , ;

[0040] Step 4.3: From the dataset of the pre-task stage of self-supervised learning In the external first The second internal The next random draw without replacement Group samples And as an external third The second internal The training samples of the next iteration; where... This represents the j-th unobstructed face image. This refers to the j-th occluded face image. This represents the occlusion mask corresponding to the j-th occluded image;

[0041] Step 4.4, place the external first The second internal Training samples for the next iteration The input is used to train the pre-task stage network model of the self-supervised learning method using gradient descent, while simultaneously calculating the total loss function. external The second internal The network model parameters are updated once;

[0042] Step 4.5, Assign to Then, make a judgment If the condition is met, proceed to step 4.6; otherwise, return to step 4.3 and continue execution.

[0043] Step 4.6, Assign to Then, make a judgment If the condition is met, it indicates that the training of the network model in the pre-task stage of self-supervised learning has ended, and the pre-feature extractor trained in the pre-task stage of self-supervised learning is obtained. Image feature extractor Occlusion Detector and feature mapping head Execute step 5; otherwise, initialize. = 1, return to step 4.3 and continue to execute;

[0044] Step 5, image preprocessing is performed on the database containing images with facial expressions of human faces;

[0045] cropping, scaling and rotating operations are performed on all facial images in the database to obtain a normalized facial image dataset , a total of training samples, wherein, represents an unoccluded facial image, is an occluded facial image after adding occlusion to , and represents the expression label of the i-th sample;

[0046] Step 6, constructing a network model of a downstream task stage of self-supervised learning, including a pre-feature extractor trained in a pre-task stage of self-supervised learning , an image feature extractor trained in a pre-task stage of self-supervised learning , and an expression classifier ;

[0047] The expression classifier is composed of a linear layer, a batch normalization layer, a ReLU activation function, a linear layer, a batch normalization layer, a ReLU activation function and a linear layer in sequence, wherein the parameters of the linear layer need to be initialized;

[0048] Step 7, constructing a classification loss;

[0049] Step 7.1, randomly taking a pair of samples from the and their true labels , inputting the sample into the pre-feature extractor to obtain pre-features , and inputting the sample into the pre-feature extractor to obtain pre-features ;

[0050] Step 7.2, inputting the pre-features into the image feature extractor respectively to obtain intermediate features , and inputting the pre-features into the image feature extractor respectively to obtain intermediate features ;

[0051] Step 7.3, inputting the intermediate features​ input expression classifier , to obtain a classification probability vector ; input the intermediate features into the input expression classifier , to obtain a classification probability vector ;

[0052] Step 7.4, constructing a classification loss using formula (6) :

[0053]

[0054] In formula (6), represents a cross-entropy loss function, represents a softmax function;

[0055] Step 8, using gradient descent method to optimize the network model parameters of the downstream task stage of the self-supervised learning;

[0056] Step 8.1, setting the total number of training steps as , and the current total number of training steps as ; setting the internal number of training steps as , and the current internal number of training steps as ; setting the number of samples sampled each time in training as ; initializing as ;

[0057] Step 8.2, externally randomly and without replacement taking out group of samples from the external internal th iteration and as the training samples of the external external th

[0058] iteration ; Step 8.3, inputting the training samples into the network model of the downstream task stage of the self-supervised learning, and using gradient descent method to optimize the pre-feature extractor , the image feature extractor and the expression classifier

[0059] according to formula (6); Step 8.4, after assigning to , judging whether

[0060] is true, if true, executing step 8.5, otherwise, returning to step 8.2 for continuous execution; assigned to After, judge whether it is established, if it is established, it indicates that the downstream task stage network model training of self-supervised learning is completed, and the pre-feature extractor of the downstream task stage after training of self-supervised learning , image feature extractor And expression classifier , execute step 9, otherwise, initialize =1, then return step 8.2 and continue to execute;

[0061] Step 9, using the pre-feature extractor of the downstream task stage after training of self-supervised learning , image feature extractor And expression classifier The face image to be tested is predicted to realize the classification and identification of facial expression.

[0062] The electronic device comprises a memory and a processor, wherein the memory is used to store a program supporting the processor to execute the occluded facial expression recognition method, and the processor is configured to execute the program stored in the memory.

[0063] The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to execute the steps of the occluded facial expression recognition method.

[0064] Compared with the prior art, the beneficial effects of the present application are as follows:

[0065] 1、The present application generates an occluded facial expression image by adding a shielding pattern to an unoccluded facial image, and uses the position information of the unoccluded facial image and the shielding pattern in this process to assist in the expression recognition of the occluded facial image. The existing occluded expression recognition method is limited by the number and diversity of samples, and the occluded facial expression recognition with better effect is realized.

[0066] 2、The present application uses the technology of contrast learning to measure the gap between the features of unoccluded facial images and the features of occluded facial images with similarity loss, so that the feature representation of the occluded facial image tends to be closer to the feature representation of the unoccluded facial image with the same expression, which guides the network to extract a feature representation more robust to shielding from the occluded facial image, and improves the availability in actual environment.

[0067] 3. This invention proposes an occlusion prediction loss, which enables the network to predict the location of the occlusion pattern on the occluded image and pay less attention to the information in the occlusion pattern area, thereby reducing the impact of occlusion, improving the correlation between the extracted feature representation and the expression, and effectively improving the recognition accuracy of occluded expressions.

[0068] 4. This invention proposes the use of self-supervised learning technology. By applying similarity loss and occlusion prediction loss in the pre-task stage of self-supervised learning, a feature extractor with better initial parameters is provided for the downstream task stage of self-supervised learning. This not only improves the recognition rate of occluded facial expressions, but also makes the method easier to transfer to other datasets and has better generalization ability. Attached Figure Description

[0069] Figure 1 This is a framework diagram of the pre-task stage of self-supervised learning in the model of this invention.

[0070] Figure 2 This is a framework diagram of the downstream task stage of the self-supervised learning model of this invention. Detailed Implementation

[0071] In this embodiment, as Figure 1 As shown, the preliminary task stage of a self-supervised learning-based occluded facial expression recognition method is carried out in the following steps:

[0072] Step 1: Perform image preprocessing on the database containing unlabeled face images:

[0073] Based on the facial landmark information of each image in the database, each image in the database containing unlabeled face images is cropped, scaled, and rotated to center the face in the image, resulting in the processed image set, denoted as . ,in, This represents the total number of training samples. This represents the i-th unobstructed face image; Represents the height of the image. Represents the width of the image; in this embodiment, the pixel size of all face images after normalization is 224×224; that is... , VGGFace2 was used as an unlabeled face image database. VGGFace2 contains 3,141,890 images from 9,131 individuals, downloaded from Google Image Search. This database exhibits significant differences in race, age, and pose. In this embodiment, it is used as the dataset for the pre-task stage of self-supervised learning. ;

[0074] From one of the occlusion patterns is randomly selected and added to the i-th unoccluded face image to generate the i-th occluded face image ; and The corresponding occlusion mask is denoted as In this embodiment, , respectively, 5 hand occlusion patterns, 5 food occlusion patterns and 5 drink occlusion patterns.

[0075] The processed image set , the synthesized occluded image set and its corresponding occlusion mask set are denoted as the pre-task stage dataset of self-supervised learning ;

[0076] Step 2, constructing a pre-task stage network model of self-supervised learning, including: a pre-feature extractor , an image feature extractor , an occlusion identifier and a feature mapping head ;

[0077] In this embodiment, the pre-feature extractor is composed of an input convolutional layer, a batch normalization layer and a ReLU activation function in turn; in this embodiment, the input convolutional layer, the batch normalization layer and the ReLU activation function are all provided by the resnet34 pre-training network, and are resnet34.conv1, resnet34.bn1 and resnet34.relu respectively;

[0078] The image feature extractor is composed of a max-pooling layer, four Resnet feature extraction layers and a mean-pooling layer in turn; in this embodiment, the max-pooling layer and the four Resnet feature extraction layers are all provided by the resnet34 pre-training network, and are resnet34.maxpool and resnet34.layer1-4 respectively, and the mean-pooling layer is an adaptive mean-pooling layer with a size of 1x1;

[0079] The occlusion identifier The input convolutional layer, the up-sampling convolutional layer and the output convolutional layer are sequentially connected, the input convolutional layer is configured as an input channel number of 64, an output channel number of 32, a convolution kernel size of 3*3 and a padding of 1, the up-sampling convolutional layer is configured as a scale factor of 2 and a sampling mode of bilinear, and the output convolutional layer is configured as an input channel number of 32, an output channel number of 1, a convolution kernel size of 3*3 and a padding of 1; the input convolutional layer and the up-sampling convolutional layer are both connected to a batch normalization layer (the batch normalization feature number is 32 in the embodiment) and a ReLU activation function, and the output convolutional layer is connected to a batch normalization layer (the batch normalization feature number is 1 in the embodiment);

[0080] In the embodiment, the feature mapping head is sequentially connected by an input linear layer (the input number of the linear layer is 512 and the output number is 128 in the embodiment), a ReLU activation function and an output linear layer (the input number of the linear layer is 128 and the output number is 128 in the embodiment);

[0081] The occlusion identifier and the feature mapping head need to be initialized; in the embodiment, all convolutional layers use kaiming normal distribution for weight initialization, and if there is an offset in the convolutional layer, the offset is initialized as a constant , the weights in all batch normalization layers are initialized as constants , the offsets in all batch normalization layers are initialized as constants , all linear layers use normal distribution to initialize the weights (normal distribution with a mean of 0 and a variance of 0.005), and the offset in the linear layer is initialized as a constant ;

[0082] Step 3, construct the similarity loss and the occlusion identification loss;

[0083] Step 3.1, similarity loss:

[0084] Step 3.1.1, input the i-th unoccluded face image into the pre-feature extractor to obtain the i-th unoccluded pre-feature ;

[0085] Input the i-th occluded face image into the pre-feature extractor to obtain the i-th occluded pre-feature ;

[0086] Step 3.1.2, the unoccluded pre-feature input image feature extractor in the processing, obtaining the unoccluded intermediate feature ;

[0087] the occluded pre-feature input image feature extractor in the processing, obtaining the occluded intermediate feature ;

[0088] Step 3.1.3, the unoccluded intermediate feature input feature mapping head in the processing, obtaining the unoccluded face representation ;

[0089] the occluded intermediate feature input feature mapping head in the processing, obtaining the occluded face representation ;

[0090] Step 3.1.4 The method expects that the occluded face representation and the unoccluded face representation of the same face are similar, and the feature that the contrastive learning maximizes the similarity between the positive pairs and minimizes the similarity between the negative pairs meets the needs of the method, so the method uses formula (1) to construct a similarity loss function according to the idea of contrastive learning :

[0091] (1)

[0092] Since is obtained by transforming , its features should be more similar, so the method regards as a positive pair, and and as negative pairs, so in formula (1), and respectively represent the similarity loss of and , and are obtained by formula (2) and formula (3):

[0093] (2)

[0094] (3)

[0095] In formula (2) and formula (3), represents a temperature parameter, represents a cosine similarity function as a similarity measurement method; represents any one of the set of all face representations except , represents any one of the set of all facial representations except ;

[0096] Step 3.2, occlusion identification loss:

[0097] Step 3.2.1, in the human face image, the occluded area usually contains only less or even no information about the expression. If the network can know the location of the occlusion and pay less attention to the information on the occluded area, the influence of the occlusion will be smaller, and the network can obtain more relevant features to the expression. Therefore, the pre-feature with occlusion in this method is processed in the input occlusion identifier to obtain the occlusion prediction matrix

[0098] Step 3.2.3, the idea of cross-entropy is used to supervise the network to predict the occlusion position, so the occlusion identification loss function is constructed as formula (4)

[0099] (4)

[0100] In formula (4), represents whether the pixel point in the gth row and the kth column of the occlusion mask belongs to the occlusion pattern, represents the prediction probability that the pixel point in the gth row and the kth column of the occlusion prediction matrix belongs to the occlusion pattern;

[0101] Step 4, the gradient descent method is used to optimize the parameters of the pre-task stage network model of the self-supervised learning;

[0102] Step 4.1, the total loss function of the pre-task stage network model of the self-supervised learning is constructed as formula (5)

[0103] (5)

[0104] In formula (5), and are weight factors for controlling the weights of the similarity loss and the occlusion identification loss (in this embodiment , );

[0105] Step 4.2, in this embodiment, the total number of external training steps is set to =2, and the current number of external training steps is ; the total number of internal training steps is set to =4910, and the current number of internal training steps is​​​​ Set the number of samples to be used in each training iteration as follows: =64; Initialize , ;

[0106] Step 4.3: From the dataset of the pre-task stage of self-supervised learning In the external first The second internal The next random draw without replacement Group samples And as an external third The second internal The training samples of the next iteration; where... This represents the j-th unobstructed face image. This represents the j-th image of a face that is obscured. This represents the occlusion mask corresponding to the j-th occluded image;

[0107] Step 4.4, place the external first The second internal Training samples for the next iteration The input is used to train the pre-task stage network model of self-supervised learning using gradient descent, while simultaneously calculating the total loss function. external The second internal The network model parameters are updated once;

[0108] Step 4.5, Assign to Then, determine whether all training samples have been retrieved, i.e., determine... If the condition is met, proceed to step 4.6; otherwise, return to step 4.3 and continue execution.

[0109] Step 4.6, Assign to Then, make a judgment If the condition is met, it indicates that the training of the network model in the pre-task stage of self-supervised learning has ended, and the pre-feature extractor trained in the pre-task stage of self-supervised learning is obtained. Image feature extractor Occlusion Detector and feature mapping head If the network serves as the initial network for the next stage, proceed to step 5; otherwise, initialize... After =1, return to step 4.3 and continue execution;

[0110] In this embodiment, as Figure 2As shown, the downstream task stage of a self-supervised learning-based occluded facial expression recognition method is carried out in the following steps:

[0111] Step 5, for those containing Image preprocessing is performed on a database of facial expression images;

[0112] Cropping, scaling, and rotation operations are performed on all face images in the database to obtain a normalized face image dataset. There are a total of training samples, of which This represents an unobstructed image of a face. Is towards Add occlusion to the occluded face image. Indicates the first The facial expression labels for each sample. In this embodiment, subsequent operations were performed on three databases: RAF-DB, AffectNet, and CK+. In RAF-DB, this method normalizes all face images to a pixel size of 224×224; that is... , , , In AffectNet, this method normalizes all face images to a size of 224×224 pixels; that is... , , , In CK+, this method normalizes all face images to a size of 48×48 pixels; that is... , , , ;

[0113] Step 6: Construct the downstream task-stage network model of self-supervised learning, including the pre-feature extractor trained in the pre-task stage of self-supervised learning. Image feature extractor trained in the pre-task stage of self-supervised learning and facial expression classifier ;

[0114] Facial Expression Classifier The linear layer is set to an input number of 512 and an output number of 256 in this embodiment, the batch normalization layer is set to a batch normalization feature number of 256 in this embodiment, the ReLU activation function, the linear layer is set to an input number of 256 and an output number of 128 in this embodiment, the batch normalization layer is set to a batch normalization feature number of 128 in this embodiment, the ReLU activation function, and the linear layer is set to an input number of 128 and an output number of 128 in this embodiment, wherein the linear layer parameters need to be initialized (in this embodiment, the weight is initialized by using a normal distribution (a normal distribution with a mean of 0 and a variance of 0.003), and the offset is initialized as a constant 0);

[0115] Step 7, constructing a classification loss;

[0116] Step 7.1, randomly taking a pair of samples from and their real labels , inputting the sample into the pre-feature extractor to obtain pre-features , and inputting the sample into the pre-feature extractor to obtain pre-features ;

[0117] Step 7.2, inputting the pre-features of the sample into the image feature extractor to obtain intermediate features , and inputting the pre-features of the sample into the image feature extractor to obtain intermediate features

[0118] ; Step 7.3, inputting the intermediate features into the expression classifier to obtain a classification probability vector , and inputting the intermediate features into the expression classifier to obtain a classification probability vector

[0119] ;

[0120]

[0121] wherein represents a cross-entropy loss function, represents a function.

[0122] Step 8: Optimize the network model parameters for the downstream task stage of self-supervised learning using gradient descent.

[0123] Step 8.1: In this embodiment, the total number of training steps is set to... =20, the current total number of training steps is Set the internal training steps to (on RAF-DB) On AffectNet On RAF-DB The current number of training steps is ;). Set the number of samples to be used in each training iteration as follows: =64; Initialize for ;

[0124] Step 8.2, from In the external first The second internal The next random draw without replacement Group samples And as an external third The second internal Training samples for the next iteration;

[0125] Step 8.3: Input the training samples into the downstream task stage network model of self-supervised learning, and optimize the pre-feature extractor using gradient descent according to equation (6). Image feature extractor and facial expression classifier .

[0126] Step 8.4, Assign to Then, determine whether all training samples have been retrieved, i.e., determine... Check if the condition is met. If it is met, proceed to step 8.5; otherwise, return to step 8.2 and continue execution.

[0127] Step 8.5, Assign to Then, make a judgment If the condition is met, it indicates that the training of the network model in the downstream task stage of self-supervised learning has ended, and the pre-feature extractor trained in the downstream task stage of self-supervised learning has been obtained. Image feature extractor and facial expression classifier Proceed to step 9, otherwise... Assign to Then return to step 8.2 and continue execution;

[0128] Step 9, pre-feature extractor trained by downstream task phase of self-supervised learning , image feature extractor and expression classifier The face image to be tested is predicted to realize the classification and recognition of the face expression.

[0129] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the occluded face expression recognition method, and the processor is configured to execute the program stored in the memory.

[0130] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the occluded face expression recognition method.

Claims

1. A method for recognizing an occluded facial expression based on self-supervised learning, characterized in that The steps are as follows: Step 1, image preprocessing is performed on a database containing untagged face images: According to the face key point information of each image in the database, each image in the database containing the untagged face image is subjected to size cutting, scaling and rotation operations, so that the face in the image is in the middle position, and a processed image set is obtained, denoted as wherein N p represents the total number of training samples, represents the i-th unoccluded face image; H represents the height of the image, and W represents the width of the image. From N c A blocking pattern is randomly selected from a set of blocking patterns and randomly added to the i-th unblocked face image to generate the i-th blocked face image and The corresponding blocking mask is denoted as M (i) ∈{0,1} H×W ; the set of processed images the set of synthesized occluded images and their corresponding set of occlusion masks are denoted as pre-task phase dataset for self-supervised learning Step 2, constructing a pre-task stage network model of self-supervised learning, comprising: a pre-feature extractor F b , an image feature extractor F, an occlusion identifier U o and a feature mapping head R; Pre-feature extractor F b consists of an input convolutional layer, a batch normalization layer, and a ReLU activation function; The image feature extractor F is composed of a max pooling layer, a Resnet feature extraction layer and a mean pooling layer; Obstruction recognizer U o The input convolutional layer, the up-sampling convolutional layer and the output convolutional layer are connected, and the input convolutional layer and the up-sampling convolutional layer are connected with the batch normalization layer and the ReLU activation function, and the output convolutional layer is connected with the batch normalization layer. The feature mapping head R is composed of an input linear layer, a ReLU activation function and an output linear layer; Obstruction identifier U o and parameters in the feature map head R need to be initialized; Step 3, construct similarity loss and occlusion recognition loss; Step 3.1, similarity loss: Step 3.1.1, inputting the i-th unoccluded face image inputting the pre-feature extractor F b processing the i-th unoccluded face image to obtain the i-th unoccluded pre-feature the ith occluded face image inputting the pre-feature extractor F b processing in the ith occluded pre-feature extractor Step 3.1.2, the unoccluded front feature inputting the image feature extractor F for processing to obtain an unoccluded intermediate feature obstructed front-facing feature inputting the image feature extractor F for processing to obtain an obstructed intermediate feature Step 3.1.3, occlusion-free intermediate feature The feature map R is input into the processing head R to obtain an occlusion-free face representation obstructed intermediate feature inputting the feature map R into the processing head to obtain an obstructed face representation Step 3.1.4 Constructing a similarity loss function using formula (1) In formula (1), and respectively represent and the similarity loss, and are obtained by formula (2) and formula (3): In formula (2) and formula (3), τ represents a temperature parameter, sim( ) represents a cosine similarity function; z represents any one of the set of all face representations except z' represents any one of the set of all face representations except z' represents any one of the set of all face representations except Step 3.2, occlusion recognition loss: Step 3.2.1, the occlusion foreground feature inputting the occlusion identifier U o occlusion prediction matrix Step 3.2.

3. Constructing the occlusion-aware loss function with formula (4) In formula (4), M[g, k] represents an occlusion mask M (i) whether a pixel point in the gth row and the kth column belongs to an occlusion pattern, represents an occlusion prediction matrix a prediction probability that a pixel point in the gth row and the kth column belongs to an occlusion pattern; Step 4, use gradient descent method to optimize the network model parameters of the pre-task stage of self-supervised learning; Step 4.1, constructing the total loss function of the pre-task stage network model of self-supervised learning with formula (5) In formula (5), λ ss and λ mask are weight factors that control the weights of the similarity loss and the occlusion identification loss, respectively. Step 4.2, set the total training step number of the outside as K, and the current training step number of the outside as k; set the total training step number of the inside as K b , and the current training step number of the inside as k b ; set the sample number of each sampling in the training as B; initialize k = 1, k b = 1; Step 4.3, from the pre-task phase dataset of self-supervised learning external k-th iteration of the k-th internal iteration b randomly and without replacement to obtain a B group of samples external k-th iteration of the k-th internal iteration b external k-th iteration of the k-th internal iteration denotes the jth unoccluded face image, denotes the jth occluded face image, M (j) denotes the jth occluded face image, M Step 4.4, update the network model parameters of the kth external kth internal iteration b training samples of the kth external kth internal iteration input into the pre-task stage network model of the self-supervised learning, and train the pre-task stage network model of the self-supervised learning using gradient descent method, and calculate the total loss function update the network model parameters of the kth external kth internal iteration b ​ Step 4.5, assign k b +1 to k b Then, judge whether k b *B≥N p is true, if true, execute Step 4.6, otherwise return to Step 4.3 to continue execution; Step 4.6, after k+1 is assigned to k, it is judged whether k≥K is established, if established, it indicates that the network model training of the pre-task stage of self-supervised learning is completed, and the pre-feature extractor F' of the pre-task stage of self-supervised learning trained is obtained b , the image feature extractor F', the occlusion identifier U' o and the feature mapping head R', step 5 is executed, otherwise, k is initialized b =1, and step 4.3 is returned to continue to be executed; Step 5, image preprocessing is performed on a database containing N-class face expression images; cropping, scaling and rotating operations are performed on all the face images in the database to obtain a normalized face image dataset N fo +N fc training samples, wherein, represents an unoccluded face image, is an occluded face image after adding occlusion to y (i’) ∈{0,1,…,N e -1} represents the expression label of the i'th sample; Step 6, constructing a downstream task phase network model of self-supervised learning, including the pre-feature extractor F' trained after the pre-task phase of self-supervised learning b , the image feature extractor F' and the expression classifier C trained after the pre-task phase of self-supervised learning The expression classifier C is composed of a linear layer, a batch normalization layer, a ReLU activation function, a linear layer, a batch normalization layer, a ReLU activation function and a linear layer in turn, wherein the linear layer parameters need to be initialized; Step 7, construct classification loss; Step 7.1, from any pair of samples in the and their true labels samples input the pre-feature extractor F b , to get pre-features samples input the pre-feature extractor F b , to get pre-features Step 7.2, inputting the pre-features respectively into the image feature extractor F' to obtain intermediate features inputting the pre-features respectively into the image feature extractor F' to obtain intermediate features Step 7.3, inputting the intermediate features inputting the expression classifier C, obtaining a classification probability vector inputting the intermediate features inputting the expression classifier C, obtaining a classification probability vector Step 7.

4. Constructing the classification loss with formula (6) In formula (6), denotes a cross-entropy loss function, and softmax denotes a softmax function. Step 8, use gradient descent method to optimize the network model parameters of the downstream task stage of self-supervised learning; Step 8.1, set the total training step number as K', the current total training step number as k'; set the internal training step number as K' b , the current internal training step number as k' b ; set the sample number of each sampling in the training as B'; initialize k', k' b as 1; Step 8.2, from the B' set of samples the k'th iteration of the k'th external iteration b Step 8.3, randomly sampling without replacement from the B' set of samples and as the training samples for the k'th iteration of the k'th external iteration b Step 8.4, repeating steps 8.1-8.3 for the k'th external iteration Step 8.3, input the training sample into the downstream task phase network model of the self-supervised learning, and use gradient descent method to optimize the preposed feature extractor F' according to formula (6) b , the image feature extractor F' and the expression classifier C; Step 8.4, assign k' b +1 to k' b Then, determine whether k' b *B'≥N fo is true, if true, execute Step 8.5, otherwise, return to Step 8.2 to continue. Step 8.5, after assigning k'+1 to k', determine whether k'≥K' is true, if true, it means that the network model training of the downstream task stage of self-supervised learning is completed, and the pre-feature extractor F” trained in the downstream task stage of self-supervised learning is obtained b , the image feature extractor F” and the expression classifier C', execute step 9, otherwise, initialize k' b =1, and return to step 8.2 for continuous execution; Step 9, training the pre-feature extractor F" with the downstream task phase of the trained self-supervised learning b The image feature extractor F" and the expression classifier C' make a prediction on the face image to be tested to realize the classification and recognition of the face expression.

2. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the method for recognizing occluded face expressions according to claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the method for recognizing occluded face expressions according to claim 1.

Images