A facial expression recognition method based on capsule network and data repair

Abstract

The application relates to a facial expression recognition method based on a capsule network and data repair, and comprises the following steps: acquiring facial data, inputting the facial data into a trained recognition model, and obtaining a recognition result, wherein the training process of the recognition model is as follows: acquiring facial training data sets, pre-processing and calibrating images in the facial training data sets, inputting the images into a ResNet18 capsule network to obtain feature vectors of the images, the ResNet18 capsule network comprises a first convolution block Conv1_x, a second convolution block Conv2_x, a third convolution block Conv3_x and an 8D capsule convolution layer; and inputting the feature vectors into a data self-repairing module to perform data repair, wherein the data self-repairing module comprises a self-attention importance weighting module, a rank regularization module and a label repair module. Compared with the prior art, the application has the advantages of accurate recognition of fuzzy data and the like.

Description

Technical Field

[0001] This invention relates to facial expression recognition, and more particularly to a facial expression recognition method based on capsule networks and data inpainting. Background Technology

[0002] Psychological problems among seafarers are a common issue in seafarer management. If seafarers experience psychological problems, it can lead to irritability, physical instability, and poor decision-making at work, severely impacting their professional skills and work quality, and even causing serious consequences. If these psychological problems are not addressed and resolved promptly during voyages, they will inevitably lead to mental illness in the long run, affecting their work, disrupting the normal operation of the ship, and even causing maritime accidents.

[0003] Due to the development of artificial intelligence, especially facial expression recognition technology, an increasing number of shipping companies are adopting facial expression recognition (FER) technology to monitor the psychological state of their crew members. Traditional FER methods rely on manual feature extraction, which is not only unable to extract deep features but also computationally expensive. With the establishment of large-scale FER databases and the improvement of computer performance, deep learning-based FER technology has made significant progress in recent years, but many problems still need to be solved. These mainly include the following two points: First, traditional CNNs have limited feature recognition capabilities, are insensitive to the relative positional relationships between facial structures and feature orientation, and cannot effectively capture key emotional expression feature regions. Second, FER datasets based on natural scenes contain many "blurry phenomena," especially in ship data. Due to the presence of waves and wind, the ship's unstable movement at sea, and changes in crew head posture, many shipboard facial image data suffer from "blurry phenomena," such as low image resolution, non-frontal poses, occlusion, or ambiguous expressions causing labeling errors, resulting in poor FER model recognition performance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a facial expression recognition method based on capsule networks and data repair. The method uses an improved capsule network to obtain the feature vector of the image, and performs data repair on the feature vector based on the data self-repair module. It learns the self-importance weights of the data and classifies them, repairs the incorrectly labeled samples in the low-importance weight samples, and achieves accurate recognition of fuzzy data.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A facial expression recognition method based on capsule networks and data inpainting includes the following steps:

[0007] Acquire facial data, input the facial data into a trained recognition model, and obtain the recognition result. The specific training process of the recognition model is as follows:

[0008] A facial training dataset is obtained, and the images in the facial training dataset are preprocessed and calibrated. The preprocessed and calibrated images are then cut to obtain images with a size of 224×224×3. The images are then input into a ResNet18 capsule network to obtain the feature vectors of the images. The ResNet18 capsule network includes a first convolutional block Conv1_x, a second convolutional block Conv2_x, a third convolutional block Conv3_x, and an 8D capsule convolutional layer. The main capsule of the 8D capsule convolutional layer is a convolutional unit containing eight 9×9 convolutional kernels, and the stride of the 8D capsule convolutional layer is 2.

[0009] The feature vector is input into the data self-repair module for data repair. The data self-repair module includes a self-attention importance weighting module, a rank regularization module, and a label repair module. The specific data repair process is as follows:

[0010] Obtain N feature vectors corresponding to N images, input the feature vectors into a self-attention importance weighting module, the self-attention importance weighting module includes a linear fully connected layer and a Sidmoid function, obtain the importance weight of each feature vector based on the linear fully connected layer, form a weight matrix according to the importance weight, perform a weighted summation on the weight matrix and the feature vector, and obtain N self-attention weights based on the weighted summation result and the Sigmoid function;

[0011] The self-attention weights are input into the rank regularization module, and high-importance weight reorganization and low-importance weight reorganization are obtained based on a preset ratio. The mean α of the high-importance weight group is calculated. H The mean α of the low importance weight group L α H and α L Satisfy: α H >α L +δ1, where δ1 is the learning parameter;

[0012] The low-importance weighted recombination is input into the label repair module to obtain the maximum prediction probability of the low-importance weighted recombination. When the maximum prediction probability of the low-importance weighted recombination exceeds the first threshold, the label of the image corresponding to the low-importance weighted recombination is updated, and the updated image is added to the face training dataset. The above training process is repeated until training is completed.

[0013] Furthermore, the expression for the importance weight is:

[0014]

[0015] Where α is the importance weight, SoftMax is the normalization function, X is the feature vector, and d x is the dimension of the feature vector.

[0016] Furthermore, the expression for the self-attention weights is:

[0017]

[0018] Where, α i ′ represents the self-attention weight of the i-th feature vector, W α Let X be the weight matrix. i Let be the i-th eigenvector, and σ be the Sidmoid function.

[0019] Furthermore, the first convolutional block Conv1_x includes a convolutional layer with a kernel size of 7×7 and a stride of 2.

[0020] Furthermore, the second convolutional block Conv2_x includes four convolutional layers with a kernel size of 3×3 and a stride of 1.

[0021] Furthermore, the third convolutional block Conv3_x includes three convolutional layers with a kernel size of 3×3 and a stride of 1, and a convolutional layer with a kernel size of 3×3 and a stride of 2.

[0022] Furthermore, the image is input into the ResNet18 capsule network to obtain the image's feature vector. Specifically, the image is input sequentially into the first convolutional block Conv1_x, the second convolutional block Conv2_x, the third convolutional block Conv3_x, and the 8D capsule convolutional layer to obtain a capsule multidimensional entity of size 10×10×8×16. The capsule multidimensional entity is then transformed into the image's feature vector based on a dynamic routing algorithm.

[0023] Furthermore, the process of obtaining high-importance weight reorganization and low-importance weight reorganization based on a preset ratio specifically involves: arranging the self-attention weights in descending order to obtain high-importance weight reorganization and low-importance weight reorganization according to a preset ratio, wherein the number of samples in the high-importance weight group is M = β * N, where β is the preset ratio, M is the number of samples in the high-importance weight group, and the number of samples in the low-importance weight group is NM.

[0024] Furthermore, the updated image is labeled with the label of the highest predicted probability.

[0025] Furthermore, obtaining the maximum prediction probability of low importance weight recombination specifically involves: the label repair module obtaining the prediction probability of the label corresponding to the image of low importance weight recombination based on the normalization function Softmax, and filtering out the maximum prediction probability.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) The self-importance weights of the data are learned by using the data self-repair mechanism, which enhances the feature expression of high-importance weight samples, suppresses the feature expression of low-importance weight samples, and repairs the erroneous labels in the low-importance weight samples. The repaired low-importance weight samples are then retrained, which can effectively identify fuzzy data, such as data with low image resolution, non-frontal pose, occlusion, or ambiguous expressions.

[0028] (2) An improved capsule network is used instead of the traditional CNN to better capture the relative positional relationship and directional features between the active units of the crew's face, forming a feature vector. Attached Figure Description

[0029] Figure 1 This is a flowchart of the present invention;

[0030] Figure 2 This is a framework diagram of the present invention;

[0031] Figure 3 This is an example diagram showing the facial expression recognition results of the present invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0033] Example 1:

[0034] This invention provides a facial expression recognition method based on capsule networks and data restoration. The flowchart of this invention is as follows: Figure 1 As shown. This method includes the following steps:

[0035] Training the recognition model involves the following steps:

[0036] S1. Obtain the facial training dataset, preprocess and calibrate the images in the facial training dataset, input the images into the ResNet18 capsule network, and obtain the feature vectors of the images.

[0037] The ResNet18 capsule network consists of a first convolutional block (Conv1_x), a second convolutional block (Conv2_x), a third convolutional block (Conv3_x), and an 8D capsule convolutional layer. The first convolutional block (Conv1_x) comprises a single convolutional layer with a kernel size of 7×7 and a stride of 2. The second convolutional block (Conv2_x) comprises four convolutional layers with a kernel size of 3×3 and a stride of 1. The third convolutional block (Conv3_x) comprises three convolutional layers with a kernel size of 3×3 and a stride of 1, and one convolutional layer with a kernel size of 3×3 and a stride of 2. The main capsule of the 8D capsule convolutional layer is a convolutional unit containing eight 9×9 kernels, and the stride of the 8D capsule convolutional layer is 2.

[0038] The specific steps of S1 are as follows:

[0039] S11. Obtain the facial training dataset, preprocess and calibrate the images in the facial training dataset, and cut the preprocessed and calibrated images to obtain images with a size of 224×224×3.

[0040] S12. Input the image with a size of 224×224×3 into the first convolutional block Conv1_x, the second convolutional block Conv2_x, the third convolutional block Conv3_x and the 8D capsule convolutional layer in sequence to obtain the feature vector of the image.

[0041] An image of size 224×224×3 is input to the first convolutional block Conv1_x, resulting in a 112×112×64 feature map. Max pooling is then performed on this feature map, outputting a 56×56×64 feature map. This 56×56×64 feature map is then input to the second convolutional block Conv2_x, also outputting a 56×56×64 feature map. This 56×56×64 feature map is then input to the third convolutional block Conv3_x, outputting a 28×28×128 feature map. This 28×28×128 feature map is then input to an 8D capsule convolutional layer, outputting a 10×10×8×16 capsule multidimensional entity. A dynamic routing algorithm is then used to transform the capsule multidimensional entity into a feature vector of the image.

[0042] S2. Input the feature vector into the data self-repair module to repair the data and obtain the trained recognition model.

[0043] The data self-repair module includes a self-attention importance weighting module, a rank regularization module, and a label repair module. The self-attention importance weighting module consists of a linear fully connected layer and a sigmoid function.

[0044] The specific steps of S2 are as follows:

[0045] S21. Obtain the N feature vectors corresponding to the N images, input the feature vectors into the self-attention importance weighting module, obtain the importance weight of each feature vector based on the linear fully connected layer, and form a weight matrix according to the importance weight.

[0046] Let N feature vectors be F[X1,X2,…,X]. N ]∈R D×N .

[0047] The importance weights of each feature vector are obtained based on a linear fully connected layer. The expression for the importance weights is as follows:

[0048]

[0049] Where α is the importance weight, SoftMax is the normalization function, X is the feature vector, and d x is the dimension of the feature vector.

[0050] SoftMax is a normalization function that normalizes the importance weights α to a range between 0 and 1. The expression for SoftMax is:

[0051]

[0052] The normalized importance weights together form the weight matrix W. α .

[0053] S22. Perform a weighted summation on the weight matrix and eigenvectors, and obtain N self-attention weights based on the weighted summation result and the Sigmoid function.

[0054] The expression for the self-attention weights is:

[0055]

[0056] Where, α i ′ represents the self-attention weight of the i-th feature vector, W α Let X be the weight matrix. i Let be the i-th eigenvector, and σ be the Sigmoid function.

[0057] S23. Input the self-attention weights into the rank regularization module, classify them according to the preset ratio to obtain high importance weight reorganization and low importance weight reorganization, and calculate the mean α of the high importance weight group. H The mean α of the low importance weight group L α H and α L Satisfy: α H >α L +δ1, where δ1 is the learning parameter.

[0058] Based on the preset proportion classification, the high importance weight reorganization and low importance weight reorganization are obtained specifically as follows: the self-attention weight α is... i Arrange the samples in descending order. Let the preset ratio be β. Use the preset ratio β to obtain the high importance weight reorganization and low importance weight reorganization. The number of samples in the high importance weight group is M = β * N, where M is the number of samples in the high importance weight group and NM is the number of samples in the low importance weight group.

[0059] S24. Input the low importance weight recombination into the label repair module, obtain the maximum prediction probability of the low importance weight recombination, and when the maximum prediction probability of the low importance weight recombination exceeds the first threshold, update the label of the image corresponding to the low importance weight recombination, add the updated image to the face training dataset, and repeat the above training process.

[0060] Since uncertain samples typically have low attention weights, a label repair module can be used to relabel mislabeled data. When sample self-attention importance is low, the label repair module uses the Softmax normalization function to obtain the predicted probability of the image's label corresponding to the low-importance weighted reassembly and selects the one with the highest predicted probability. This highest predicted probability is compared with the probabilities of known classifications. When the highest predicted probability of a low-importance weighted reassembly exceeds a first threshold, the label of the image corresponding to that low-importance weighted reassembly is updated. The updated image's label is the label with the highest predicted probability.

[0061] The update tag module in the tag repair module can be represented as:

[0062]

[0063] Where y′ represents the new label, δ2 is the first threshold, and P max For the maximum prediction probability, P gtInd Given the pre-predicted probability of a sample label, i.e., the probability of a known classification, l org and l max These are the index numbers of the original tag and the updated tag, respectively.

[0064] Low-quality samples are used to obtain lower weight values, reducing their negative impact on network training. Images with repaired labels, however, may receive higher weights in the next iteration, thus having a positive effect. The updated images are then added to the facial training dataset, and the above training process is repeated until training is complete.

[0065] S3. Obtain facial data and input the facial data into the trained recognition model to obtain the recognition result.

[0066] Example diagram of the facial expression recognition results of the present invention is shown below. Figure 3 As shown.

[0067] The framework diagram of the present invention is as follows: Figure 2 As shown.

[0068] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A facial expression recognition method based on capsule network and data repair, characterized in that, The method comprises the following steps: Obtaining facial data, inputting the facial data into a trained recognition model, and obtaining a recognition result, wherein the specific training process of the recognition model is as follows: Obtaining a facial training data set, preprocessing and calibrating images in the facial training data set, cutting the preprocessed and calibrated images to obtain images with a size of 224x224x3, inputting the images into a ResNet18 capsule network to obtain feature vectors of the images, the ResNet18 capsule network comprising a first convolutional block Conv1_x, a second convolutional block Conv2_x, a third convolutional block Conv3_x, and an 8D capsule convolutional layer, the main capsule of the 8D capsule convolutional layer being a convolutional unit comprising eight 9x9 convolutional kernels, and the stride of the 8D capsule convolutional layer being 2; Inputting the feature vectors into a data self-repairing module to perform data repair, the data self-repairing module comprising a self-attention importance weighting module, a rank regularization module, and a label repair module, and the specific process of data repair being as follows: Obtaining N feature vectors corresponding to N images, inputting the feature vectors into the self-attention importance weighting module, the self-attention importance weighting module comprising a linear fully connected layer and a Sigmoid function, obtaining importance weights of each feature vector based on the linear fully connected layer, composing a weight matrix according to the importance weights, performing weighted summation on the weight matrix and the feature vectors, and obtaining N self-attention weights based on the weighted summation result and the Sigmoid function; The self-attention weight is input into a rank regularization module, a high importance weight group and a low importance weight group are obtained based on a preset proportion classification, a mean value a of the high importance weight group is calculated H and a mean value a of the low importance weight group L , a H and a L satisfy: a H > a L + δ1, wherein δ1 is a learning parameter Inputting low importance weight groups into the label repair module, obtaining the maximum prediction probability of the low importance weight groups, updating the labels of the images corresponding to the low importance weight groups when the maximum prediction probability of the low importance weight groups exceeds a first threshold, adding the updated images to the facial training data set, and repeating the above training process until the training is completed.

2. The method of claim 1, wherein, The expression of the importance weight is as follows: where a is an importance weight, SoftMax is a normalization function, X is a feature vector, d x is the dimension of the feature vector.

3. The method of claim 1, wherein the method is based on a capsule network and data repairing. The expression of the self-attention weight is as follows: wherein α′ i is the self-attention weight of the i-th feature vector, W α is the weight matrix, X i is the i-th feature vector, and σ is the Sigmoid function.

4. The method of claim 1, wherein, The first convolutional block Conv1_x comprises one convolutional layer with a convolutional kernel size of 7x7 and a stride of 2.

5. The method of claim 1, wherein the method is based on a capsule network and data repairing. The second convolutional block Conv2_x comprises four convolutional layers with a convolutional kernel size of 3x3 and a stride of 1.

6. The method of claim 1, wherein the method is based on a capsule network and data repairing. The third convolutional block Conv3_x comprises three convolutional layers with a convolutional kernel size of 3x3 and a stride of 1, and one convolutional layer with a convolutional kernel size of 3x3 and a stride of 2.

7. The method of claim 1, wherein the method is based on a capsule network and data repairing. Inputting the images into the ResNet18 capsule network to obtain the feature vectors of the images specifically comprises: inputting the images into the first convolutional block Conv1_x, the second convolutional block Conv2_x, the third convolutional block Conv3_x, and the 8D capsule convolutional layer in sequence to obtain a capsule multidimensional entity with a size of 10x10x8x16, and converting the capsule multidimensional entity into the feature vectors of the images based on a dynamic routing algorithm.

8. The method of claim 1, wherein the method is based on a capsule network and data repairing. The high-importance weight group and the low-importance weight group obtained based on the preset proportion classification are specifically: arranging the self-attention weights in descending order to obtain the high-importance weight group and the low-importance weight group in a preset proportion, wherein the number of samples of the high-importance weight group is M=β*N, wherein β is the preset proportion, M is the number of samples of the high-importance weight group, and the number of samples of the low-importance weight group is N-M.

9. The method of claim 1, wherein the method is based on a capsule network and data repairing. The label of the updated image is the label with the maximum prediction probability.

10. The method of claim 1, wherein the method is based on a capsule network and data repairing. The maximum prediction probability of the low-importance weight group is specifically obtained as follows: the label repairing module obtains the prediction probability corresponding to the label of the image corresponding to the low-importance weight group based on a normalization function Softmax, and screens the maximum prediction probability.

Images