Shielding expression recognition method and system, storage medium and electronic equipment
By using residual module, multi-scale feature extraction module and densely connected structure network in facial expression recognition, combining channel loss function and space loss function, extracting and learning the features of occluding expression images, the background and lighting interference is solved. The problem of low recognition accuracy is achieved, and higher recognition accuracy and robustness are achieved.
Patent Information
- Application Number
- CN202410977385.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-22
AI Technical Summary
In the recognition of facial expressions, the recognition accuracy of the existing technology is low due to interference from factors such as background, lighting and occlusion. Traditional convolutional neural networks cannot effectively extract multi-dimensional image features, especially the recognition accuracy of the occluded expression images is not high.
The local feature information of the occluded expression image is extracted using a network containing residual module, multi-scale feature extraction module and dense connection structure, and input it into the ViT network for further learning, combining the channel loss function and the space loss function for training to maximize the features intra-class compactness and inter-class separability.
The recognition accuracy of occluded expression images is improved by 2.88%, the expression and generalization capabilities of the network are enhanced, and high performance and robustness can be maintained in complex environments.
Smart Images

Figure CN119007258A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision human face expression recognition, and in particular to an expression masking method, system, storage medium and electronic equipment. Background Art
[0002] Expression recognition technology has received widespread attention in human-computer interaction and intelligent information processing. However, in real life scenarios, facial images are often interfered by environmental factors such as background, lighting and occlusion, which greatly reduces the recognition accuracy of facial expressions.
[0003] Traditional convolutional neural networks cannot extract feature information of images from multiple dimensions, resulting in the network model being unable to extract useful expression features, and the recognition accuracy of occluded expression images is also low. Therefore, for the recognition of occluded facial expression images, a network including a residual module, a multi-scale feature extraction module, and an Inception module with a dense connection structure is used to extract local feature information of occluded expression images, improve the network model's perception and expression capabilities for different features, and then fuse these feature information and input them into the ViT network for further learning. During the training process, channel loss functions and spatial loss functions specific to expression recognition tasks are used to maximize the intra-class compactness and inter-class separability of the features extracted by the network, which helps suppress the overfitting phenomenon of the model, improve the generalization ability of the model, and improve the accuracy of occluded expression recognition.
[0004] In the prior art, the Chinese invention patent publication number is "CN117152722", and its name is "Lightweight driver facial expression recognition method based on multi-scale convolutional neural network". This method extracts facial image features from multiple scales by designing a multi-scale residual attention module, and constructs an efficient network to take corresponding measures according to the driver's facial expression state. However, during driving, the driver's face may be affected by factors such as background, lighting and occlusion, resulting in the face being occluded and the driver's expression cannot be accurately recognized; therefore, we propose an occluded expression recognition method to solve the above problem. Summary of the invention
[0005] 1. Technical issues to be resolved
[0006] In view of the deficiencies in the prior art, the present invention provides a method for recognizing occluded facial expressions, which solves the problems raised in the above-mentioned background technology.
[0007] (II) Technical solution
[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:
[0009] A method for recognizing occluded facial expressions comprises the following steps:
[0010] Step 1, preprocessing the facial expression image, including grayscale normalization, data enhancement and scale normalization;
[0011] Step 2: Build a facial expression recognition network model based on occlusion, including a residual module, a multi-scale feature extraction module, and a densely connected Inception module fusion network to obtain local feature information of the occluded expression image. After fusing these features, input them into the ViT network for further learning, and finally output the expression classification result, effectively improving the recognition accuracy of the occluded expression image.
[0012] Step 3, iteratively train the occluded expression recognition network model. During the training process, the channel loss function and the spatial loss function are combined to obtain a new loss function, which is used to determine the training loss and maximize the intra-class compactness and inter-class separability of the features extracted by the network;
[0013] Step 4: Input the occluded facial expression image into the trained network model and output the occluded facial expression recognition result.
[0014] Furthermore, in step 1, the occluded expression image is preprocessed, including grayscale normalization and data enhancement: a black rectangular box is added to the facial expression as an occlusion, and the data set is expanded and scaled by rotating it at different angles.
[0015] Furthermore, the facial expression recognition network model based on occlusion constructed in the step 2 is a 15-layer residual neural network, including 6 layers of Inception modules, 7 layers of residual feature extraction modules, and 2 layers of multi-scale feature extraction modules. The local features of the occluded facial expressions are extracted and fused, and the fused feature information is input into the ViT network for further learning to output the expression category.
[0016] Furthermore, the training process in step 3 combines the channel loss function and the spatial loss function.
[0017] The calculation formula of the channel loss function is:
[0018] L Channel-loss (F) = 0.01 × (L1 (F) - 9 × L2 (F))
[0019] The spatial loss function is:
[0020]
[0021] The total loss function calculation formula is:
[0022]
[0023] Where α and It is the influence weight that controls the channel loss function and the spatial loss function. Selecting appropriate weights can maximize the intra-class compactness and inter-class separability of the features extracted by the network.
[0024] Furthermore, in step 4, the occluded facial expression image is input into the trained network model, and the occluded facial expression recognition result is output.
[0025] An occluded expression recognition system includes the following modules:
[0026] Preprocessing module: Processing of occluded expression images, including face cropping, grayscale normalization, data enhancement and scale normalization.
[0027] Feature extraction module: The feature extraction module includes the Inception module, residual module 1, residual module 2 and multi-scale feature extraction module. The dense connection structure is used for the Inception module to complete the extraction of local feature information of occluded expression images. After the local feature information is fused, it is input into the ViT network for further learning, and finally the expression classification result is output, which effectively improves the recognition accuracy of occluded expression images.
[0028] Network training module: During the network training process, the channel loss function and spatial loss function specific to the expression recognition task are used to reduce the intra-class distance of different expression features and increase the inter-class distance. This effectively establishes the connection between the deep feature information and the expression category in the feature extraction network, maximizes the intra-class compactness and inter-class separability of the features extracted by the network, and enables the network to extract deep features with expression discrimination capabilities.
[0029] A storage medium stores a computer program, which, when executed by a processor, implements an occluded expression recognition method as described in claims 1-5.
[0030] An electronic device, comprising:
[0031] A memory for storing instructions;
[0032] A processor is used to execute the instructions; the processor is configured to execute the method for recognizing occluded facial expressions described in claims 1-5 by executing executable instructions.
[0033] (III) Beneficial effects
[0034] Compared with the prior art, the present invention provides a method, system, medium and electronic device for recognizing occluded facial expressions, which have the following beneficial effects:
[0035] 1. The present invention uses a network with a residual module, a multi-scale feature extraction module and an Inception module with a dense connection structure to extract local feature information of occluded expressions, solving the problem of insufficient feature extraction.
[0036] 2. In the present invention, the extracted local feature information is fused and then input into the ViT network for further learning, and finally the expression classification result is output, which improves the recognition accuracy of occluded expressions by 2.88%
[0037] 3. The present invention combines the channel loss function and the space loss function during the network training process, so that the intra-class distance of different expression features is reduced and the inter-class distance is increased, so that the network can extract deep features with expression discrimination ability while optimizing the classification task and the discrimination ability of features. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a flow chart of the facial expression recognition method of the present invention;
[0039] Figure 2 It is a local feature extraction network structure diagram of the present invention;
[0040] Figure 3 Schematic diagram of the residual module designed in the method of the present invention;
[0041] Figure 4 Schematic diagram of a multi-scale feature extraction module designed in the method of the present invention;
[0042] Figure 5 Schematic diagram of the Inception module in the method of the present invention;
[0043] Figure 6 This is the ViT network structure diagram of the present invention;
[0044] Figure 7 FIG. 4 is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] Example
[0047] like Figure 1-7 As shown, an embodiment of the present invention provides a method for recognizing occluded facial expressions, comprising the following steps:
[0048] Step 1: pre-process the occluded expression image to remove irrelevant background information to prevent this information from causing obstacles in subsequent expression recognition and affecting the final expression recognition accuracy.
[0049] Furthermore, grayscale normalization: while retaining image details, the number of channels is reduced, the amount of data of the original image is reduced, and the facial expression image after face detection is converted into a grayscale image. Data enhancement: a large number of virtual images are generated using the original image. These images are different from the original images, but the label categories remain unchanged. First, black rectangular blocks of the same area are randomly placed in the original image to expand the data set to twice the original size, and then the image is rotated at different angles. The rotation angles are set to -10°, -5°, 0°, 5°, and 10°. The image is expanded by another 5 times through rotation, and finally the original data set is expanded by 10 times.
[0050] Step 2, build a facial expression recognition network model based on occlusion, which is a 15-layer residual neural network, including 6 layers of Inception modules, 7 layers of residual feature extraction modules, and 2 layers of multi-scale feature extraction modules. The local features of the occluded facial expressions are extracted and fused, and the fused feature information is input into the ViT network for further learning and then the expression category is output.
[0051] Furthermore, the Inception feature extraction module is composed of 6 Inception modules and adopts a densely connected structure to achieve feature reuse of expression images, reducing the number of parameters and computational complexity while improving the network's ability to represent and discriminate expressions.
[0052] Furthermore, the residual feature extraction module uses the Swish activation function, which is a nonlinear activation function with smoothness and nonlinear characteristics. The Swish activation function is used in the residual network to alleviate the problem of gradient disappearance in network training and produce a stronger regularization effect. The activation function is as follows:
[0053]
[0054] Furthermore, the feature map F′ obtained by the multi-scale feature extraction module can be expressed as:
[0055] F1=ReLU(ReLU(ReLU(AvgPool(conv(F,k 1×1 )+b2),k 3×3 ),k 3×3 ))
[0056] F2=ReLU(ReLU(ReLU(conv(F,k 1×1 )+b3),k1×1 ),k 1×1 )
[0057] F3=ReLU(conv(MaxPool(F,k 3×3 )+b1)
[0058] F′=Concat(F1,F2,F3)
[0059] Among them, F1, F2 and F3 are the feature maps obtained after passing through the three branches, k i×i is a convolution kernel of size i×i. i For bias.
[0060] Furthermore, the local information extracted by the residual module, multi-scale feature extraction module and Inception module is fused and input into the Vit network for further learning. The ViT network includes an image embedding module, a category embedding module, 6 Transformer encoders and a multi-layer perceptron classification layer. The image is divided into images of fixed size, each image is linearly embedded and a learnable classification label is added, and the obtained vector sequence is used as the input of the Transformer encoder.
[0061] The feature extraction network of the present invention can extract features at multiple scales, help suppress the interference of interference factors on feature extraction, and by learning the correlation and importance of features at different scales, can reduce feature disturbances caused by factors such as background noise, occlusion or deformation, obtain key feature information in expression images, weaken invalid feature information, which is conducive to the effective feature extraction of occluded expression images, improve the expression ability and generalization ability of the model, and thus improve the classification accuracy of occluded expression images.
[0062] Step 3: Combine the channel loss function and the spatial loss function during training. The channel loss function consists of two components: the discriminative loss L1(F) and the diversity loss L2(F).
[0063] First, the feature F of each category i Perform a global average pooling operation to obtain a c-dimensional vector y(F i ), the calculation formula is as follows:
[0064]
[0065] W and H are the width and height of the feature map, M i Represents the random selection of features for each type of input Perform a weight-returning mask operation, then multiply it with the original feature point and redistribute the weight. max[] represents the maximum value of the corresponding channel feature, and y(·) represents the global average pooling operation.
[0066] The formula for calculating the discriminative loss is:
[0067]
[0068] y represents the true label and the discriminative loss helps to distinguish this class from other classes.
[0069] L CE is the cross entropy loss function:
[0070]
[0071] Where M is the total number of samples, and for N categories, an N-dimensional vector α is output. i (i=1,…,N), target is the sample label, y i Represents the label vector of the sample, It represents the probability that the sample is classified into the i-th category.
[0072] The formula for calculating diversity loss is:
[0073]
[0074] The diversity loss is used to calculate the total similarity of all channels. The Softmax function is first used to normalize the spatial dimension, and then the maximum pooling is used to obtain the maximum response feature map of each category. Finally, the features calculated on each category are averaged to obtain the diversity loss.
[0075] The calculation formula of the channel loss function is:
[0076] L Channel-loss (F) = 0.01 (L1 (F) - 9 × L2 (F))
[0077] The spatial loss function is:
[0078]
[0079] Where m is the number of samples in a training batch, N H is the number of sample categories, x belongs to x i Deep features within k nearest neighbors of the same category.
[0080] The total loss function calculation formula is:
[0081]
[0082] Where α and It is the influence weight that controls the channel loss function and the spatial loss function. Selecting appropriate weights can maximize the intra-class compactness and inter-class separability of the features extracted by the network.
[0083] Step 4: Input the occluded expression image into the trained network model and output the occluded expression category.
[0084] Furthermore, the images in the dataset are divided into training set and test set in a ratio of 4:1. The occluded facial expression images in the training set are input into the trained network model for processing and output a probability vector representing the probability of each expression category recognized by the network model. The maximum value is taken as the corresponding expression category as the recognition result of the facial expression image in the test set. Finally, the recognition result of each facial expression image in the test set is obtained.
[0085] The present invention is a method for recognizing occluded expressions. A residual module, a multi-scale feature extraction module, and an Inception module with a dense connection structure are designed to extract local feature information of facial expression images. During the training process, a channel loss function and a spatial loss function specific to the expression recognition task are used to maximize the intra-class compactness and inter-class separability of the features extracted by the network, thereby improving the recognition accuracy of occluded expressions. Under the same conditions, the feasibility and superiority of the method are further verified by calculating the relevant indicators of the expression category recognized by the existing method. The accuracy comparison between the existing technology and the method proposed by the present invention is shown in Table 1:
[0086] Table 1 Comparison of the accuracy of the prior art and the method proposed in this invention
[0087]
[0088] It can be seen from Table 1 that the accuracy of the method proposed in the present invention in the training set is 2.88% higher than that of the prior art. The method proposed in the present invention has a higher accuracy, which further illustrates that the network model of expression recognition in the present invention shows strong robustness when facing noise, illumination changes, posture changes, etc., and it can still maintain high performance and generalization ability in complex environments.
[0089] An occluded expression recognition system includes the following modules:
[0090] Preprocessing module: Processing of occluded expression images, including face cropping, grayscale normalization, data enhancement and scale normalization.
[0091] Feature extraction module: The feature extraction module includes the Inception module, residual module 1, residual module 2 and multi-scale feature extraction module. The dense connection structure is used for the Inception module to complete the extraction of local feature information of occluded expression images. After the local feature information is fused, it is input into the ViT network for further learning, and finally the expression classification result is output, which effectively improves the recognition accuracy of occluded expression images.
[0092] Network training module: During the network training process, the channel loss function and spatial loss function specific to the expression recognition task are used to reduce the intra-class distance of different expression features and increase the inter-class distance. This effectively establishes the connection between the deep feature information and the expression category in the feature extraction network, maximizes the intra-class compactness and inter-class separability of the features extracted by the network, and enables the network to extract deep features with expression discrimination capabilities.
[0093] The embodiment of the present invention further provides a computer-readable storage medium. A person skilled in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by specifying a processor through a program. The program can be stored in a computer-readable storage medium. The storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a tape, a floppy disk, an optical disk, and any combination thereof. The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated therein.
[0094] An embodiment of the present invention further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store computer programs, including various media that can store program codes, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.
[0095] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned occluded expression recognition method. The processor can be a general-purpose processor, including a central processing unit, a network processor, etc. The electronic device of the present invention is expressed in the form of a general-purpose computing device. The components of the electronic device include but are not limited to one or more processors or processing units, memories, and buses connecting different system components.
[0096] Electronic devices typically include a variety of computer system readable media, which can be any available media that can be accessed by the electronic device and includes both volatile and nonvolatile media, removable and non-removable media.
[0097] The memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM), cache memory.
[0098] The electronic device can also communicate with one or more networks through a network adapter, and the electronic device can communicate with one or more external devices. Such communication can be carried out through an input / output (I / O) interface. The processor and memory in the electronic device of the present invention can be set in the electronic device. The electronic device calls the expression recognition program stored in the memory through the processor and executes the expression recognition method provided by the embodiment of the present invention.
[0099] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for recognizing occluded facial expressions, Features: The following steps are included: Step 1, preprocessing the facial expression image, including grayscale normalization, data enhancement and scale normalization; Step 2: Build a facial expression recognition network model based on occlusion, including a residual module, a multi-scale feature extraction module, and a densely connected Inception module fusion network to obtain local feature information of the occluded expression image. After fusing these features, input them into the ViT network for further learning, and finally output the expression classification result, effectively improving the recognition accuracy of the occluded expression image. Step 3, iteratively train the occluded expression recognition network model. During the training process, the channel loss function and the spatial loss function are combined to obtain a new loss function, which is used to determine the training loss and maximize the intra-class compactness and inter-class separability of the features extracted by the network; Step 4: Input the occluded facial expression image into the trained network model and output the occluded facial expression recognition result.
2. The method for recognizing occluded facial expressions according to claim 1, Features: In the step 1, the image is preprocessed before the occluded expression image is input into the network, including grayscale normalization and data enhancement: a black rectangular box is added to the facial expression as an occlusion, and the data set is expanded and scaled by rotating it at different angles.
3. The method for recognizing occluded facial expressions according to claim 1, Features: The multi-scale feature extraction module of the facial expression recognition network model with occlusion constructed in step 2 can extract multi-scale feature information of the occluded expression image, obtain key feature information in the occluded expression image, and weaken invalid feature information.
4. The method for recognizing occluded facial expressions according to claim 1, Features: The multi-scale feature extraction module in step 2 enhances the network's perception of information at different scales and improves the generalization ability of the model. A dense connection structure is used for the Inception module. Each convolutional layer and each subsequent layer are connected. The input of each convolutional layer is all the feature maps before it, and its own feature map is also input to all subsequent convolutional layers. The dense connection structure realizes feature reuse.
5. The method for recognizing occluded facial expressions according to claim 1, Features: In the step 4, during the network training process, a channel loss function and a spatial loss function specific to the expression recognition task are used to maximize the intra-class compactness and inter-class separability of the features extracted by the network.
6. An occluded expression recognition system, Features: Includes the following modules: Preprocessing module: Processing of occluded expression images, including face cropping, grayscale normalization, data enhancement and scale normalization; Feature extraction module: The feature extraction module includes the Inception module, residual module 1, residual module 2 and multi-scale feature extraction module. The dense connection structure is used for the Inception module to complete the extraction of local feature information of the occluded expression image. After the local feature information is fused, it is input into the ViT network for further learning, and finally the expression classification result is output, which effectively improves the recognition accuracy of the occluded expression image. Network training module: During the network training process, the channel loss function and spatial loss function specific to the expression recognition task are used to reduce the intra-class distance of different expression features and increase the inter-class distance, effectively establishing the connection between the deep feature information and the expression category in the feature extraction network, maximizing the intra-class compactness and inter-class separability of the features extracted by the network, so that the network can extract deep features with expression discrimination ability; Implement the occluded expression recognition method described in claims 1-5.
7. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for recognizing occluded facial expressions described in claims 1-5 is implemented.
8. An electronic device, It is characterized in that include: A memory for storing instructions; A processor, configured to execute the instructions; The processor is configured to execute the method for recognizing occluded facial expressions described in claims 1-5 by executing executable instructions.
Citation Information
Patent Citations
Lightweight driver facial expression recognition method based on multi-scale convolutional neural network
CN117152722A
