Lightweight dynamic facial expression detection method and system based on self-supervision

Abstract

The invention relates to the technical field of detection, in particular to a self-supervision-based lightweight dynamic facial expression detection method and system, and the method comprises the steps: obtaining a training image from a video set with a facial image, and constructing and training a self-supervision neural network model based on the training image, thereby achieving the detection of the facial expression. The facial expression recognition method comprises the steps of obtaining a facial recognition self-supervised neural network model, processing a dynamic image, and inputting the processed dynamic image into the facial recognition self-supervised neural network model, thereby outputting a facial expression recognition result, and realizing the recognition and detection of the dynamic facial expression.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a lightweight dynamic facial expression detection method and system based on self-supervision. Background Art

[0002] Psychological research shows that in people's daily communication, facial expressions account for a high proportion of various factors in understanding other people's emotions, reaching 55%, while language information is only 7%, and the remaining 38% is conveyed through voice, timbre, tone and other information. Therefore, using facial expressions to identify emotions is a very effective way.

[0003] Currently, facial expression detection or recognition is mostly performed through static images. However, since facial expressions are constantly changing with the fluctuation of emotions, recognizing or detecting dynamic facial expressions is one of the problems that urgently need to be solved. Summary of the invention

[0004] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a lightweight dynamic facial expression detection method and system based on self-supervision, which is helpful to realize the recognition and detection of dynamic facial expressions.

[0005] In order to implement the above technical solution, in a first aspect, the present invention provides a lightweight dynamic facial expression detection method based on self-supervision, comprising: Step 1: Obtain a video set with facial images; Step 2: extracting images from the acquired video set with facial images, and cropping and processing the extracted images to obtain training images; Step 3: Build a self-supervised neural network model based on training images; Step 4: training the constructed self-supervised neural network model to obtain a facial recognition self-supervised neural network model; Step 5: Obtain a fixed number of images to be identified from the dynamic images to be identified, perform image reconstruction and cropping, so as to obtain facial images to be identified; Step 6: Input the obtained facial image to be recognized into the facial recognition self-supervised neural network model to output the facial expression recognition result.

[0006] Furthermore, the step 2 comprises: Use the OpenCV VideoCapture function to read each video in the video set, save each frame in each video as an image, and number each image according to the frame sequence; Use numpy's linspace function to evenly select a fixed number of images from each video frame sequence; Performing facial positioning on each of the selected fixed number of images, and cropping the facial image based on the positioning result; Normalize the cropped facial image; Data augmentation is performed on the normalized facial images to obtain more training images for each video.

[0007] Furthermore, performing facial positioning on each of the selected fixed number of images and cropping the facial image according to the positioning result includes: Obtain the basic facial feature map L_F from each facial image through the local feature extraction network; Obtain the attention feature map A_N from each facial image through the attention network; The acquired facial basic feature map L_F and attention feature map A_N are input into the reconstruction network, and the facial image in each video is reconstructed to separate the facial image from the image, and the separated facial image is cropped to obtain the facial image.

[0008] Furthermore, the self-supervised neural network model includes: an input layer, a first residual block, a second residual block, an attention block, an average pooling layer, a fully connected layer and a Rule activation function layer.

[0009] Furthermore, the step six comprises: Input the obtained facial image to be recognized into the input layer; Performing feature extraction on the input facial image to be recognized by using the first residual block and the second residual block to extract facial features to be recognized from the facial image to be recognized; Extracting attention weights of facial features through an attention module, and obtaining a feature map to be identified of the image to be identified based on the extracted attention weights; Perform average pooling on the feature map to be identified to obtain the feature vector; The obtained feature vectors are concatenated and input into the Rule activation function layer; The Rule activation function outputs the final recognized facial expression based on the splicing result.

[0010] In a second aspect, the present invention provides a lightweight dynamic facial expression detection system based on self-supervision, comprising: A training set acquisition module, used to acquire a video set with facial images; A first processing module is used to extract images from the acquired video set with facial images, and to crop and process the extracted images to obtain training images; A building module for building a self-supervised neural network model based on training images; A training module, used for training the constructed self-supervised neural network model to obtain a facial recognition self-supervised neural network model; A second processing module is used to obtain a fixed number of images to be identified from the dynamic images to be identified, perform image reconstruction and cropping, so as to obtain facial images to be identified; The facial recognition self-supervised neural network model is used to output facial expression recognition results based on the obtained facial image to be recognized.

[0011] Furthermore, the first processing module includes: Image extraction unit: used to obtain a fixed number of images from the video set using the OpenCV VideoCapture function and the NumPy linspace function; Image cropping unit: used to locate the face of each image in a fixed number of images, and crop the face image according to the positioning result; A normalization unit, used for normalizing the cropped facial image; The data enhancement unit is used to perform data enhancement on the normalized facial image to obtain more training images.

[0012] In a third aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned lightweight dynamic facial expression detection method based on self-supervision.

[0013] The beneficial effects of the present invention are: (1) The present invention obtains training images from a video set with facial images, and constructs and trains a self-supervised neural network model based on the training images to obtain a facial recognition self-supervised neural network model. The dynamic image is processed and input into the facial recognition self-supervised neural network model to output facial expression recognition results, thereby realizing the recognition and detection of dynamic facial expressions.

[0014] (2) The present invention uses the VideoCapture function of OpenCV and the Linspace function of NumPy to obtain images from a video set or a dynamic image to be recognized, and normalizes the images, thereby avoiding the huge negative impact of changes in lighting and head posture on the images, which helps to improve the accuracy of recognition.

[0015] (3) The present invention adds an attention module to the self-supervised neural network model. The attention module obtains feature maps through channel attention mechanism and spatial attention mechanism, so that the model can focus on features that are useful for expression recognition. In addition, an average pooling layer is set after the attention module. By performing average pooling on the obtained feature maps, a feature vector is obtained, which helps to reduce the number of parameters and computational complexity of the model and avoid overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0017] Figure 1 The present invention is a flowchart of a lightweight dynamic facial expression detection method based on self-supervision. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0019] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present invention. Unless otherwise specified, each technical and scientific term used in this embodiment has the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0020] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0021] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0022] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. Relevant scientific research or technical personnel in this field can determine the specific meanings of the above terms in the present invention according to specific circumstances, and they should not be understood as limiting the present invention. Embodiment 1: like Figure 1 As shown, this embodiment provides a lightweight dynamic facial expression detection method based on self-supervision, comprising the following steps: S1: Get a video set with facial images.

[0023] S2: extracting images from the acquired video set with facial images, and cropping and processing the extracted images to obtain training images.

[0024] S21: Use the VideoCapture function of opencv to read each video in the video set, save each frame in each video as an image, and number each image according to the frame sequence.

[0025] S22: Use numpy's linspace function to uniformly select a fixed number of images from each video frame sequence; S23: performing face positioning on each of the fixed number of images selected from each video, and cropping the face image according to the positioning result.

[0026] Specifically, the steps include: A) Obtain the basic facial feature map L_F from each facial image in each video through the local feature extraction network (Local Feature Net); B) Obtain the attention feature map A_N from each facial image in each video through the Attention Net; C) Input all the basic feature maps L_F and all the attention feature maps A_N of all faces in each video into the reconstruction network (Rec-Net), reconstruct the facial image in each video to separate the facial image from the image, and crop the separated facial image to obtain the facial image in each video.

[0027] S24: Normalize the cropped facial image.

[0028] Specifically, since changes in illumination and head posture will have a huge negative impact on the image, in order to improve the ability of facial recognition, the facial image is normalized, where normalization includes illumination normalization (e.g., global contrast normalization (GCN)) and posture normalization.

[0029] S25: Perform data augmentation on the normalized facial images to obtain more training images for each video.

[0030] Specifically, the normalized facial images are augmented by rotation, flipping, and scaling to increase the number of samples in the training set and improve the stability and generalization of the model. Among them, data augmentation techniques such as random rotation and random horizontal flipping are used. For example, the rotation angle range is (-15, 15) degrees, and the scaling range is (0.8, 1.2) times.

[0031] S3: Build a self-supervised neural network model based on training images.

[0032] The self-supervised neural network model includes: input layer, first residual block, second residual block, attention block, average pooling layer, fully connected layer and Rule activation function layer.

[0033] A) Input layer, used to input the obtained training images.

[0034] B) The first residual block and the second residual block are used to extract features from the input training image to extract facial features from the training image. The first residual block and the second residual block both use dilated convolution to extract facial features from the input facial image, which helps to expand the range of the receptive field.

[0035] C) an attention module, used to extract attention weights according to the extracted facial features, and obtain a final feature map based on the extracted attention weights; Among them, the attention layer adopts channel attention mechanism and spatial attention mechanism.

[0036] 1) Channel attention mechanism is used to: (a) The extracted facial features are subjected to maximum pooling and average pooling respectively to obtain the maximum pooling attention matrix and the average pooling attention matrix.

[0037] (b) Add the obtained maximum pooling attention matrix and the average pooling attention matrix, and input the addition result into the Sigmoid function to obtain the channel attention weight.

[0038] (c) Multiply the obtained channel attention weight by the input face feature to obtain the channel attention feature map.

[0039] 2) The spatial attention mechanism is used to: (a) The obtained channel attention feature map is subjected to global average pooling and maximum pooling in sequence to obtain a two-dimensional feature map; (b) Use 2D convolution (e.g., 7×7 2D convolution) to extract spatial attention from the 2D feature map, and input the extracted attention into the Sigmoid function to obtain the spatial attention weight; (c) Multiply the spatial attention weights with the channel attention feature map to obtain the spatial feature map.

[0040] This paper uses attention mechanism and spatial attention mechanism to enable the model to focus on features that are useful for expression recognition.

[0041] D) Average pooling layer, which is used to perform average pooling on the spatial feature map of each video to obtain a feature vector.

[0042] E) Fully connected layer, used to concatenate the obtained feature vectors and input them into the Rule activation function layer; F) Rule activation function layer, which is used for the Rule activation function to output the final recognized facial expression based on the splicing result.

[0043] S4: Train the constructed self-supervised neural network model to obtain a facial recognition self-supervised neural network model.

[0044] Specifically, the steps include: A) Use cross entropy as the loss function to update the weight parameters of the model. The specific formula is as follows:

[0045] in, is the true value; is the predicted value; N is the number of samples, and i is the i-th sample in sample N. By continuously reducing the loss value through back propagation, the weight parameters of the neural network are updated to improve the accuracy of model prediction.

[0046] B) Use the Adam optimizer to optimize the model parameters, where the optimizer formula is as follows:

[0047] Where t is the current time step; and They represent the first-order moment estimate and second-order moment estimate of the gradient at the current time step t, β1 and β2 are the exponential decay rates of the first-order moment and second-order moment of the gradient, respectively; and They represent the first-order moment estimate and second-order moment estimate of the gradient after bias correction respectively; is a small constant, α represents the learning rate, Represents the weights and biases of the model.

[0048] S5: A fixed number of images to be recognized are obtained from the dynamic images to be recognized, and image reconstruction and cropping are performed to obtain facial images to be recognized.

[0049] The specific steps include: S41: Use the VideoCapture function of opencv to read the dynamic image, save each frame in the dynamic image as an image, and number each image according to the frame sequence.

[0050] S42: Use numpy's linspace function to evenly select a fixed number of images from the saved images; S43: Positioning the face in each of the selected fixed number of images, and cropping the face image to be recognized according to the positioning result.

[0051] Specifically, the steps include: A) Obtain the basic facial feature map L_F from each image through the local feature extraction network (Local Feature Net); B) Obtain the attention feature map A_N from each facial image through the Attention Net; C) The acquired basic feature maps L_F of all faces and all attention feature maps A_N are input into the reconstruction network (Rec-Net) for reconstruction. According to the reconstruction results, the face is separated from the image, and the separated face is cropped to obtain the facial image to be recognized.

[0052] S6: Input the obtained facial image to be recognized into the facial recognition self-supervised neural network model to output the facial expression recognition result.

[0053] The specific steps include: S51: Input the obtained facial image to be recognized into the input layer; S52: extracting features of the input facial image to be identified by using the first residual block and the second residual block to extract facial features to be identified from the facial image to be identified. The first residual block and the second residual block both use dilated convolution to extract facial features of the input facial image, which helps to expand the range of the receptive field.

[0054] S53: extracting attention weights of facial features through an attention module, and obtaining a feature map to be recognized of the image to be recognized based on the extracted attention weights; Among them, the attention layer includes: channel attention mechanism and spatial attention mechanism.

[0055] A) The channel attention mechanism is used to: (1) The extracted facial features are subjected to maximum pooling and average pooling respectively to obtain the maximum pooling attention matrix and the average pooling attention matrix.

[0056] (2) Add the obtained maximum pooling attention matrix and the average pooling attention matrix, and input the addition result into the Sigmoid function to obtain the channel attention weight.

[0057] (3) Multiply the obtained channel attention weight by the input facial features to obtain the channel attention feature map; B) The spatial attention mechanism is used to: (1) Perform global average pooling and maximum pooling on the obtained channel attention feature map to obtain a two-dimensional feature map; (2) Use two-dimensional convolution (for example, 7×7 two-dimensional convolution) to extract spatial attention from the two-dimensional feature map, and input the extracted attention into the Sigmoid function to obtain the spatial attention weight; (3) Multiply the spatial attention weight and the channel attention feature map to obtain the feature map to be identified.

[0058] This paper uses attention mechanism and spatial attention mechanism to enable the model to focus on features that are useful for expression recognition.

[0059] S54: performing average pooling on the feature map to be identified to obtain a feature vector; S55: concatenate the obtained feature vectors and input them into the Rule activation function layer; S56: Rule activation function outputs the final recognized facial expression based on the concatenation result.

[0060] Embodiment 2: This embodiment provides a lightweight dynamic facial expression detection system based on self-supervision, including: The training set acquisition module is used to acquire a video set with facial images.

[0061] A first processing module is used to extract images from the acquired video set with facial images, and to crop and process the extracted images to obtain training images; The first processing module comprises: Image extraction unit: used to obtain a fixed number of images from a video set using OpenCV's VideoCapture function and NumPy's linspace function.

[0062] Image cropping unit: used to locate the face in each of a fixed number of images and crop the face image based on the positioning result.

[0063] The normalization unit is used to normalize the cropped facial image.

[0064] The data enhancement unit is used to perform data enhancement on the normalized facial image to obtain more training images.

[0065] A building block for building a self-supervised neural network model based on training images.

[0066] The self-supervised neural network model includes: A) an input layer, which is used to input the obtained training image.

[0067] B) The first residual block and the second residual block are used to extract features from the input training image to extract facial features from the training image. The first residual block and the second residual block both use dilated convolution to extract facial features from the input facial image, which helps to expand the range of the receptive field.

[0068] C) an attention module, used to extract attention weights according to the extracted facial features, and obtain a final feature map based on the extracted attention weights; Among them, the attention layer adopts channel attention mechanism and spatial attention mechanism.

[0069] 1) Channel attention mechanism is used to: (a) The extracted facial features are subjected to maximum pooling and average pooling respectively to obtain the maximum pooling attention matrix and the average pooling attention matrix.

[0070] (b) Add the obtained maximum pooling attention matrix and the average pooling attention matrix, and input the addition result into the Sigmoid function to obtain the channel attention weight.

[0071] (c) Multiply the obtained channel attention weight by the input face feature to obtain the channel attention feature map.

[0072] 2) The spatial attention mechanism is used to: (a) The obtained channel attention feature map is subjected to global average pooling and maximum pooling in sequence to obtain a two-dimensional feature map; (b) Use 2D convolution (e.g., 7×7 2D convolution) to extract spatial attention from the 2D feature map, and input the extracted attention into the Sigmoid function to obtain the spatial attention weight; (c) Multiply the spatial attention weights with the channel attention feature map to obtain the spatial feature map.

[0073] This paper uses attention mechanism and spatial attention mechanism to enable the model to focus on features that are useful for expression recognition.

[0074] D) Average pooling layer, which is used to perform average pooling on the spatial feature map of each video to obtain a feature vector.

[0075] E) Fully connected layer, used to concatenate the obtained feature vectors and input them into the Rule activation function layer; F) Rule activation function layer, which is used for the Rule activation function to output the final recognized facial expression based on the splicing result.

[0076] The training module is used to train the constructed self-supervised neural network model to obtain a facial recognition self-supervised neural network model.

[0077] A second processing module is used to obtain a fixed number of images to be identified from the dynamic images to be identified, perform image reconstruction and cropping, so as to obtain facial images to be identified; The facial recognition self-supervised neural network model is used to output facial expression recognition results based on the obtained facial image to be recognized.

[0078] Embodiment 3: This embodiment provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the lightweight dynamic facial expression detection method based on self-supervision described in Example 2.

[0079] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0080] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, it should be noted that the flowchart in the accompanying drawings shows the method of the embodiment of the present disclosure. In the corresponding description in the flowchart or block diagram in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be performed substantially in parallel, and they can sometimes be performed in the opposite order, which can depend on the functions involved. Each block in the block diagram and / or flow chart, and the combination of blocks in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A lightweight dynamic facial expression detection method based on self-supervision, characterized in that: include: Step 1: Obtain a video set with facial images; Step 2: extracting images from the acquired video set with facial images, and cropping and processing the extracted images to obtain training images; Step 3: Build a self-supervised neural network model based on training images; Step 4: training the constructed self-supervised neural network model to obtain a facial recognition self-supervised neural network model; Step 5: Obtain a fixed number of images to be identified from the dynamic images to be identified, perform image reconstruction and cropping, so as to obtain facial images to be identified; Step 6: Input the obtained facial image to be recognized into the facial recognition self-supervised neural network model to output the facial expression recognition result.

2. The self-supervised lightweight dynamic facial expression detection method according to claim 1, characterized in that: The second step comprises: Use the OpenCV VideoCapture function to read each video in the video set, save each frame in each video as an image, and number each image according to the frame sequence; Use numpy's linspace function to evenly select a fixed number of images from each video frame sequence; Performing facial positioning on each of the selected fixed number of images, and cropping the facial image based on the positioning result; Normalize the cropped facial image; Data augmentation is performed on the normalized facial images to obtain more training images for each video.

3. The self-supervised lightweight dynamic facial expression detection method according to claim 2, characterized in that: The step of performing facial positioning on each of the selected fixed number of images and cropping the facial image according to the positioning result comprises: Obtain the basic facial feature map L_F from each facial image through the local feature extraction network; Obtain the attention feature map A_N from each facial image through the attention network; The acquired facial basic feature map L_F and attention feature map A_N are input into the reconstruction network, and the facial image in each video is reconstructed to separate the facial image from the image, and the separated facial image is cropped to obtain the facial image.

4. The self-supervised lightweight dynamic facial expression detection method according to claim 1, characterized in that: The self-supervised neural network model includes: an input layer, a first residual block, a second residual block, an attention block, an average pooling layer, a fully connected layer and a Rule activation function layer.

5. The self-supervised lightweight dynamic facial expression detection method according to claim 4, characterized in that: The step six comprises: Input the obtained facial image to be recognized into the input layer; Performing feature extraction on the input facial image to be recognized by using the first residual block and the second residual block to extract facial features to be recognized from the facial image to be recognized; Extracting attention weights of facial features through an attention module, and obtaining a feature map to be identified of the image to be identified based on the extracted attention weights; Perform average pooling on the feature map to be identified to obtain the feature vector; The obtained feature vectors are concatenated and input into the Rule activation function layer; The Rule activation function outputs the final recognized facial expression based on the splicing result.

6. A lightweight dynamic facial expression detection system based on self-supervision, characterized in that: include: A training set acquisition module, used to acquire a video set with facial images; A first processing module is used to extract images from the acquired video set with facial images, and to crop and process the extracted images to obtain training images; A building module for building a self-supervised neural network model based on training images; A training module, used for training the constructed self-supervised neural network model to obtain a facial recognition self-supervised neural network model; A second processing module is used to obtain a fixed number of images to be identified from the dynamic images to be identified, perform image reconstruction and cropping, so as to obtain facial images to be identified; The facial recognition self-supervised neural network model is used to output facial expression recognition results based on the obtained facial image to be recognized.

7. The self-supervised lightweight dynamic facial expression detection system according to claim 6, characterized in that: The first processing module comprises: Image extraction unit: used to obtain a fixed number of images from the video set using the OpenCV VideoCapture function and the NumPy linspace function; Image cropping unit: used to locate the face of each image in a fixed number of images, and crop the face image according to the positioning result; A normalization unit, used for normalizing the cropped facial image; The data enhancement unit is used to perform data enhancement on the normalized facial image to obtain more training images.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute a lightweight dynamic facial expression detection method based on self-supervision as described in any one of claims 1 to 5.

Images