A method of classifying facial expressions and an electronic device

Abstract

This application relates to the field of image processing and provides a method for classifying facial expressions, comprising: acquiring a target image including the facial expression of a target object; inputting the target image into an expression classification model to obtain a classification result, the classification result indicating the degree of emotional expression of the facial expression; the expression classification model includes: a first feature extraction network, a second feature extraction network, and a fusion classification module; extracting local features of the target image through the first feature extraction network to obtain a first feature of the region where the facial expression is located; extracting global features of the target image through the second feature extraction network to obtain a second feature; and fusing and classifying the first and second features through the fusion classification module to obtain a classification result. The above method can improve the extraction rate of emotional features of facial expressions, thereby solving the problem of low accuracy in facial expression classification.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a method for classifying facial expressions and an electronic device. Background Technology

[0002] In recent years, facial expression classification has become a research hotspot in the field of image processing. For example, the classification of facial pain expressions is one of the hot topics in medical research. Convolutional neural networks are commonly used to classify the pain level of facial pain expressions in newborns, critically ill patients, and patients with aphasia. However, the results of existing convolutional neural networks in extracting pain features from facial pain expressions are not ideal, thus affecting the accuracy of their pain level classification based on these extracted pain features.

[0003] Therefore, improving the accuracy of facial expression classification is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method and electronic device for classifying facial expressions, which can solve the problem of low accuracy in facial expression classification.

[0005] In a first aspect, a method for classifying facial expressions is provided, comprising: acquiring a target image, the target image including a facial expression of a target object; inputting the target image into an expression classification model to obtain a classification result, the classification result being used to indicate the degree of emotional expression of the facial expression; the expression classification model comprising: a first feature extraction network, a second feature extraction network, and a fusion classification module; performing local feature extraction on the target image through the first feature extraction network to obtain a first feature of the region where the facial expression is located; performing global feature extraction on the target image through the second feature extraction network to obtain a second feature; and performing feature fusion and classification on the first feature and the second feature through the fusion classification module to obtain the classification result.

[0006] The above method can be executed by a chip on an electronic device. Compared with existing convolutional neural network methods that only extract local features of facial expressions of target objects, this application uses a parallel first feature extraction network and a second feature extraction network to form an expression classification model to extract both local and global features of facial expressions of target objects. The second feature extraction network extracts global features of emotional features of facial expressions, which can make up for important feature information missed by the first feature extraction network when extracting local features of facial expressions, thereby improving the extraction rate of emotional features of facial expressions and thus improving the accuracy of classifying the degree of emotional expression based on the feature extraction results of facial expressions.

[0007] Optionally, the first feature extraction network is VGG16, and the input layer of the VGG16 includes a local attention layer, which is used to perform information attenuation operation on the area outside the facial expression area in the target image.

[0008] The aforementioned local attention layer performs irrelevant information attenuation on regions outside the facial expression area in the target image, while conversely enhancing the important relevant information in the facial expression area of ​​the target image. This helps improve the accuracy of the expression classification model in classifying based on the emotional features extracted from facial expressions.

[0009] Optionally, the input layer of the VGG16 includes a first convolutional layer, a first batch of normalization layers, a first activation layer, the local attention layer, and a first max pooling layer connected in sequence. The local attention layer performs information attenuation on regions outside the facial expression area in the target image as follows: after receiving the output information of the first activation layer, the local attention layer determines a two-dimensional image mask based on the output information of the first activation layer, and multiplies the two-dimensional image mask with the output information of the first activation layer to obtain the output information of the local attention layer. The output information of the local attention layer is used as input to the network layer connected after the local attention layer for local feature extraction.

[0010] Optionally, determining the two-dimensional image mask based on the output information of the first activation layer includes: calculating the average activation value of the feature map of each channel in the output information of the first activation layer to obtain N average activation values; determining a first channel based on the N average activation values, wherein the first channel is the channel corresponding to the largest average activation value among the N average activation values; setting a mask for each pixel in the first channel, wherein when the first pixel in the first channel is greater than or equal to the largest average activation value, the mask value at the position corresponding to the first pixel value is set to 1; when the first pixel in the first channel is less than the largest average activation value, the mask value at the position corresponding to the first pixel value is set to 0; the first pixel is any pixel in the first channel, and N is a positive integer.

[0011] Optionally, the second feature extraction network is ResNet18.

[0012] Optionally, the fusion classification module includes an orthogonal module and a classification module. The orthogonal module is used to perform orthogonal operation on the first feature and the second feature of the facial expression region using a preset orthogonal function to obtain an orthogonal result. The classification module is used to perform feature aggregation and classification on the orthogonal result using a preset classification function to obtain the classification result.

[0013] Optionally, the target image is a pain expression image.

[0014] In a second aspect, a facial expression classification device is provided, including a module for performing any of the methods in the first aspect.

[0015] Thirdly, an electronic device is provided, including a module for performing any of the methods in the first aspect.

[0016] Fourthly, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, causes the processor to perform the method described in any one of the first aspects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating the implementation steps of the facial expression classification method in an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the facial expression classification model in an embodiment of the present invention;

[0020] Figure 3 This is a partial structural diagram of the residual network in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram illustrating the results of the facial expression classification model classifying the pain level of pain expression images in an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram illustrating the specific process steps of the facial expression classification method in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the facial expression classification device in an embodiment of the present invention;

[0024] Figure 7 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, pixels and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, pixels, components and / or collections thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] In recent years, facial expression classification has become a research hotspot in the field of image processing. For example, in medical research, convolutional neural networks are often used to classify the degree of pain in the facial pain expressions of newborns, critically ill patients, and aphasic patients. However, existing convolutional neural networks lose a lot of key feature information when extracting pain features from facial pain expressions, resulting in very low accuracy in facial pain expression classification.

[0031] This application employs a dual-parallel expression classification model to classify facial expressions. A first feature extraction network extracts the first feature of the facial expression region in the expression image, while a second feature extraction network extracts global features of the expression image to compensate for the first feature extraction network's failure to extract other feature information. This dual-parallel expression classification model improves the extraction rate of emotional features of facial expressions, thereby solving the problem of low accuracy in facial expression classification.

[0032] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0033] To improve the extraction rate of emotional features from facial expressions and thus address the problem of low accuracy in facial expression classification, this application proposes a facial expression classification method, such as... Figure 1 As shown, the method is performed by an electronic device, and the method includes:

[0034] S101, acquire the target image, which includes the facial expression of the target object.

[0035] For example, an electronic device acquires target images (i.e., facial expression images of a person's face), wherein the target object includes newborns, aphasic patients, and ordinary healthy individuals; the target images include: happy expression images, fear expression images, angry expression images, and pain expression images. This application only uses the pain level classification of pain expression images as an example to illustrate the pain level classification method for facial pain expressions. The classification methods for other types of expression images are similar and will not be elaborated here.

[0036] For example, electronic devices can obtain a dataset of facial pain expressions from the UNBC-McMaster Shoulder Pain Expression Archive Database (UNBC). This dataset contains shoulder pain video data from 25 volunteers, totaling 200 video sequences; these 200 video sequences contain a total of 48,198 frames of pain expression images. Furthermore, these 48,198 pain expression images are all stored in PNG format, and each frame has a resolution of approximately 352×240 pixels. In practical applications, each frame of the pain expression image can be cropped to obtain target data with an image dimension of 3*200*200 (i.e., image data with a size of 200*200 and 3 channels).

[0037] The existing pain expression dataset has already categorized the pain level of each frame of the image according to the PSPI standard, and has divided it into 16 levels based on the severity of pain. Higher pain levels indicate greater pain intensity. However, the existing pain expression dataset exhibits an uneven distribution of data across different pain levels. Therefore, based on the existing pain level classification, a new clustering and dimensionality reduction operation is performed on the pain expression data for different pain levels. For example, the existing pain level classifications of 0, 1, 2, and 3 are retained; the original pain level data of 4 and 5 are merged into a new level, i.e., level 4; and the original pain level data of 6 and above are merged into a new level, i.e., level 5. Ultimately, the pain level of the existing pain expression dataset is reclassified into 6 levels.

[0038] Because the number of pain expression images varies across different pain levels after reclassification, some pain levels correspond to a significantly larger number of pain expression images than others. For example, there are 31,200 pain expression images for level 0; 4,000 for level 1; 3,409 for level 2; 1,989 for level 3; 3,600 for level 4; and 4,100 for level 5. Clearly, the number of pain expression images for level 0 is extremely high. In practical applications, 1 / 10 of the data can be randomly extracted from the level 0 pain expression images, i.e., 3,120 (31,200 divided by 10) frames. Alternatively, 1 / 8 or 1 / 11 of the data can be randomly extracted from the extremely large number of pain expression images (e.g., level 0) for practical applications. This application does not impose any limitations on the random extraction ratio; users can choose according to their actual needs.

[0039] In practical applications, since the pain expression image data for the same pain level after reclassification is stored according to the order of different volunteers, to avoid only extracting pain expression images from a subset of volunteers during data extraction, the storage method of the pain expression image data for each pain level is randomly shuffled. Then, according to a certain ratio (e.g., an 8:2 ratio, or other ratios, this application does not impose any limitations on this), the pain expression image data for each pain level is divided into a training dataset and a test dataset. The training dataset and the test dataset are used to train the expression classification model and test the expression classification model, respectively.

[0040] S102, the target image is input into the expression classification model to obtain the classification result; the classification result is used to indicate the degree of emotional expression of facial expressions; the expression classification model includes: a first feature extraction network, a second feature extraction network, and a fusion classification module; the first feature extraction network extracts local features of the target image to obtain the first feature of the area where the facial expression is located; the second feature extraction network extracts global features of the target image to obtain the second feature; the fusion classification module fuses and classifies the first feature and the second feature to obtain the classification result.

[0041] For example, the above classification result refers to the result of the expression classification model classifying the emotional level of the input target image. This classification result can indicate the degree of emotional expression of facial expressions, where emotions include: pain, pleasure, fear, and anger; the above classification result includes: pain level, fear level, anger level, and pleasure level; for example, the pain level of the above pain expression image can be divided into 6 levels, namely level 0, level 1, level 2, level 3, level 4, and level 5; for another example, the expression classification model extracts pain features from the input pain expression image (i.e., the target image) and outputs the pain level classification result; for another example, the expression classification model extracts anger features from the input anger expression image and outputs the anger level classification result. For example, if a pain expression image with an unknown pain level is input into the expression classification model, the expression classification model outputs that the pain level corresponding to the pain expression image is level 1. For another example, if a fear expression image with an unknown fear level is input into the expression classification model, the expression classification model outputs that the fear level corresponding to the fear expression image is level 2.

[0042] For example, such as Figure 2As shown, the above-mentioned expression classification model includes: a first feature extraction network 202, a second feature extraction network 203, and a fusion classification module 204; the electronic device extracts local important features (i.e., emotional features of the face region in the target image) from the area where the facial expression is located in the target image through the first feature extraction network 202 to obtain the first feature of the area where the facial expression is located; at the same time, it extracts supplementary global features from the target image through the second feature extraction network 203 to obtain the second feature; then, the fusion classification module 204 performs feature fusion and classification on the first feature and the second feature of the area where the facial expression is located to obtain the classification result. For example, the pain expression image 201 is input into the first feature extraction network 202, which extracts important feature information of the facial expression region in the pain expression image 201 (i.e., local feature extraction process) to obtain the first feature of the facial pain expression region; at the same time, the pain expression image 201 is input into the second feature extraction network 203, which extracts global feature information of the facial pain expression in the pain expression image 201 (i.e., global feature extraction process) to obtain the second feature; subsequently, the first and second features of the facial pain expression region are fused by the fusion classification module 204, and the pain level is classified based on the feature fusion result to obtain the pain level corresponding to the pain expression image 201 (i.e., classification result).

[0043] For example, the first feature extraction network is VGG16. The input layer of VGG16 includes a local attention layer, which is used to perform information attenuation operations on areas outside the facial expression region in the target image. For instance, the local attention layer attenuates irrelevant information outside the facial pain expression region in the pain expression image 201, which can remove unimportant information unrelated to the facial pain expression in the pain expression image 201, thereby conversely enhancing the important relevant information in the facial pain expression region in the pain expression image 201 (i.e., the target image).

[0044] For example, the input layer of the VGG16 layer includes a first convolutional layer, a first batch of normalization layers, a first activation layer, the aforementioned local attention layer, and a first max pooling layer connected in sequence. The local attention layer performs information attenuation on areas outside the facial expression region in the target image as follows: after receiving the output information from the first activation layer, the local attention layer determines a two-dimensional image mask based on the output information of the first activation layer, and multiplies the two-dimensional image mask with the output information of the first activation layer to obtain the output information of the local attention layer. The output information of the local attention layer is then input to the network layer connected after the local attention layer for local feature extraction. The kernel size of the first convolutional layer is 3×3, the kernel size of the first batch of normalization layers is 64, and the kernel size of the first max pooling layer is 2×2.

[0045] For example, a pain expression image 201 with a size of 200×200 and 3 channels (i.e., 200×200×3) can be input into the first convolutional layer. Of course, pain expression images 201 of other sizes can also be input, and users can choose according to their actual situation. This application does not limit this. The first convolutional layer outputs a first convolution result with a size of 200×200 and 64 channels (i.e., 200×200×64). The first normalization layer normalizes the 200×200×64 first convolution result and outputs a first normalized result of 200×200×64. The first activation layer normalizes the 200×200×64 first convolution result. The first batch of normalized results (200×64) is normalized and output as a first activation result of 200×200×64. The local attention layer receives this first activation result (output information of the first activation layer) and generates a two-dimensional image mask based on it. The two-dimensional image mask is then multiplied by the first activation result of 200×200×64 to obtain the output information of the local attention layer. In other words, the local attention layer uses the two-dimensional image mask to attenuate the area outside the pain expression region in the first activation result of 200×200×64. The output information of the local attention layer is used as input to the network layers connected after it for local feature extraction. The first max-pooling layer performs max-pooling on the output information of the local attention layer and outputs a feature map of dimension 100×100×64.

[0046] For example, determining a two-dimensional image mask based on the output information of the first activation layer includes: calculating the average activation value of the feature map of each channel in the output information of the first activation layer to obtain N average activation values; determining a first channel based on the N average activation values, wherein the first channel is the channel corresponding to the largest average activation value among the N average activation values; setting a mask for each pixel in the first channel, wherein when the first pixel in the first channel is greater than or equal to the largest average activation value, the mask value at the position corresponding to the first pixel value is set to 1; when the first pixel in the first channel is less than the largest average activation value, the mask value at the position corresponding to the first pixel value is set to 0; the first pixel is any pixel in the first channel, and N is a positive integer.

[0047] For example, the local attention layer calculates the average activation value of the feature map of each channel in the 200×200×64 first activation result (i.e., the output information of the first activation layer), obtaining N=64 average activation values. The local attention layer selects the largest average activation value and the channel corresponding to the largest average activation value (i.e., the first channel) from the 64 average activation values. Each channel has 200×200 pixel values. Taking a maximum average activation value of 0.6 and the first pixel being A, B, C, or D as an example, we can illustrate the process of the mask generation module setting the mask for each pixel in the first channel. For example, if pixel A in the first channel is 0.71, pixel B is 0.52, pixel C is 0.64, and pixel D is 0.42, since pixel A in the first channel is 0.71 (greater than 0.6), the mask value at the position corresponding to pixel A is set to 1. The B pixel in the first channel has a value of 0.52 (less than 0.6), so the mask value corresponding to the B pixel is set to 0. The C pixel in the first channel has a value of 0.64 (greater than 0.6), so the mask value corresponding to the C pixel is set to 1. The D pixel in the first channel has a value of 0.42 (less than 0.6), so the mask value corresponding to the D pixel is set to 0. And so on, each pixel in the first channel is masked in this way to generate a 200×200 two-dimensional image mask. The 200×200 two-dimensional image mask includes 200×200 mask values. The 200×200 two-dimensional image mask serves as a reference template (i.e., a reference basis) for the subsequent local attention layer to perform pixel value attenuation operation on the 200×200×64 first activation result based on the 200×200×64 two-dimensional image mask.

[0048] For example, the local attention layer performs pixel value attenuation on the 200×200×64 first activation result based on a 200×200 two-dimensional image mask to obtain a 200×200×64 first feature map. The random pixel value attenuation factor r is set to a specific value between (0, 1) through a random number selection mechanism. If the current mask value at a certain position in the reference two-dimensional image mask is 0, then the pixel value at the corresponding position in the 200×200×64 first activation result is multiplied by r (i.e., the pixel value at the corresponding position is attenuated) to obtain the attenuated 200×200×64 first activation result (i.e., the 200×200×64 second feature map); if the current mask value at a certain position in the reference two-dimensional image mask is 1, then the pixel value at the corresponding position in the 200×200×64 first activation result remains unchanged. Specifically, the local attention layer iterates through each pixel value in the 200×200×64 first activation result using a 2D image mask. This involves performing a decay operation on each pixel value in each of the 64 layers (each layer containing 200×200 pixels) to attenuate less important feature information. This process extracts the important feature information of the 64*200*200 region containing the facial pain expression. Therefore, the local attention layer's pixel value attenuation operation on the 200×200×64 first activation result using a 200×200 2D image mask not only weakens features with low correlation to pain but also conversely enhances features with high correlation to pain (i.e., extracting the important feature information of the region containing the facial pain expression).

[0049] The aforementioned VGG16 also includes a convolution processing module, which comprises two convolutional activation layers and multiple dual convolutional activation layers. The first convolutional activation layer in the two convolutional activation layers is connected to the first dual convolutional activation layer in the multiple dual convolutional activation layers, and the last dual convolutional activation layer in the dual convolutional activation layers is connected to the second convolutional activation layer in the two convolutional activation layers. Each of the two convolutional activation layers comprises a second convolutional layer, a second batch normalization layer, a second ReLU activation layer, and a second max pooling layer, connected sequentially. The kernel size of the second convolutional layer is 3×3, and the kernel size of the second max pooling layer is 2×2. Each of the multiple dual convolutional activation layers comprises a third convolutional layer, a third batch normalization layer, a third ReLU activation layer, a fourth convolutional layer, a fourth batch normalization layer, a fourth ReLU activation layer, and a third max pooling layer, connected sequentially. The kernel size of the third and fourth convolutional layers is 3×3, and the kernel size of the third max pooling layer is 2×2.

[0050] For example, the 100×100×64 feature map output by the first max pooling layer is sequentially processed by the second convolutional layer, the second batch normalization layer (core size 128), the second ReLU activation layer, and the second max pooling layer in the first convolutional activation layer, and finally outputs a feature map with a dimension of 50×50×128.

[0051] For example, the convolutional processing module includes three dual convolutional activation layers. These three dual convolutional activation layers consist of a first dual convolutional activation layer, a second dual convolutional activation layer, and a third dual convolutional activation layer connected in sequence. Specifically, the third convolutional layer in the first dual convolutional activation layer performs convolution processing on the 50×50×128 feature map output from the second max pooling layer, resulting in a 50×50×256 feature map. The third batch normalization layer (with a core size of 256) in the first dual convolutional activation layer performs batch normalization processing on the 50×50×256 feature map output from the third convolutional layer, resulting in a 50×50×256 third batch normalized result. The third ReLU activation layer in the first dual convolutional activation layer activates the input 50×50×256 third batch normalized result, resulting in a 50×50×256 third activation result. The fourth convolutional layer in the first dual convolutional activation layer performs convolution processing on the third activation result of 50×50×256, resulting in a fourth convolutional layer output of 50×50×256. The fourth batch normalization layer (core size 256) in the first dual convolutional activation layer performs batch normalization processing on the output result of the fourth convolutional layer of 50×50×256, resulting in a fourth batch normalized result of 50×50×256. The fourth ReLU activation layer in the first dual convolutional activation layer activates the fourth batch normalized result of 50×50×256, resulting in a fourth activation result of 50×50×256. The third max pooling layer in the first dual convolutional activation layer performs max pooling processing on the fourth activation result of 50×50×256, and outputs a feature map with a dimension of 25×25×256.

[0052] The fourth max pooling layer in the first double convolutional activation layer outputs a 25×25×256 feature map, which then passes through the third convolutional layer in the second double convolutional activation layer, the third batch normalization layer (core size 512) in the second double convolutional activation layer, the third ReLU activation layer in the second double convolutional activation layer, the fourth convolutional layer in the second double convolutional activation layer, the fourth batch normalization layer (core size 512) in the second double convolutional activation layer, the fourth ReLU activation layer in the second double convolutional activation layer, and the third max pooling layer in the second double convolutional activation layer, finally outputting a feature map with a dimension of 12×12×512.

[0053] The fourth max pooling layer in the second double convolutional activation layer outputs a 12×12×512 feature map, which then passes through the third convolutional layer, the third batch normalization layer (core size 512), the third ReLU activation layer, the fourth convolutional layer, the fourth batch normalization layer (core size 512), the fourth ReLU activation layer, and the third max pooling layer in the third double convolutional activation layer, finally outputting a 6×6×512 feature map.

[0054] The 6×6×512 feature map output by the fourth max pooling layer in the third double convolutional activation layer is sequentially passed through the second convolutional layer in the second convolutional activation layer, the second batch normalization layer (core size 512) in the second convolutional activation layer, the second ReLU activation layer in the second convolutional activation layer, and the second max pooling layer in the second convolutional activation layer, finally outputting a feature vector with a dimension of 1×2048 (i.e., the first feature of the region where the pain expression is located).

[0055] For example, such as Figure 2 As shown, the second feature extraction network 203 includes ResNet18, ResNet50, etc. Taking ResNet18 as an example, the second feature extraction network 203 includes an input module, a residual network, and an output module. The input module performs convolution and activation processing on the 200×200×3 pain expression image 201 to obtain the first output information. The residual network performs global feature extraction (supplementary extraction of global features) on the first output information to obtain the second output information. The output module performs average pooling processing on the second output information to obtain the second feature.

[0056] The input module consists of a fifth convolutional layer, a fifth batch normalization layer, a fifth ReLU activation layer, and a fourth max pooling layer connected in sequence. The kernel size of the fifth convolutional layer is 7×7, the kernel size of the fifth batch normalization layer is 64, and the kernel size of the fourth max pooling layer is 3×3. The 200×200×3 pain expression image 201 passes through the fifth convolutional layer (output dimension 100×100×64), the fifth batch normalization layer (output dimension 100×100×64), the fifth ReLU activation layer (output dimension 100×100×64), and the fourth max pooling layer, ultimately outputting a feature map with a dimension of 50×50×64.

[0057] The aforementioned residual network includes a direct mapping subnetwork and multiple residual subnetworks. The direct mapping subnetwork connects to the first residual subnetwork among the multiple residual subnetworks, while the last residual subnetwork among the multiple residual subnetworks connects to the output module. For example, the direct mapping subnetwork performs convolution processing on the 50×50×64 feature map output from the fourth max-pooling layer to obtain the output information of the direct mapping subnetwork, which has a dimension of 50×50×64.

[0058] like Figure 3 As shown in (a), the aforementioned direct mapping subnetwork includes: two cascaded residual modules 301 (i.e., weight layer 301) and 302 (i.e., weight layer 302), and a first direct mapping branch 303. The residual module 301 includes a sixth convolutional layer, a sixth batch normalization layer, a sixth ReLU activation layer, a seventh convolutional layer, and a seventh batch normalization layer connected in sequence. The kernel size of the sixth and seventh convolutional layers is 3×3, and the kernel size of the sixth and seventh batch normalization layers is 64. The residual module 302 includes an eighth convolutional layer, an eighth batch normalization layer, a seventh ReLU activation layer, a ninth convolutional layer, and a ninth batch normalization layer connected in sequence. The kernel size of the eighth and ninth convolutional layers is 3×3, and the kernel size of the eighth and ninth batch normalization layers is 64.

[0059] For example, residual module 301 processes the 50×50×64 feature map output by the fourth max pooling layer through the sixth convolutional layer, the sixth batch normalization layer, the sixth ReLU activation layer, the seventh convolutional layer, and the seventh batch normalization layer for convolution and normalization, resulting in the output information of residual module 301, which has a dimension of 50×50×64. Residual module 302 processes the 50×50×64 output information of residual module 301 through the eighth convolutional layer, the eighth batch normalization layer, the seventh ReLU activation layer, the ninth convolutional layer, and the ninth batch normalization layer for convolution and normalization, resulting in the output information of residual module 302, which also has a dimension of 50×50×64. The first direct mapping branch 303 directly maps the 50×50×64 feature map output by the fourth max pooling layer to obtain a first mapping result, which is still a 50×50×64 feature map output by the fourth max pooling layer. The output information of the residual module 302 is concatenated with the first mapping result to obtain a first concatenated result with a dimension of 50×50×64.

[0060] like Figure 3As shown in (b), each residual subnetwork in the above-mentioned plurality of residual subnetworks includes a residual module 304 (i.e., weight layer 304), a residual module 305 (i.e., weight layer 305), and a residual branch 306 connected in sequence. The residual module 304 includes a tenth convolutional layer, a tenth batch normalization layer, a seventh ReLU activation layer, an eleventh convolutional layer, and an eleventh batch normalization layer connected in sequence. The kernel size of the tenth and eleventh convolutional layers is 3×3. The kernel size of the tenth and eleventh batch normalization layers is also 3×3. The kernel size of each batch of normalized layers is 64; the residual module 305 includes a twelfth convolutional layer, a twelfth batch of normalized layers, an eighth ReLU activation layer, a thirteenth convolutional layer, and a thirteenth batch of normalized layers connected in sequence. The kernel size of the twelfth and thirteenth convolutional layers is 3×3, and the kernel size of the twelfth and thirteenth batch of normalized layers is 64; the residual branch 306 includes a fourteenth convolutional layer (with a kernel size of 1×1) and a fourteenth batch of normalized layers connected in sequence.

[0061] Taking the above residual network, which includes three residual subnetworks, as an example, the first residual subnetwork processes the 50×50×64 first splicing result as follows: the 50×50×64 first splicing result passes sequentially through residual module 304 and residual module 305 in the first residual subnetwork to obtain 25×25×128 first residual information; simultaneously, the 50×50×64 first splicing result is sampled (i.e., dimensionality upscaling) through residual branch 306 in the first residual subnetwork (the core size of the fourteenth batch normalization layer in residual branch 306 is 128) to obtain 25×25×128 first sampling information. This sampling process includes at least one of upsampling and downsampling, selected according to the actual situation; the 25×25×128 first residual information and the 25×25×128 first sampling information are spliced ​​to obtain a 25×25×128 second splicing result.

[0062] The residual module 304 in the first residual sub-network performs convolution and normalization processing on the 50×50×64 first concatenation result through the tenth convolutional layer, the tenth batch normalization layer (core size 128), the seventh ReLU activation layer, the eleventh convolutional layer, and the eleventh batch normalization layer (core size 128) to obtain the output information of the residual module 304, which has a dimension of 25×25×128. The residual module 305 in the first residual sub-network performs convolution and normalization processing on the output information of the residual module 304 through the twelfth convolutional layer, the twelfth batch normalization layer (core size 128), the eighth ReLU activation layer, the thirteenth convolutional layer, and the thirteenth batch normalization layer (core size 128) to obtain the output information of the residual module 305 (i.e., the first residual information of 25×25×128), which has a dimension of 25×25×128.

[0063] The second residual subnetwork in the three residual subnetworks processes the second splicing result of 25×25×128 as follows: the second splicing result of 25×25×128 passes sequentially through the residual module 304 and the residual module 305 in the second residual subnetwork to obtain the second residual information of 13×13×256; at the same time, the second splicing result of 25×25×128 is sampled (i.e., dimensionality increase) through the residual branch 306 in the second residual subnetwork (the core size of the fourteenth batch normalization layer in the residual branch 306 is 256) to obtain the second sampling information of 13×13×256. This sampling process includes at least one of upsampling and downsampling, selected according to the actual situation; the second residual information of 13×13×256 and the second sampling information of 13×13×256 are spliced ​​to obtain the third splicing result of 13×13×256.

[0064] The residual module 304 in the second residual sub-network performs convolution and normalization processing on the 25×25×128 second concatenation result through the tenth convolutional layer, the tenth batch normalization layer (core size 256), the seventh ReLU activation layer, the eleventh convolutional layer, and the eleventh batch normalization layer (core size 256) to obtain the output information of the residual module 304, which has a dimension of 13×13×256. The residual module 305 in the second residual sub-network performs convolution and normalization processing on the output information of the residual module 304 through the twelfth convolutional layer, the twelfth batch normalization layer (core size 256), the eighth ReLU activation layer, the thirteenth convolutional layer, and the thirteenth batch normalization layer (core size 256) to obtain the output information of the residual module 305 (i.e., the second residual information of 13×13×256), which has a dimension of 13×13×256.

[0065] The third residual subnetwork in the three residual subnetworks processes the third splicing result of 13×13×256 as follows: the third splicing result of 13×13×256 passes through the residual module 304 and the residual module 305 in the third residual subnetwork in sequence to obtain the third residual information of 7×7×512; at the same time, the third splicing result of 13×13×256 is sampled (i.e., dimensionality increase operation) through the residual branch 306 in the third residual subnetwork (the core size of the fourteenth batch normalization layer in the residual branch 306 is 512) to obtain the third sampling information of 7×7×512. This sampling process includes at least one of upsampling and downsampling, which is selected according to the actual situation; the third residual information of 7×7×512 and the third sampling information of 7×7×512 are spliced ​​to obtain the fourth splicing result of 7×7×7.

[0066] The residual module 304 in the aforementioned third residual sub-network sequentially processes the 13×13×256 third concatenation result through the tenth convolutional layer, the tenth batch normalization layer (core size 512), the seventh ReLU activation layer, the eleventh convolutional layer, and the eleventh batch normalization layer (core size 512) to obtain the output information of the residual module 304, which has a dimension of 7×7×512. The residual module 305 in the aforementioned third residual sub-network sequentially processes the output information of the residual module 304 through the twelfth convolutional layer, the twelfth batch normalization layer (core size 512), the eighth ReLU activation layer, the thirteenth convolutional layer, and the thirteenth batch normalization layer (core size 512) to obtain the output information of the residual module 305 (i.e., the 7×7×512 third residual information), which also has a dimension of 7×7×512.

[0067] The above output module includes an average pooling layer with a size of 3×3. The average pooling layer performs an average pooling operation on the fourth concatenation result of 7×7×7 to obtain a feature vector (i.e., the second feature) with a dimension of 1×1×512.

[0068] For example, the fusion classification module 204 includes an orthogonal module 2041 and a classification module 2042. The orthogonal module 2041 is used to perform orthogonal operation on the first feature and the second feature of the facial expression region using a preset orthogonal function to obtain an orthogonal result. The classification module 2042 is used to perform feature aggregation and classification on the orthogonal result using a preset classification function to obtain a classification result. The orthogonal module 2041 includes the Bilinear function provided in the PyTorch deep learning library. The classification module 2042 includes the Linear function and the Softmax classification function provided in the PyTorch deep learning library, which are connected in sequence. For example, the Bilinear function performs orthogonal operation on the feature vector with an output dimension of 1×2048 (i.e., the first feature of the pain expression region) and the feature vector with a dimension of 1×1×512 (i.e., the second feature) to obtain an orthogonal result; thereby realizing the fusion operation of the first feature and the second feature of the pain expression region. The Linear function performs feature aggregation (i.e., dimensionality reduction) on the orthogonal results, resulting in an aggregated result with an output dimension of 6; the Softmax classification function classifies this aggregated result, and finally obtains the classification result (for example, the pain level corresponding to the 200×200×3 pain expression image 201 is level 3).

[0069] For example, when training the above-mentioned facial expression classification model, the cross-entropy function provided by the PyTorch deep learning library can be selected as the loss function, and stochastic gradient descent can be selected as the training optimizer. The loss after Softmax classification is calculated stepwise through forward propagation, and the weights of the facial expression classification model are updated based on the calculated loss value through backpropagation until the facial expression classification model tends to converge. At this point, the training of the facial expression classification model can be stopped, and the weights of the facial expression classification model can be saved.

[0070] For example, the accuracy of the facial expression classification model can be evaluated using the following metrics: accuracy (Acc), root mean square error (RMSE), and Pearson correlation coefficient (PCC). Other evaluation metrics can also be used, and this application does not limit this. Here, Acc represents the prediction accuracy of the facial expression classification model, i.e., the proportion of predicted label values ​​for facial expression images that match the actual true label values. RMSE is calculated as the error between the predicted and true values, i.e., the error between the predicted and true label values ​​for facial expression images. The PCC coefficient reflects the performance of the facial expression classification model in predicting facial expression images at different time dimensions. For example, Acc can be used to indicate the proportion of predicted label values ​​for pain facial expression images that match the actual true label values. RMSE can be used to indicate the error between the predicted and true values ​​for pain facial expression images. The PCC coefficient reflects the performance of the facial expression classification model in predicting pain facial expression images at different time dimensions.

[0071] For example, a sequence of 80 consecutive and temporally adjacent images of facial expressions of pain is selected and sequentially input into an expression classification model for classification testing; the classification test results are as follows: Figure 4 As shown, the actual curve 401 represents the true value, the predicted value distribution 402 is represented by a black plus sign, the horizontal axis represents the image frame, and the vertical axis represents the pain level; Figure 4 It can be seen that after classifying the pain level of 80 frames of facial pain expression images by facial pain classification model, only 6 frames were incorrectly classified as having the pain level (e.g., ...). Figure 4 (The black '+' is not on the solid black line). Therefore, it can be seen that the dual-parallel facial expression classification model proposed in this invention can effectively classify the degree of pain in pain expression images.

[0072] The formulas for calculating Acc, RMSE, and PCC are as follows:

[0073]

[0074]

[0075]

[0076] Among them, M i y represents the number of correctly classified samples in each classification result of the experiment using the facial expression classification model. N represents the total number of facial expression images in the experiment, and y represents the number of correctly classified samples in each classification result of the experiment. i and These represent the true label value and the predicted label value of an expression image (e.g., an image of a pain expression), respectively. and Representing sequences respectively and {y1,y2,...,y N The average value of}.

[0077] For example, the facial expression classification model provided in this application was trained and validated on publicly available datasets, and its performance (i.e., classification accuracy) was quantitatively evaluated based on three evaluation metrics: accuracy (Acc), root mean square error (RMSE), and Pearson correlation coefficient (PCC). The accuracy (Acc) was 92.11%, the RMSE was 0.48, and the PCC was 0.95. The performance evaluation results of the facial expression classification model provided in this application are very close to those of existing advanced experimental methods (e.g., a novel 3D deep network model SCN, which uses multiple convolutional layers of different temporal depths to capture the extensive spatiotemporal variations of facial expressions, ultimately achieving a RMSE of 0.57 and a PCC of 0.92), demonstrating the effectiveness of the facial expression classification model provided in this application.

[0078] To facilitate understanding, the following will be combined with... Figure 5 The overall process of the facial expression classification method provided in this application is illustrated by way of example. Taking the classification of pain levels in facial pain expressions as an example, the steps of the facial expression classification method are explained as follows:

[0079] Preparation Phase 501: Prepare a dataset of facial pain expression images with pain level labels. For example, the pain expression images have been divided into 6 pain levels according to the degree of pain.

[0080] Modeling Phase 502: Construct a model for pain expression classification based on a dual-parallel expression classification model that combines local attention mechanisms. For example, the input layer of VGG16 is fused with a local attention layer and ResNet18 to form a dual-parallel expression classification model. This dual-parallel expression classification model can be used to classify the degree of pain in pain expression images.

[0081] Training Phase 503: Create a training dataset based on the prepared pain expression image data, and use the training dataset to iteratively train the constructed expression classification model. For example, divide the pain expression image dataset into a training dataset and a test dataset. The training dataset is used to iteratively train the constructed expression classification model to obtain an expression classification model that meets the requirements.

[0082] Classification Phase 504: Use the trained facial expression classification model to classify the pain level of facial pain expressions in the test set. For example, use the trained facial expression classification model to classify the pain level of pain expression images in the test set to obtain the final classification result (i.e., the specific pain level corresponding to the pain expression image).

[0083] Compared to existing convolutional neural network methods that only extract local features from facial expressions of target objects, this application employs a parallel first feature extraction network and a second feature extraction network to form an expression classification model that extracts both local and global features from facial expressions of target objects. The second feature extraction network extracts global features of facial expressions to compensate for important feature information missed by the first feature extraction network when extracting local features of facial expressions, thereby improving the extraction rate of emotional features of facial expressions and thus improving the accuracy of classifying the degree of emotional expression based on the feature extraction results of facial expressions.

[0084] The local attention layer of this application performs irrelevant information attenuation operation on areas outside the facial expression region in the target image, while conversely enhancing the important relevant information in the facial expression region of the target image, thereby improving the accuracy of the expression classification model in classifying based on the emotional features extracted from facial expressions.

[0085] Figure 6 This is a schematic diagram of a facial expression classification device provided in this application. The classification device 600 includes an acquisition module 601 and a processing module 602.

[0086] The acquisition module 601 is used to: acquire a target image, which includes the facial expression of the target object;

[0087] The processing module 602 is used to: input the target image into the expression classification model to obtain the classification result, which is used to indicate the degree of emotional expression of facial expressions;

[0088] The facial expression classification model includes: a first feature extraction network, a second feature extraction network, and a fusion classification module; the first feature extraction network extracts local features from the target image to obtain the first feature of the facial expression region; the second feature extraction network extracts global features from the target image to obtain the second feature; and the fusion classification module fuses and classifies the first and second features to obtain the classification result.

[0089] The specific method by which the classification device 600 performs the facial expression classification method and the beneficial effects thereof can be found in the relevant description in the method embodiments.

[0090] Figure 7A schematic diagram of the structure of an electronic device provided in this application is shown. Figure 7 The dashed lines indicate that the unit or module is optional. The electronic device 700 can be used to implement the methods described in the above method embodiments. The electronic device 700 can be a terminal device, a server, or a chip.

[0091] Electronic device 700 includes one or more processors 701, which enable electronic device 700 to implement Figure 1 The method described in the corresponding method embodiment. Processor 701 can be a general-purpose processor or a dedicated processor. For example, processor 701 can be a central processing unit (CPU). The CPU can be used to control the electronic device 700, execute software programs, and process data from the software programs. The electronic device 700 may also include a communication unit 705 for implementing signal input (reception) and output (transmission).

[0092] For example, electronic device 700 can be a chip, communication unit 705 can be the input and / or output circuit of the chip, or communication unit 705 can be the communication interface of the chip, and the chip can be a component of terminal device.

[0093] For example, electronic device 700 can be a terminal device, communication unit 705 can be the transceiver of the terminal device, or communication unit 705 can be the transceiver circuit of the terminal device.

[0094] The electronic device 700 may include one or more memories 702, on which a program 704 is stored. The program 704 can be executed by a processor 701 to generate instructions 703, causing the processor 701 to execute the method described in the above method embodiments according to the instructions 703. Optionally, the memory 702 may also store data. Optionally, the processor 701 may also read data stored in the memory 702, which may be stored at the same memory address as the program 704, or it may be stored at a different memory address than the program 704.

[0095] The processor 701 and memory 702 can be configured separately or integrated together, for example, integrated on the system-on-chip (SOC) of the terminal device.

[0096] The specific manner in which the processor 701 executes the facial expression classification method can be found in the relevant description in the method embodiment.

[0097] It should be understood that the steps of the above method embodiments can be implemented by hardware logic circuits or software instructions in the processor 701. The processor 701 can be a CPU, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0098] This application also provides a computer program product that, when executed by processor 701, implements the methods described in any of the method embodiments of this application.

[0099] The computer program product can be stored in memory 702, for example, program 704. Program 704 is finally converted into an executable object file that can be executed by processor 701 after processing such as preprocessing, compilation, assembly and linking.

[0100] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0101] The computer-readable storage medium is, for example, memory 702. Memory 702 can be volatile memory or non-volatile memory, or memory 702 can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0103] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0104] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for classifying facial expressions, characterized in that, The method includes: Acquire a target image, the target image including the facial expression of the target object; The target image is input into an expression classification model to obtain a classification result, which is used to indicate the degree of emotional expression of the facial expression. The facial expression classification model includes: a first feature extraction network, a second feature extraction network, and a fusion classification module. The first feature extraction network extracts local features from the target image to obtain a first feature of the facial expression region. The second feature extraction network extracts global features from the target image to obtain a second feature. The fusion classification module fuses and classifies the first and second features to obtain the classification result. The fusion classification module includes an orthogonal module and a classification module. The orthogonal module uses a preset orthogonal function to perform an orthogonal operation on the first and second features of the facial expression region to obtain an orthogonal result. The classification module uses a preset classification function to perform feature aggregation and classification on the orthogonal result to obtain the classification result.

2. The classification method according to claim 1, characterized in that, The first feature extraction network is VGG16, and the input layer of the VGG16 includes a local attention layer. The local attention layer is used to perform information attenuation operations on areas outside the facial expression region in the target image.

3. The classification method according to claim 2, characterized in that, The input layer of the VGG16 includes a first convolutional layer, a first batch of normalization layers, a first activation layer, the local attention layer, and a first max pooling layer connected in sequence. The local attention layer performs information attenuation on areas outside the facial expression region in the target image as follows: after receiving the output information of the first activation layer, the local attention layer determines a two-dimensional image mask based on the output information of the first activation layer, and multiplies the two-dimensional image mask with the output information of the first activation layer to obtain the output information of the local attention layer; wherein, the output information of the local attention layer is used to input to the network layer connected after the local attention layer for local feature extraction.

4. The classification method according to claim 3, characterized in that, The step of determining the two-dimensional image mask based on the output information of the first activation layer includes: The average activation value is calculated for the feature map of each channel in the output information of the first activation layer to obtain N average activation values; The first channel is determined based on the N average activation values, and the first channel is the channel corresponding to the largest average activation value among the N average activation values; A mask is set for each pixel in the first channel, wherein when the first pixel in the first channel is greater than or equal to the maximum average activation value, the mask value at the position corresponding to the first pixel value is set to 1; when the first pixel in the first channel is less than the maximum average activation value, the mask value at the position corresponding to the first pixel value is set to 0; the first pixel is any pixel in the first channel, and N is a positive integer.

5. The classification method according to any one of claims 1 to 4, characterized in that, The second feature extraction network is ResNet18.

6. The classification method according to any one of claims 1 to 4, characterized in that, The classification result is the degree of pain.

7. A facial expression classification device, characterized in that, Includes an acquisition module and a processing module. The acquisition module is used to: acquire a target image, the target image including the facial expression of the target object; The processing module is used to: input the target image into the expression classification model to obtain a classification result, the classification result being used to indicate the degree of emotional expression of the facial expression; The facial expression classification model includes: a first feature extraction network, a second feature extraction network, and a fusion classification module. The first feature extraction network extracts local features from the target image to obtain a first feature of the facial expression region. The second feature extraction network extracts global features from the target image to obtain a second feature. The fusion classification module fuses and classifies the first and second features to obtain the classification result. The fusion classification module includes an orthogonal module and a classification module. The orthogonal module uses a preset orthogonal function to perform an orthogonal operation on the first and second features of the facial expression region to obtain an orthogonal result. The classification module uses a preset classification function to perform feature aggregation and classification on the orthogonal result to obtain the classification result.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program from the memory, causing the electronic device to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 6.

Images