Expression recognition method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides an expression recognition method and device, equipment, a medium and a program product. The method comprises the steps of obtaining a DWT low-frequency sub-band graph and a CTP feature graph of a to-be-recognized image, inputting the DWT low-frequency sub-band graph and the CTP feature graph into an expression recognition model, and obtaining an expression output by the expression recognition model. Wherein the to-be-recognized image comprises a human face, and the expression recognition model is obtained by training according to a sample DWT low-frequency sub-band graph and a sample CTP feature graph of the sample image in advance. The method is used for achieving the effect of improving the face recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an expression recognition method, apparatus, device, medium, and program product. Background Technology

[0002] Because facial expressions are instantaneous, non-invasive, and have high information density, they can capture the most authentic human emotional responses in real time. Therefore, facial expression recognition has an irreplaceable advantage in scenarios where real emotional feedback is required.

[0003] In practical applications, considering the difficulty in obtaining a large amount of high-quality facial expression annotation data required for facial expression recognition through deep neural networks, and the potential for facial shadows or overexposure due to varying lighting conditions, which can affect the extraction of key facial expression features, this paper proposes generating "virtual samples" from given original training samples to increase the number of training samples. Then, based on these virtual samples, three nonlinear feature extraction methods—kernel principal component analysis, kernel discriminant analysis, and kernel local preserving projection—are used to extract features from the samples. This results in three feature patterns, for which a sparse representation model is constructed. A total of three sparse representation models are built for each sample, and finally, classification is performed based on the representation results.

[0004] However, this method still suffers from low accuracy in facial expression recognition. Summary of the Invention

[0005] This application provides methods, apparatus, devices, media, and program products for facial expression recognition, in order to achieve the technical effect of improving the accuracy of facial expression recognition.

[0006] In a first aspect, embodiments of this application provide an expression recognition method, including:

[0007] Obtain the DWT low-frequency subband map and CTP feature map of the image to be identified, wherein the image to be identified contains a human face;

[0008] The DWT low-frequency subband map and the CTP feature map are input into the expression recognition model to obtain the expression output by the expression recognition model. The expression recognition model is trained in advance based on the sample DWT low-frequency subband map and sample CTP feature map of the sample image.

[0009] In one possible implementation, obtaining the CTP feature map of the image to be identified includes:

[0010] For each first pixel in the image to be identified, a pixel region of a preset window size centered on the first pixel is determined;

[0011] Within the pixel area, construct the largest possible circle with the first pixel as its center;

[0012] Determine the second pixel point where the radius and side length of the circle are located;

[0013] Determine the number of the second pixels and the pixel value of each second pixel;

[0014] The average pixel value of the second pixel is determined based on the number of the second pixel and the pixel value of each second pixel.

[0015] The CTP value of the first pixel is determined based on the pixel value of the first pixel, the radius, the average pixel value, the number of second pixels, and the pixel value of each second pixel.

[0016] Based on the CTP values ​​of all first pixels, construct the CTP feature map of the image to be identified.

[0017] In one possible implementation, determining the CTP value of the first pixel based on the pixel value of the first pixel, the radius, the average pixel value, the number of second pixels, and the pixel value of each second pixel includes:

[0018] The CTP value of the first pixel is calculated using formula (1):

[0019]

[0020] Wherein, CTP(n,R,g) c ) represents the CTP value of the first pixel, n represents the number of the second pixels, R represents the radius, and g represents the value of the second pixel. c The pixel value of the first pixel, g d Let d be the pixel value of the second pixel. For the first The pixel value of the second pixel; s(x) is a three-valued function, expressed by formula (2):

[0021]

[0022] Where t is the average pixel value.

[0023] In one possible implementation, the facial expression recognition model includes smCNN, which comprises an input layer, a first sampling layer, a multi-scale convolutional layer, a second sampling layer, a first fully connected layer, and a second fully connected layer arranged in sequence.

[0024] In one possible implementation, the method further includes:

[0025] Obtain the Haar feature map of the image to be identified;

[0026] The step of inputting the DWT low-frequency subband map and the CTP feature map into the expression recognition model to obtain the expression output by the expression recognition model includes:

[0027] The DWT low-frequency subband map, the Haar feature map, and the CTP feature map are input into the expression recognition model to obtain the expression output by the expression recognition model.

[0028] In one possible implementation, acquiring the DWT low-frequency subband map and CTP feature map of the image to be identified includes:

[0029] Obtain the grayscale image of the image to be identified;

[0030] Obtain the DWT low-frequency subband map and CTP feature map of the grayscale image;

[0031] Accordingly, obtaining the Haar feature map of the image to be identified includes:

[0032] Obtain the Haar feature map of the grayscale image.

[0033] Secondly, embodiments of this application provide an expression recognition device, comprising:

[0034] The acquisition module is used to acquire the DWT low-frequency subband map and CTP feature map of the image to be identified, wherein the image to be identified contains a human face;

[0035] The input module is used to input the DWT low-frequency subband map and the CTP feature map into the expression recognition model to obtain the expression output by the expression recognition model. The expression recognition model is trained in advance based on the sample DWT low-frequency subband map and sample CTP feature map of the sample image.

[0036] In one possible implementation, the acquisition module is specifically used for:

[0037] For each first pixel in the image to be identified, a pixel region of a preset window size centered on the first pixel is determined;

[0038] Within the pixel area, construct the largest possible circle with the first pixel as its center;

[0039] Determine the second pixel point where the radius and side length of the circle are located;

[0040] Determine the number of the second pixels and the pixel value of each second pixel;

[0041] The average pixel value of the second pixel is determined based on the number of the second pixel and the pixel value of each second pixel.

[0042] The CTP value of the first pixel is determined based on the pixel value of the first pixel, the radius, the average pixel value, the number of second pixels, and the pixel value of each second pixel.

[0043] Based on the CTP values ​​of all first pixels, construct the CTP feature map of the image to be identified.

[0044] In one possible implementation, the acquisition module is specifically used for:

[0045] The CTP value of the first pixel is calculated using formula (1):

[0046]

[0047] Wherein, CTP(n,R,g) c ) represents the CTP value of the first pixel, n represents the number of the second pixels, R represents the radius, and g represents the value of the second pixel. c The pixel value of the first pixel, g d Let d be the pixel value of the second pixel. For the first The pixel value of the second pixel; s(x) is a three-valued function, expressed by formula (2):

[0048]

[0049] Where t is the average pixel value.

[0050] In one possible implementation, the facial expression recognition model includes smCNN, which comprises an input layer, a first sampling layer, a multi-scale convolutional layer, a second sampling layer, a first fully connected layer, and a second fully connected layer arranged in sequence.

[0051] In one possible implementation, the acquisition module is further configured to:

[0052] Obtain the Haar feature map of the image to be identified;

[0053] The input module is specifically used for:

[0054] The DWT low-frequency subband map, the Haar feature map, and the CTP feature map are input into the expression recognition model to obtain the expression output by the expression recognition model.

[0055] In one possible implementation, the acquisition module specifically includes:

[0056] Obtain the grayscale image of the image to be identified;

[0057] Obtain the DWT low-frequency subband map and CTP feature map of the grayscale image;

[0058] Obtain the Haar feature map of the grayscale image.

[0059] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0060] The memory stores computer-executed instructions;

[0061] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0062] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0063] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0064] The facial expression recognition method, apparatus, device, medium, and program products provided in this application acquire the DWT low-frequency subband map and CTP feature map of the image to be recognized, and input the DWT low-frequency subband map and CTP feature map into the facial expression recognition model to obtain the facial expression output by the model. The image to be recognized contains a human face, and the facial expression recognition model is pre-trained based on sample DWT low-frequency subband maps and sample CTP feature maps of sample images. In this technical solution, the DWT low-frequency subband map preserves the global structural features of the face, while the CTP feature map depicts stable texture details in local areas, ensuring the richness and completeness of the features. These two complementary deep features are jointly input into the facial expression recognition model for joint training and decision-making, thereby improving the accuracy of facial expression recognition using the model. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0066] Figure 1 A schematic diagram illustrating a scenario for the facial expression recognition method provided in an embodiment of this application;

[0067] Figure 2A flowchart illustrating the facial expression recognition method provided in this application embodiment;

[0068] Figure 3 A schematic diagram illustrating the calculation principle of the CTP feature map provided in this application embodiment;

[0069] Figure 4 This is a schematic diagram of the structure of smCNN provided in an embodiment of this application;

[0070] Figure 5 A schematic diagram illustrating the calculation principle of the Haar feature integral map provided in this application embodiment;

[0071] Figure 6 A flowchart illustrating the model training process of the facial expression recognition method provided in this application embodiment;

[0072] Figure 7 A schematic diagram of the facial expression recognition device provided in this application;

[0073] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0074] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0075] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0076] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0077] First, the relevant terms used in this application will be explained:

[0078] A Convolutional Neural Network (CNN) is a type of feedforward neural network and one of the most representative network architectures in deep learning. Its artificial neurons respond to surrounding units within a certain coverage area, processing image features. Generally, the basic structure of a CNN consists of two layers: a feature extraction layer, where the input of each neuron is connected to the local receptive field of the previous layer, extracting features from that local receptive field; and a feature mapping layer, where each feature mapping layer consists of multiple feature maps, each representing a plane. The feature mapping layer is equipped with an activation function, typically a non-linear activation function such as the sigmoid function or the ReLU function. It's important to note that a CNN is composed of a large number of interconnected nodes (also called "neurons" or "units"), each node representing a specific output function. The connection between any two nodes represents parameter values ​​called weights. Different weights and activation functions result in different outputs from the CNN. One of the advantages of convolutional neural networks (CNNs) over traditional image processing algorithms is that they avoid complex preprocessing steps (such as extracting artificial features) and can directly input the original image for end-to-end learning. Another advantage of CNNs over traditional neural networks is that traditional neural networks use fully connected layers, meaning all neurons from the input layer to the hidden layers are connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNNs, however, avoid this problem through local connections and parameter sharing.

[0079] Single-level multi-scale convolutional neural networks (smCNN) are an improved architecture that integrates multi-scale feature extraction within a single network layer. The core idea is to capture multi-scale spatial information of the input data directly at the same layer through parallel or cascaded convolutional kernel designs (such as kernels of different sizes or dilated convolutions), avoiding the loss of detail caused by the layer-by-layer abstraction of traditional multi-level networks. For example, in image processing, smCNN may simultaneously use 3×3, 5×5, and 7×7 convolutional kernels to process the same input region, extracting local details, intermediate structure, and global contextual features respectively, and then generating a comprehensive representation through feature fusion (such as concatenation or weighted summation). This design is particularly suitable for scenarios that require simultaneous processing of fine-grained and coarse-grained features (such as medical image segmentation or small object detection), reducing redundant computation in deep networks and mitigating the gradient vanishing problem, balancing model efficiency and accuracy. Its parameter sharing mechanism still follows traditional CNNs, but the flexible combination of multi-scale kernels further enhances the model's scale invariance.

[0080] First, the application background of this application will be explained:

[0081] Facial expression recognition, as a core technology of affective computing and interaction, is important because of the rich emotional information contained in human facial expressions. Compared with traditional methods of emotion assessment such as questionnaires or verbal expressions, facial expressions are characterized by immediacy, non-invasiveness, and high information density, enabling them to capture the most authentic emotional responses in real time. Since facial expression changes are autonomous physiological responses directly controlled by the limbic system of the brain, they more accurately reflect inner states than verbal descriptions, giving facial expression recognition an irreplaceable advantage in scenarios requiring genuine emotional feedback.

[0082] However, facial expression recognition also faces numerous technical challenges in practical applications. First, current mainstream expression recognition algorithms rely on deep neural networks, such as CNNs or Transformer architectures. While these models can learn complex facial features from large-scale data, they also lead to high computational resource consumption and demanding training data requirements. Due to the difficulty in obtaining high-quality labeled facial data, especially the scarcity of natural facial expression data in real-world scenarios, many algorithms perform well in laboratory environments but may experience a significant drop in recognition accuracy during actual deployment. Second, differences in lighting conditions can cause facial shadows or overexposure, affecting the extraction of key facial features; changes in head posture can cause facial distortion, making it difficult to distinguish subtle expressions such as anger and disgust; and occlusions (such as masks and glasses) can directly obstruct the observation of parts of the face, preventing the algorithm from acquiring complete facial information and compromising the accuracy of expression recognition.

[0083] To address the aforementioned technical issues, the main approach involves generating "virtual samples" from the given original training samples to increase the number of training samples. Then, based on the generated virtual samples, three nonlinear feature extraction methods—kernel principal component analysis, kernel discriminant analysis, and kernel local preserving projection algorithm—are used to extract features from the samples. This results in three types of feature patterns, and a sparse representation model is constructed for each feature pattern. A total of three sparse representation models are constructed for each sample, and finally, the classification is performed based on the representation results.

[0084] However, existing technologies rely solely on error metrics between models for classification decisions, excessively depending on pixel-level or shallow feature reconstruction errors. The core of face recognition lies in capturing deep semantic features related to identity (such as facial topological relationships and micro-expressions), and error metrics cannot effectively distinguish between inter-class differences and intra-class variations (such as interference factors like lighting changes and occlusion). Under small sample conditions, sparse training data leads to blurred class boundaries in the error space. Furthermore, the feature space generated by traditional dimensionality reduction methods may deviate from the optimal direction for identity discrimination, ultimately making it difficult for classifiers based on minimum error decisions to establish highly discriminative feature representations.

[0085] In other words, existing technologies have relatively low accuracy in facial recognition.

[0086] Based on this, the technical concept of this application is as follows: Existing technologies rely excessively on pixel-level reconstruction errors for decision-making, making it difficult to capture the essential discriminative features of a face, namely, deep semantic features unaffected by illumination or occlusion. Considering that Discrete Wavelet Transform (DWT) low-frequency sub-band maps can preserve global facial structural features (such as contours and relative organ positions) and filter high-frequency noise interference, while Circular Texture Pattern (CTP) feature maps can depict stable texture details in local areas (such as skin folds and facial muscle movement patterns), if we directly extract and fuse these two robust facial representations, and input these two complementary deep features into the expression recognition model for joint training and decision-making, the model can learn identity semantic representations that are illumination-invariant and occlusion-robust. This can solve the problems of insufficient discriminative power in the feature space and blurred error boundaries caused by small sample sizes in traditional methods, thereby improving the accuracy of recognition.

[0087] For example, Figure 1 This is a schematic diagram of a scenario for the facial expression recognition method provided in the embodiments of this application, such as... Figure 1 As shown, the scenario includes terminal devices and servers, and the terminal devices and servers can communicate with each other via wired or wireless methods.

[0088] The terminal devices can be laptops, desktop computers, and tablets, etc.

[0089] The terminal device includes a video capture unit, such as the front-facing camera of a mobile phone, to capture video of the user's face. It also includes an image noise reduction unit to process each frame of the video captured by the video capture unit, thereby improving image quality.

[0090] Furthermore, the server is equipped with an image feature extractor to extract the DWT low-frequency subband map, CTP feature map, and Haar feature map of the denoised image sent by the terminal device. The server is also equipped with a feature fusion unit to fuse the DWT low-frequency subband map, CTP feature map, and Haar feature map. The server is also equipped with a classifier to determine the facial expression in the image based on the fused feature map.

[0091] It should be understood that feature fusionists and classifiers can be integrated into facial expression recognition models.

[0092] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0093] Figure 2 This is a flowchart illustrating the facial expression recognition method provided in an embodiment of this application, as shown below. Figure 2 As shown, this method can be implemented through the following steps:

[0094] S21. Obtain the DWT low-frequency subband map and CTP feature map of the image to be identified.

[0095] The image to be identified contains a human face.

[0096] Furthermore, the low-frequency subband map of DWT and the feature map of CTP will be introduced separately.

[0097] 1. DWT low-frequency subband diagram:

[0098] The image to be identified is transformed by a DWT algorithm layer and decomposed into four sub-band images, each with a size of 1 / 4 of the original size (A×A), i.e., A / 2×A / 2 pixels. These are the DWT low-frequency sub-band image LL, the horizontal high-frequency sub-band image LH, the vertical high-frequency sub-band image HL, and the diagonal high-frequency sub-band image HH.

[0099] 2. CTP Feature Map

[0100] For each first pixel in the image to be recognized, a pixel region of a preset window size centered on the first pixel is determined. Then, within this pixel region, a circle with the largest possible area is constructed, centered on the first pixel. Further, the radius of the circle and the second pixel containing its side length are determined. Next, the number of second pixels and the pixel value of each second pixel are determined. Then, based on the number of second pixels and their pixel values, the average pixel value of each second pixel is determined. Finally, based on the pixel values ​​of the first pixels, the radius, the average pixel value, the number of second pixels, and their pixel values, the CTP value of each first pixel is determined, and a CTP feature map of the image to be recognized is constructed based on the CTP values ​​of all first pixels.

[0101] Specifically, the CTP value of the first pixel is determined based on the pixel value, radius, average pixel value of the first pixel, the number of second pixels, and the pixel value of each second pixel, including:

[0102] The CTP value of the first pixel is calculated using formula (1):

[0103]

[0104] Wherein, CTP(n,R,g) c ) represents the CTP value of the first pixel, n represents the number of second pixels, R represents the radius, and g represents the value of the second pixel. c The pixel value of the first pixel, g d Let d be the pixel value of the second pixel. For the first The pixel value of the second pixel. s(x) is a three-valued function, expressed by formula (2):

[0105]

[0106] Where t is the average pixel value.

[0107] It should be understood that d = 0, 1, 2...n-1.

[0108] That is, in the calculation formula (1) Then Substitute x into formula (2) for calculation, and similarly in formula (1) Then Substitute x into formula (2) for calculation.

[0109] In practical applications, each first pixel point g in the image to be identified c Placed in a small window, and with the first pixel g c Centered on g. Then, using g as the center point. cUsing the circle as the center, construct the largest possible circle within the small window and determine its radius R. Then, determine the second pixels distributed along the side (ring) of this circle, and determine the pixel value of each second pixel, as well as the average pixel value t and the number n of all second pixels.

[0110] It should be understood that, due to the first pixel g c Since it inherently possesses rich feature texture information, incorporating the influence of the first pixel when calculating the CTP feature map can enhance the richness of information in the CTP feature map.

[0111] Figure 3 This is a schematic diagram illustrating the calculation principle of the CTP feature map provided in an embodiment of this application. Figure 3 As shown, the nine squares in the image represent nine pixels. The gray areas around the edges are the second pixels, and the white area in the center is the first pixel. These nine pixels belong to a pixel region. The dashed circle within the pixel region represents the largest possible circle centered on the first pixel within that region, and the dashed line represents the radius of this circle.

[0112] S22. Input the DWT low-frequency subband map and CTP feature map into the expression recognition model to obtain the expression output by the expression recognition model.

[0113] The facial expression recognition model is trained in advance based on the sample DWT low-frequency subband map and sample CTP feature map of the sample image.

[0114] In one possible implementation, the DWT low-frequency subband map and the CTP feature map can be concatenated, and the concatenated feature map can be input into the expression recognition model to obtain the expression output by the expression recognition model.

[0115] In one possible implementation, the DWT low-frequency subband map and CTP feature map can be directly input into the expression recognition model to obtain the expression output by the expression recognition model.

[0116] The facial expression recognition model includes smCNN, which is used to fuse the DWT low-frequency subband map and the CTP feature map to obtain a fused feature map.

[0117] Figure 4 This is a schematic diagram of the smCNN structure provided in an embodiment of this application. Figure 4 As shown, smCNN includes an input layer, a first sampling layer, a multi-scale convolutional layer, a second sampling layer, a first fully connected layer, and a second fully connected layer arranged in sequence.

[0118] Furthermore, the expression recognition model also includes a discriminator, which determines the facial expression of the person in the image to be recognized based on the fused feature map.

[0119] In practical applications, the input concatenated feature map is obtained through the input layer. The first sampling layer uses a 2×2 Max sampling method to sample the concatenated feature map, resulting in a first sampled feature map. Then, a multi-scale convolutional layer performs convolution processing on the first sampled feature map. This multi-scale convolutional layer uses four different scale convolutional kernels: 2×2, 3×3, 4×4, and 5×5. Each scale contains 30 different kernel parameters. After convolution, each scale kernel corresponds to 30 different convolutional feature maps, resulting in a total of 120 convolutional feature maps. Next, a second sampling layer reduces the dimensionality of these 120 convolutional feature maps. The second sampling layer uses a 3×3 Max sampling method, and the 120 convolutional feature maps are concatenated into a single second sampled feature map after passing through the second sampling layer.

[0120] The convolution operation can be represented by the following formula (3):

[0121]

[0122] Among them, l j Let u be the convolutional feature map output for the j-th image. i Let k be the first sampled feature map of the i-th input. ij For l j and u i The convolution kernel.

[0123] Furthermore, by using two adjacent fully connected layers, the first and second fully connected layers, the second sampled feature map is effectively reduced in dimensionality, thus obtaining the fused feature map. This completes the multi-feature fusion using smCNN.

[0124] For example, suppose there are C categories in a classification problem, the number of neurons in the first fully connected layer is set to 4×C, and the number of neurons in the second fully connected layer is set to 2×C.

[0125] Furthermore, the fused feature map can be processed using a Softmax classifier to generate the facial expression of the image to be recognized.

[0126] The facial expression recognition method provided in this application obtains the DWT low-frequency subband map and CTP feature map of the image to be recognized, and inputs the DWT low-frequency subband map and CTP feature map into the facial expression recognition model to obtain the facial expression output by the model. The image to be recognized contains a face, and the facial expression recognition model is pre-trained based on sample DWT low-frequency subband maps and sample CTP feature maps of sample images. In this technical solution, the DWT low-frequency subband map preserves the global structural features of the face, while the CTP feature map depicts stable texture details in local areas, ensuring the richness and completeness of the features. These two complementary deep features are jointly input into the facial expression recognition model for joint training and decision-making, thereby improving the accuracy of facial expression recognition using the model.

[0127] Optionally, in some embodiments, the Haar feature map of the image to be recognized can also be obtained. Based on this, S12 can be implemented as follows: inputting the DWT low-frequency subband map, Haar feature map, and CTP feature map into the expression recognition model to obtain the expression output by the expression recognition model.

[0128] In one possible implementation, the DWT low-frequency subband map, Haar feature map, and CTP feature map can be concatenated to obtain a concatenated feature map. The concatenated feature map is then input into the expression recognition model to obtain the expression output by the expression recognition model.

[0129] Specifically, the main idea of ​​Haar feature integral maps is to store the sum of pixels in rectangular regions formed by the starting point and all other points in the image as elements of an array in memory. When calculating the pixel sum of a certain region, the array element can be directly indexed without recalculating the pixel sum of that region, thus speeding up the computation. Haar feature integral maps can calculate different features at multiple scales using the same amount of time (constant time), thus greatly improving detection speed. Haar feature integral maps are a matrix representation method that can describe global information.

[0130] Specifically, the Haar feature integral map can be represented by the following formula (4):

[0131]

[0132] Where ii(i,j) represents the pixel value of all pixels at the bottom left corner of (i,j) in the image to be recognized (the bottom left corner is the origin), and f(k,l) represents the pixel value of pixel (k,l).

[0133] Specifically, the steps for constructing the Haar feature integral map are as follows:

[0134] 1. Let s(i,j) represent the cumulative sum in the row direction, where s(i,-1) is initialized to 0;

[0135] 2. Let ii(i,j) represent the integral graph, where ii(-1,j) is initialized to 0;

[0136] 3. Scan the image to be recognized line by line, and recursively calculate the cumulative sum s(i,j) and the integral image ii(i,j) of each pixel (i,j) in the row direction using the following formulas (5) and (6):

[0137] s(i,j)=s(i,j-1)+f(i,j) (5)

[0138] ii(i,j)=ii(i-1,j)+s(i,j) (6)

[0139] The image to be identified is scanned once, and when the top right pixel of the image is reached, the Haar feature integral map is constructed. After the Haar feature integral map is constructed, the sum of pixels in any matrix region of the image to be identified can be obtained through simple calculations.

[0140] Figure 5 This is a schematic diagram illustrating the calculation principle of the Haar feature integral map provided in an embodiment of this application. Figure 5 As shown, the pixel sum Dsum of region D in the image to be identified can be expressed by the following formula (7):

[0141] Dsum=ii(β)+ii(γ)-ii(α)-ii(δ) (7)

[0142] Furthermore, the Haar feature integral map of the image to be identified can be determined in the above manner, and then the Haar feature integral map can be averaged by 2×2 to extract the Haar feature map.

[0143] Optionally, in some embodiments, S11 can be implemented as follows:

[0144] Obtain the grayscale image of the image to be identified, and obtain the DWT low-frequency subband map and CTP feature map of the grayscale image.

[0145] Accordingly, the above-mentioned acquisition of the Haar feature map of the image to be identified can be achieved as follows:

[0146] Obtain the Haar feature map of the grayscale image.

[0147] Among them, the image to be recognized can be preprocessed to convert the RGB space of the image to be recognized to the gray space, and the gray image can be obtained by using the following formula (8).

[0148] I gray =0.299R + 0.587G + 0.114B (8)

[0149] Among them, I gray The image is a grayscale image, where R is the red channel of the image to be identified, G is the green channel of the image to be identified, and B is the blue channel of the image to be identified.

[0150] Furthermore, the grayscale image can be normalized in size.

[0151] It should be understood that obtaining the DWT low-frequency subband map of a grayscale image can be done by referring to the process described above for determining the DWT low-frequency subband map based on the image to be identified; obtaining the CTP feature map of a grayscale image can be done by referring to the process described above for determining the CTP feature map based on the image to be identified; and obtaining the Haar feature map of a grayscale image can be done by referring to the process described above for determining the CTP feature map based on the image to be identified. The difference lies in replacing "image to be identified" with "grayscale image" and replacing "pixel values ​​of the image to be identified" with "grayscale values ​​of the grayscale image".

[0152] Figure 6 This is a flowchart illustrating the model training process of the facial expression recognition method provided in the embodiments of this application, as shown below. Figure 6 As shown, this method can be implemented through the following steps:

[0153] S61. Preprocess the sample images.

[0154] The preprocessing involves converting the sample image to grayscale to generate a sample grayscale image.

[0155] S62. Collect the sample DWT low-frequency subband map, sample CTP feature map, and sample Haar feature map of the sample image.

[0156] Among them, the sample DWT low-frequency subband map, sample CTP feature map, and sample Haar feature map are generated based on the sample grayscale map.

[0157] It should be understood that the method of generating sample DWT low-frequency subband maps, sample CTP feature maps, and sample Haar feature maps during model training can refer to the method of generating DWT low-frequency subband maps, CTP feature maps, and Haar feature maps during model inference, and will not be elaborated here.

[0158] S63. The sample DWT low-frequency subband map, sample CTP feature map and sample Haar feature map of the sample image are fused by smCNN.

[0159] Furthermore, the specific fusion process can be referred to in S12 for the fusion of the DWT low-frequency subband map, CTP feature map, and Haar feature map, which will not be elaborated here.

[0160] S64. Train the Softmax classifier using the fused features.

[0161] The fused feature map (fused features) of the samples obtained from S63 is input into a C-dimensional Softmax classifier. The Softmax classifier is trained by supervised backpropagation algorithm based on the class labels corresponding to the sample DWT low-frequency subband map, sample CTP feature map and sample Haar feature map, and the mapping relationship R = g(e) of the Softmax classifier is obtained.

[0162] The category label is the facial expression contained in the sample image.

[0163] Where R represents the class output, and g(·) is the mapping from input to output established by the Softmax classifier.

[0164] Next, the error function is calculated based on the sample fusion feature map, and then the gradient descent method is used to calculate the minimum error, thereby obtaining the optimal network weights and biases. The model parameters of smCNN are then optimized to obtain a trained expression recognition model.

[0165] In summary, the facial expression recognition method provided in this application first performs image preprocessing on the video frame (image to be recognized) (converting from RGB space to grayscale space), then extracts the DWT low-frequency subband image, Haar feature map, and CTP feature map, and utilizes smCNN in the facial expression recognition model to process the DWT low-frequency subband image, Haar feature map, and CTP feature map. Figure 3 The features are fused together, and finally the classifier in the expression recognition model is used to process the fused feature map to determine the character's expression in the video frame.

[0166] Specifically, the DWT low-frequency subband map can filter out redundant information and noise, improving robustness to facial expressions and lighting conditions; the Haar feature map is divided into edge features, linear features, center features, and diagonal features, which are combined into a feature template. The feature template contains white and black rectangles, and its feature value is defined as the sum of the white rectangle pixels minus the sum of the black rectangle pixels; the CTP feature map considers the rich texture features of the center pixels. This invention extracts the DWT low-frequency subband map, Haar feature map, and CTP feature map. Figure 3 The three features are then fused at different scales using smCNN. The fused feature map has strong pose robustness and is insensitive to occlusion. The classification result is then predicted by a Softmax classifier, thus obtaining the facial micro-expression recognition result.

[0167] It is understood that the facial expression recognition method provided in this application has the following technical advantages:

[0168] 1. Based on DWT low-frequency subband map, Haar feature map and CTP feature Figure 3These three features and smCNN are used to extract facial expressions from each frame of a video. These three feature values ​​are easy to extract and simple to calculate, and can be used to extract facial regions in real time during video acquisition.

[0169] 2. Based on DWT low-frequency subband map, Haar feature map and CTP feature Figure 3 We use these features for facial expression recognition, ensuring feature integrity while reducing noise.

[0170] 3. Combine the DWT low-frequency subband map, Haar feature map, and CTP feature map. Figure 3 This approach combines various features with smCNN, leveraging the strengths of each to achieve robustness against facial pose, lighting, occlusion, and expression. Furthermore, it is designed from two different perspectives: data dimensionality reduction and reducing the number of keyframes. This effectively solves the problem of facial micro-expression recognition during video sampling.

[0171] 4. Combining DWT low-frequency subband map, Haar feature map, and CTP feature Figure 3 Based on these features, an expression recognition model incorporating smCNN is constructed, which reduces the number of convolutional layers while maintaining classification capabilities, thus improving quality and efficiency.

[0172] Figure 7 A schematic diagram of the facial expression recognition device provided in this application is shown below. Figure 7 As shown, the facial expression recognition device 70 provided in this embodiment includes:

[0173] The acquisition module 701 is used to acquire the DWT low-frequency subband map and CTP feature map of the image to be identified, and the image to be identified contains a face.

[0174] The input module 702 is used to input the DWT low-frequency subband map and CTP feature map into the expression recognition model to obtain the expression output by the expression recognition model. The expression recognition model is trained in advance based on the sample DWT low-frequency subband map and sample CTP feature map of the sample image.

[0175] In one possible implementation, the acquisition module 701 is specifically used for:

[0176] For each first pixel in the image to be identified, a pixel region of a preset window size centered on the first pixel is determined.

[0177] Within a pixel area, construct the largest possible circle with the first pixel as its center.

[0178] Determine the radius of the circle and the second pixel point where the side length of the circle is located.

[0179] Determine the number of second pixels and the pixel value of each second pixel.

[0180] The average pixel value of the second pixel is determined based on the number of second pixels and the pixel value of each second pixel.

[0181] The CTP value of the first pixel is determined based on the pixel value, radius, average pixel value of the first pixel, number of second pixels, and pixel value of each second pixel.

[0182] Based on the CTP values ​​of all first pixels, construct the CTP feature map of the image to be identified.

[0183] In one possible implementation, the acquisition module 701 is specifically used for:

[0184] The CTP value of the first pixel is calculated using formula (1):

[0185]

[0186] Wherein, CTP(n,R,g) c ) represents the CTP value of the first pixel, n represents the number of second pixels, R represents the radius, and g represents the value of the second pixel. c The pixel value of the first pixel, g d Let d be the pixel value of the second pixel. For the first The pixel value of the second pixel. s(x) is a three-valued function, expressed by formula (2):

[0187]

[0188] Where t is the average pixel value.

[0189] In one possible implementation, the facial expression recognition model includes smCNN, which comprises an input layer, a first sampling layer, a multi-scale convolutional layer, a second sampling layer, a first fully connected layer, and a second fully connected layer arranged in sequence.

[0190] In one possible implementation, the acquisition module 701 is further configured to:

[0191] Obtain the Haar feature map of the image to be identified.

[0192] Input module 702 is specifically used for:

[0193] Input the DWT low-frequency subband map, Haar feature map, and CTP feature map into the expression recognition model to obtain the expression output by the expression recognition model.

[0194] In one possible implementation, the acquisition module 701 specifically includes:

[0195] Obtain the grayscale image of the image to be identified.

[0196] Obtain the DWT low-frequency subband map and CTP feature map of the grayscale image.

[0197] Obtain the Haar feature map of the grayscale image.

[0198] The facial expression recognition device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0199] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the electronic device 80 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus 804.

[0200] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to perform the above-described method.

[0201] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0202] Optionally, the electronic device 80 can be Figure 1 The server in the middle.

[0203] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0204] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0205] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0206] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0207] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0208] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0209] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0210] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0211] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0213] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0214] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0215] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A facial expression recognition method, characterized in that, include: The discrete wavelet transform (DWT) low-frequency subband map and the circular texture pattern (CTP) feature map of the image to be identified are obtained, and the image to be identified contains a human face. The DWT low-frequency subband map and the CTP feature map are input into the expression recognition model to obtain the expression output by the expression recognition model. The expression recognition model is trained in advance based on the sample DWT low-frequency subband map and sample CTP feature map of the sample image.

2. The method according to claim 1, characterized in that, Obtain the CTP feature map of the image to be identified, including: For each first pixel in the image to be identified, a pixel region of a preset window size centered on the first pixel is determined; Within the pixel area, construct the largest possible circle with the first pixel as its center; Determine the second pixel point where the radius and side length of the circle are located; Determine the number of the second pixels and the pixel value of each second pixel; The average pixel value of the second pixel is determined based on the number of the second pixel and the pixel value of each second pixel. The CTP value of the first pixel is determined based on the pixel value of the first pixel, the radius, the average pixel value, the number of second pixels, and the pixel value of each second pixel. Based on the CTP values ​​of all first pixels, construct the CTP feature map of the image to be identified.

3. The method according to claim 2, characterized in that, Determining the CTP value of the first pixel based on the pixel value of the first pixel, the radius, the average pixel value, the number of second pixels, and the pixel value of each second pixel includes: The CTP value of the first pixel is calculated using formula (1): Wherein, CTP(n,R,g) c ) represents the CTP value of the first pixel, n represents the number of the second pixels, R represents the radius, and g represents the value of the second pixel. c The pixel value of the first pixel, g d Let d be the pixel value of the second pixel. For the first The pixel value of the second pixel; s(x) is a three-valued function, expressed by formula (2): Where t is the average pixel value.

4. The method according to any one of claims 1-3, characterized in that, The facial expression recognition model includes a single-layer multi-scale convolutional neural network (smCNN), which comprises an input layer, a first sampling layer, a multi-scale convolutional layer, a second sampling layer, a first fully connected layer, and a second fully connected layer arranged in sequence.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain the Haar feature map of the image to be identified; The step of inputting the DWT low-frequency subband map and the CTP feature map into the expression recognition model to obtain the expression output by the expression recognition model includes: The DWT low-frequency subband map, the Haar feature map, and the CTP feature map are input into the expression recognition model to obtain the expression output by the expression recognition model.

6. The method according to claim 5, characterized in that, The acquisition of the DWT low-frequency subband map and CTP feature map of the image to be identified includes: Obtain the grayscale image of the image to be identified; Obtain the DWT low-frequency subband map and CTP feature map of the grayscale image; Accordingly, obtaining the Haar feature map of the image to be identified includes: Obtain the Haar feature map of the grayscale image.

7. An expression recognition device, characterized in that, include: The acquisition module is used to acquire the Discrete Wavelet Transform (DWT) low-frequency subband map and the circular texture pattern (CTP) feature map of the image to be identified, wherein the image to be identified contains a human face; The input module is used to input the DWT low-frequency subband map and the CTP feature map into the expression recognition model to obtain the expression output by the expression recognition model. The expression recognition model is trained in advance based on the sample DWT low-frequency subband map and sample CTP feature map of the sample image.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images