Micro-expression recognition method and device, electronic equipment, storage medium and program product

Abstract

The application provides a micro-expression recognition method and device, electronic equipment, storage medium and program product, and belongs to the technical field of artificial intelligence. The method comprises the following steps: determining shallow high-resolution feature maps of different scales of a face image; determining deep-shallow fusion feature maps corresponding to the shallow high-resolution feature maps based on the shallow high-resolution feature maps and deep semantic feature maps corresponding to the shallow high-resolution feature maps; determining a multi-scale global fusion feature map of the face image based on the deep-shallow fusion feature maps; and inputting the multi-scale global fusion feature map into a micro-expression recognition model to obtain a micro-expression recognition result output by the micro-expression recognition model. The application first determines shallow high-resolution feature maps of different scales from a face image, and then determines deep-shallow fusion feature maps according to the shallow high-resolution feature maps and deep semantic feature maps corresponding to the shallow high-resolution feature maps, so that the shallow high-resolution feature maps and the deep semantic feature maps are effectively combined, and the feature representation capability of micro-expression recognition is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a micro-expression recognition method, device, electronic device, storage medium, and program product. Background Technology

[0002] Micro-expression recognition has become an important part of many human-computer interaction scenarios.

[0003] Currently, commonly used micro-expression recognition methods divide the Lip Reading Recognition (LRR) task into two steps: visual feature extraction and micro-expression recognition. These two steps are performed by two separate models. For example, in the visual feature extraction step, facial region images are extracted using methods such as Active Shape Model (ASM) or Active Appearance Model (AAM), and visual features such as facial shape and texture are calculated. These visual features are then input into a micro-expression recognition model built and trained based on Hidden Markov Models (HMMs), and the Viterbi algorithm is used to decode and obtain the recognition result.

[0004] However, when extracting visual features of facial micro-expressions based on shallow models such as ASM and AAM or hand-designed methods, it is difficult to fully capture the complex semantic information of the expression, resulting in insufficient feature representation capabilities and limited recognition performance. Summary of the Invention

[0005] This application provides a micro-expression recognition method, apparatus, electronic device, storage medium, and program product to address the shortcomings of insufficient feature representation capability and limited recognition performance in existing micro-expression recognition technologies.

[0006] This application provides a micro-expression recognition method, including: Determine shallow high-resolution feature maps at different scales of facial images; Based on each of the shallow high-resolution feature maps and their corresponding deep semantic feature maps, determine each deep-shallow fusion feature map corresponding to each of the shallow high-resolution feature maps. Based on the aforementioned deep and shallow fusion feature maps, a multi-scale global fusion feature map of the facial image is determined; The multi-scale global fusion feature map is input into the micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model.

[0007] According to the micro-expression recognition method provided in this application, the step of determining the deep-shallow fusion feature map corresponding to each of the shallow high-resolution feature maps and their corresponding deep semantic feature maps includes: sorting the shallow high-resolution feature maps in order from deep to shallow scale to obtain a shallow feature map sequence; adjusting the channel number of the first shallow high-resolution feature map in the shallow feature map sequence to obtain the deep-shallow fusion feature map corresponding to the first shallow high-resolution feature map; taking the second shallow high-resolution feature map in the shallow feature map sequence as the current shallow feature map; and determining the deep-shallow fusion feature map based on the upsampled feature map corresponding to the previous shallow high-resolution feature map of the current shallow feature map and the current shallow feature map. The deep semantic feature map corresponding to the current shallow feature map is described; the upsampled feature map corresponding to the previous shallow high-resolution feature map is obtained by upsampling the deep-shallow fusion feature map corresponding to the previous shallow high-resolution feature map; a convolution operation is performed on the deep semantic feature map corresponding to the current shallow feature map to obtain the deep-shallow fusion feature map corresponding to the current shallow feature map; the next shallow high-resolution feature map of the current shallow feature map is taken as the new current shallow feature map, and the process returns to the step of performing a convolution operation on the upsampled feature map corresponding to the previous shallow high-resolution feature map of the current shallow feature map and the current shallow feature map, until the current shallow feature map is the last shallow high-resolution feature map in the shallow feature map sequence.

[0008] According to the micro-expression recognition method provided in this application, the step of determining the multi-scale global fusion feature map of the facial image based on the deep and shallow fusion feature maps includes: determining the multi-scale global fusion feature map of the facial image based on the deep and shallow fusion feature maps whose micro-expression correlation is greater than a preset correlation threshold.

[0009] According to the micro-expression recognition method provided in this application, the step of determining the multi-scale global fusion feature map of the facial image based on the deep and shallow fusion feature maps includes: determining a first facial feature map based on the deep and shallow fusion feature maps; determining an attention-weighted second facial feature map based on the first facial feature map and its corresponding attention mapping matrix; and determining the multi-scale global fusion feature map based on the first facial feature map and the second facial feature map.

[0010] According to the micro-expression recognition method provided in this application, the step of determining the attention-weighted second facial feature map based on the first facial feature map and its corresponding attention mapping matrix includes: performing channel compression on the first facial feature map using a first convolution kernel with first weights to obtain a first compressed feature map; performing channel compression on the first facial feature map using a second convolution kernel with second weights to obtain a second compressed feature map; wherein the first weight is not equal to the second weight; determining the attention mapping matrix corresponding to the first facial feature map based on the similarity between the first compressed feature map and the second compressed feature map; and determining the attention-weighted second facial feature map based on the element-wise multiplication result of the first facial feature map and the attention mapping matrix.

[0011] According to the micro-expression recognition method provided in this application, the step of determining the multi-scale global fusion feature map based on the first facial feature map and the second facial feature map includes: determining a residual attention feature map based on the first facial feature map and the second facial feature map; determining the correlation weight between the second facial feature map and the residual attention feature map based on the optical parameter vector of the facial image, the scaling dot product attention calculation result of the second facial feature map and the residual attention feature map; and determining the multi-scale global fusion feature map based on the weighted summation result of the correlation weight and the residual attention feature map.

[0012] According to the micro-expression recognition method provided in this application, the step of determining shallow high-resolution feature maps of different scales of a facial image includes: inputting an initial feature map of the facial image into the micro-expression recognition model to obtain an initial micro-expression recognition result output by the micro-expression recognition model; if the confidence level of the initial micro-expression recognition result is less than a preset confidence threshold, then obtaining a region-enlarged image of the facial image; inputting the region-enlarged image into a multi-level feature extraction model to obtain shallow high-resolution feature maps of different scales output by the multi-level feature extraction model.

[0013] This application also provides a micro-expression recognition device, including: The first determining module is used to determine shallow high-resolution feature maps of different scales of the facial image; The second determining module is used to determine each shallow-deep fusion feature map corresponding to each shallow high-resolution feature map based on each shallow high-resolution feature map and its corresponding deep semantic feature map. The third determining module is used to determine the multi-scale global fusion feature map of the facial image based on the deep and shallow fusion feature maps; The recognition module is used to input the multi-scale global fusion feature map into the micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model.

[0014] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the micro-expression recognition method as described above.

[0015] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the micro-expression recognition method as described above.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the micro-expression recognition method as described above.

[0017] The micro-expression recognition method, apparatus, electronic device, storage medium, and program product provided in this application first determine shallow high-resolution feature maps of different scales from facial images, and then determine deep-shallow fusion feature maps based on each shallow high-resolution feature map and its corresponding deep semantic feature map. This allows for the simultaneous capture of local details and global semantic information of facial images. Furthermore, multi-scale global fusion feature maps of facial images are determined based on each deep-shallow fusion feature map. This achieves an effective combination of shallow high-resolution features and deep semantic features, obtaining a more comprehensive multi-scale global fusion feature representation that captures the complex semantic information of micro-expressions. This enhances the richness and discriminativeness of visual features in facial images, improves the feature representation capability of micro-expression recognition, and thus improves the accuracy performance of micro-expression recognition. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the micro-expression recognition method provided in this application.

[0020] Figure 2 This is a structural example diagram of the multi-level feature extraction model provided in this application.

[0021] Figure 3 This is a structural example diagram of the micro-expression recognition model provided in this application.

[0022] Figure 4 This is a schematic diagram of the micro-expression recognition device provided in this application.

[0023] Figure 5This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] It should be noted that, in the description of this application, the term "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Those skilled in the art will understand the specific meaning of the above terms in this application according to the specific circumstances.

[0026] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0027] The following is combined Figures 1-5 This application describes the micro-expression recognition method, apparatus, electronic device, storage medium, and program product provided.

[0028] Currently, the commonly used micro-expression recognition method divides the expression recognition task into two steps: visual feature extraction and micro-expression recognition. These two steps are completed by one or two models.

[0029] In the scheme that completes these two steps using two models, on the one hand, facial region images are extracted using methods such as ASM and AAM, and visual features such as facial shape and texture are calculated. On the other hand, the visual features are input into a micro-expression recognition model built and trained based on HMM, and the recognition result is obtained by decoding using the Viterbi algorithm.

[0030] However, when extracting visual features of facial micro-expressions based on shallow models such as ASM and AAM or hand-designed methods, it is difficult to fully capture the complex semantic information of the expression, resulting in insufficient feature representation capabilities and limited recognition performance.

[0031] In schemes that accomplish both steps using a single model, facial expression recognition is typically modeled as a sequence-to-sequence problem. Convolutional Neural Networks (CNNs) or Long Short-Term Memory (LSTMs) are used to automatically learn feature representations from image sequences including facial regions, and end-to-end modeling and recognition of the expression sequences are performed. Compared to ASM / AAM+HMM methods, this approach, which completes both feature extraction and task recognition with a single model, has stronger feature learning capabilities and can capture high-level semantic information of facial expressions. However, it faces challenges such as requiring more training data and higher model complexity. In other words, due to the need to design a large number of parameters and computational load, the high computational complexity and demanding hardware resources make this computationally intensive approach difficult to widely apply in resource-constrained scenarios such as mobile and embedded devices.

[0032] In view of this, this application provides a micro-expression recognition method, apparatus, electronic device, storage medium, and program product, which can more comprehensively capture the complex semantic information of expressions and improve feature representation capabilities by completing feature extraction and expression recognition steps through two models respectively, thereby improving micro-expression recognition performance.

[0033] Figure 1 This is a flowchart illustrating the micro-expression recognition method provided in this application, as shown below. Figure 1 As shown, the micro-expression recognition method includes, but is not limited to, steps 101 to 104.

[0034] It should be noted that the micro-expression recognition method provided in this application is executed by a micro-expression recognition device in the corresponding emotion recognition application field. Specifically, it can be a server, computer equipment, such as a smart assistant, home robot, mobile phone, tablet computer, laptop computer, handheld computer, vehicle electronic equipment, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0035] Step 101: Determine shallow high-resolution feature maps at different scales of the facial image.

[0036] Shallow high-resolution feature maps are feature representations extracted from facial images that have high spatial resolution and can finely reflect the original spatial structure of local regions of the facial image.

[0037] Shallow high-resolution feature maps of different scales have different spatial receptive field scales, which can be obtained by performing different convolution operations on facial images. The different convolution operations can be at least one of the following: kernel size, stride, and dilation rate.

[0038] Specifically, after waking up the robot, the robot's image acquisition device is used to acquire the user's facial image, and the facial image is input into the multi-scale shallow feature extraction layer of the pre-trained multi-level feature extraction model to obtain multiple shallow high-resolution feature maps of different scales output by the multi-level feature extraction model.

[0039] Among them, the multi-level feature extraction model is a neural network model used to extract multi-scale global fusion feature maps from facial images; the multi-scale shallow feature extraction layer is a neural network layer in the multi-level feature extraction model used to extract shallow high-resolution feature maps of different scales from facial images.

[0040] Optionally, the multi-scale shallow feature extraction layer is implemented based on the ResNet-101 model.

[0041] Figure 2 This is a structural example diagram of the multi-level feature extraction model provided in this application, such as... Figure 2 As shown, the multi-level feature extraction model includes a multi-scale shallow feature extraction layer, which consists of a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block, and a fifth convolutional block connected in sequence. The facial image is processed by the convolution of the first, second, third, fourth, and fifth convolutional blocks in sequence, resulting in shallow high-resolution feature maps of different scales output by the five convolutional blocks. .

[0042] Optionally, the facial image is obtained by preprocessing the video data collected by the robotic device, extracting video keyframes from the preprocessed video data, and cropping the facial region from the video keyframes to obtain the facial image.

[0043] Step 102: Based on each of the shallow high-resolution feature maps and their corresponding deep semantic feature maps, determine each deep-shallow fusion feature map corresponding to each of the shallow high-resolution feature maps.

[0044] Deep semantic feature maps are low-resolution semantic feature representations extracted from facial images that have low spatial resolution and hierarchical association and semantic matching relationships with shallow high-resolution feature maps.

[0045] The deep-shallow fusion feature map is a fused feature representation obtained by fusing shallow high-resolution feature maps and deep semantic feature maps that have hierarchical associations and semantic matching relationships.

[0046] Specifically, each shallow high-resolution feature map is input into the deep semantic feature extraction layer of the multi-level feature extraction model, and each deep semantic feature map corresponding to each shallow high-resolution feature map output by the deep semantic feature extraction layer and having hierarchical association and semantic matching relationship is obtained.

[0047] Among them, the deep semantic feature extraction layer is a neural network layer in the multi-level feature extraction model used to extract the corresponding deep semantic feature map based on the shallow high-resolution feature map; the multi-scale shallow feature extraction layer is close to the input layer of the multi-level feature extraction model, while the deep semantic feature extraction layer is close to the output layer of the multi-level feature extraction model.

[0048] For each shallow high-resolution feature map, a feature fusion operation is performed on the shallow high-resolution feature map and its corresponding deep semantic feature map to obtain the shallow-deep fused feature map corresponding to the shallow high-resolution feature map. By repeating the step of obtaining the shallow-deep fused feature map corresponding to each shallow high-resolution feature map, the shallow-deep fused feature maps corresponding to each shallow high-resolution feature map can be obtained.

[0049] Combination Figure 2 As shown, the multi-level feature extraction model also includes a deep semantic feature extraction layer. This deep semantic feature extraction layer comprises a first deconvolution block, a second deconvolution block, a third deconvolution block, a fourth deconvolution block, and a fifth deconvolution block connected sequentially. The first, second, third, fourth, and fifth deconvolution blocks correspond to the fifth, fourth, third, second, and first convolution blocks, respectively. This allows for the extraction of shallow high-resolution feature maps at different scales from the multi-scale shallow feature extraction layer. Each deconvolution block input to the deep semantic feature extraction layer is deconvolved to obtain a high-resolution feature map of each shallow layer. Corresponding deep semantic feature maps Furthermore, feature fusion processing is performed on each shallow high-resolution feature map and its corresponding deep semantic feature map to obtain each shallow high-resolution feature map. corresponding .

[0050] Step 103: Based on the aforementioned deep and shallow fusion feature maps, determine the multi-scale global fusion feature map of the facial image.

[0051] Multi-scale global fusion feature maps are fused feature maps obtained by fusing shallow high-resolution features and deep semantic features extracted from facial images.

[0052] Specifically, the multi-level feature extraction model further performs direct feature addition and fusion operations on all the deep and shallow fusion feature maps output by the deep semantic feature extraction layer to obtain a multi-scale global fusion feature map of the facial image; or, the multi-level feature extraction model further determines several target deep and shallow fusion feature maps related to micro-expressions from each deep and shallow fusion feature map, and performs feature addition and fusion operations on several target deep and shallow fusion feature maps to obtain a multi-scale global fusion feature map of the facial image, which is then used as the output of the multi-level feature extraction model.

[0053] Step 104: Input the multi-scale global fusion feature map into the micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model.

[0054] Optionally, the micro-expression recognition model is trained using multiple expression recognition training samples. Each expression recognition training sample includes a multi-scale global fusion feature map sample extracted from facial image samples and its corresponding micro-expression category label.

[0055] The micro-expression recognition results include several micro-expression category results and their corresponding confidence scores. The micro-expression category results include, but are not limited to, at least one of the emotion categories such as happiness, surprise, anger, dislike, fear, and sadness.

[0056] For example, the micro-expression recognition result is {(happy, 80%), (surprised, 15%), (fear, 5%)}.

[0057] Specifically, before recognizing micro-expressions in facial images, a micro-expression recognition model is pre-built and pre-trained. A multi-scale global fusion feature map is then input into the pre-trained micro-expression recognition model, which identifies micro-expressions in the facial image based on the multi-scale global fusion feature map, outputs the micro-expression recognition result, and determines the micro-expression category of the facial image based on the micro-expression category with the highest confidence among the recognition results.

[0058] Optionally, the micro-expression recognition model is built on an Hidden Markov Model (HMM).

[0059] Optionally, Figure 3 This is a structural example diagram of the micro-expression recognition model provided in this application, such as... Figure 3 As shown, the micro-expression recognition model includes an encoder module, a decoder module, and an output layer connected in sequence; The encoder module includes a position encoding layer and multiple stacked encoder units connected in sequence; each encoder unit includes a multi-head attention mechanism layer, a feedforward network layer, a normalization layer and a first residual connection layer connected in sequence; the input of the first residual connection layer is the output of the normalization layer and the output of the position encoding layer; The decoder module includes multiple stacked decoder units connected in sequence; each decoder unit includes a multi-head attention mechanism layer, an encoder-decoder attention mechanism layer, a feedforward network layer, a normalization layer, and a second residual connection layer connected in sequence; the input of the second residual connection layer is the output of the normalization layer and the input of the decoder module. Each feedforward network layer consists of two linear layers connected in sequence and a ReLU activation function; The output layer is a fully connected layer, which is used to map the output of the decoder module to the probability distribution corresponding to each micro-expression category result in micro-expression recognition. The probability distribution is also known as the confidence level.

[0060] The expression for the output layer is shown below: ; in, This represents the probability distribution of micro-expression recognition results; The results of micro-expression recognition; The input is a multi-scale global fusion feature map for the micro-expression recognition model; These are the weights of the fully connected layer; This is the output of the decoder module; For biasing the fully connected layer; is the activation function used to map the micro-expression category results in the output micro-expression recognition results to the range [0,1], and obtain the confidence score of each micro-expression category result.

[0061] By further processing the output of the encoder-decoder attention mechanism through the feedforward network layer in the decoder, the micro-expression recognition model can better capture complex patterns and semantic information in multi-scale global fusion feature maps. By introducing adaptive residual connections and layer normalization operations in the encoder and decoder, the output of the feedforward network is layer normalized, and then the normalized output is adaptively residually connected to the input of the decoder / encoder unit. This helps the model capture local and global features and accelerates the training process. Through the multi-layer stacking of decoder units, the output of each decoder unit is used as the input of the next decoder unit, thereby processing and enhancing the feature representation layer by layer, thus improving the expressive power and modeling depth of the decoder.

[0062] The micro-expression recognition method provided in this application first determines shallow high-resolution feature maps of different scales from facial images, and then determines deep-shallow fusion feature maps based on each shallow high-resolution feature map and its corresponding deep semantic feature map. This method can simultaneously capture local details and global semantic information of facial images. Furthermore, it determines multi-scale global fusion feature maps of facial images based on each deep-shallow fusion feature map, achieving an effective combination of shallow high-resolution features and deep semantic features. This results in a more comprehensive multi-scale global fusion feature representation that captures the complex semantic information of micro-expressions, enhancing the richness and discriminativeness of visual features in facial images, improving the feature representation capability of micro-expression recognition, and thus improving the accuracy performance of micro-expression recognition.

[0063] In another embodiment, a multi-scale global fusion feature map is input into a micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model, including: constructing a multimodal feature vector based on the multi-scale global fusion feature map and speech features; inputting the multimodal feature vector into the micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model; wherein, the speech features are determined based on audio signal data collected by the robot; the audio signal data and the facial image are collected at the same time and on the same object.

[0064] Alternatively, the speech features are extracted from the noise-filtered audio signal data using an acoustic model; for example, the speech features are Mel-Frequency Cepstral Coefficients (MFCC) features.

[0065] Optionally, a multimodal feature vector is constructed based on the multi-scale global fusion feature map and the speech features, including: obtaining the visual feature vector corresponding to the multi-scale global fusion feature map and the speech feature vector corresponding to the speech features, and concatenating the visual feature vector and the speech feature vector to obtain the multimodal feature vector.

[0066] By combining image and speech modal information for facial expression recognition, and using speech features as auxiliary information for micro-expression recognition, the robustness of micro-expression recognition can be improved.

[0067] Based on the above embodiments, as an optional embodiment, determining the shallow-sun fusion feature map corresponding to each of the shallow high-resolution feature maps and their corresponding deep semantic feature maps includes: The shallow high-resolution feature maps are sorted in order of scale from deep to shallow to obtain a shallow feature map sequence. The number of channels is adjusted in the first shallow high-resolution feature map of the shallow feature map sequence to obtain the deep-shallow fusion feature map corresponding to the first shallow high-resolution feature map; The second shallow high-resolution feature map in the shallow feature map sequence is used as the current shallow feature map; Based on the upsampled feature map corresponding to the previous shallow high-resolution feature map and the current shallow feature map, the deep semantic feature map corresponding to the current shallow feature map is determined; the upsampled feature map corresponding to the previous shallow high-resolution feature map is obtained by performing an upsampling operation on the deep-shallow fusion feature map corresponding to the previous shallow high-resolution feature map. Perform a convolution operation on the deep semantic feature map corresponding to the current shallow feature map to obtain the deep-shallow fusion feature map corresponding to the current shallow feature map; The next shallow high-resolution feature map of the current shallow feature map is taken as the new current shallow feature map, and the process returns to the step of performing a convolution operation on the upsampled feature map corresponding to the previous shallow high-resolution feature map of the current shallow feature map and the current shallow feature map, until the current shallow feature map is the last shallow high-resolution feature map in the shallow feature map sequence.

[0068] Specifically, the step of determining the deep-shallow fusion feature map corresponding to each shallow high-resolution feature map based on each shallow high-resolution feature map and its corresponding deep semantic feature map is as follows: Figure 2 Taking the multi-level feature extraction model shown as an example, combined with Figure 2 As shown, high-resolution feature maps of shallow layers at different scales are presented in order from deep to shallow. Sort the data to obtain a shallow feature map sequence. .

[0069] For shallow feature map sequences The first shallow high-resolution feature map, which is also the deepest shallow high-resolution feature map. The first shallow high-resolution feature map Corresponding deep semantic feature map That is, itself. The deep semantic feature map corresponding to the first shallow high-resolution feature map. Perform a convolution operation on the first deconvolution block to adjust the number of channels, resulting in the first shallow high-resolution feature map. Corresponding shallow and deep fusion feature maps .

[0070] For example, the convolution operation of the first deconvolution block is a convolution operation with a kernel size of 1×1, thus the deep and shallow fused feature maps... .

[0071] Then the shallow feature map sequence The second shallow high-resolution feature map As the current shallow feature map, the second deconvolution block processes the current shallow feature map. The previous shallow high-resolution feature map Corresponding shallow and deep fusion feature maps Perform an upsampling operation to obtain the previous shallow layer high-resolution feature map. Corresponding upsampled feature map The second deconvolution block further modulates the current shallow feature map. The upsampled feature map corresponding to the previous shallow high-resolution feature map and the current shallow feature map Determine the current shallow feature map Corresponding deep semantic feature map .For example, .

[0072] The second deconvolution block then processes the current shallow feature map. Corresponding deep semantic feature map Perform convolution operations to fuse features at different scales to obtain the current shallow feature map. Corresponding shallow and deep fusion feature maps .

[0073] For example, the convolution operation of the second deconvolution block is a convolution operation with a kernel size of 3×3, therefore the deep and shallow fused feature maps .

[0074] Optionally, the expression for the second deconvolution block is as follows: ; ; in, For the current shallow feature map The corresponding deep and shallow fusion feature maps; For the current shallow feature map The corresponding deep semantic feature map; This is the current shallow feature map; To implement the upsampling operation of deconvolution; For the current shallow feature map The deep-shallow fusion feature map corresponding to the previous shallow high-resolution feature map; for The convolution operation is used to fuse features at different scales.

[0075] Further improve the current shallow feature map The next shallow high-resolution feature map As the new current shallow feature map, we return to the step of performing a convolution operation between the upsampled feature map corresponding to the previous shallow high-resolution feature map and the current shallow feature map, that is, using the third deconvolution block to perform a convolution operation on the new current shallow feature map. Corresponding upsampled feature map and the new current shallow feature map Perform a convolution operation to obtain the current shallow feature map. Corresponding shallow and deep fusion feature maps This process continues until the current shallow feature map is the last shallow high-resolution feature map in the shallow feature map sequence. Gradually generated .

[0076] The micro-expression recognition method provided in this application constructs a multi-scale feature extraction mechanism that extracts information from shallow high-resolution feature maps at various scales, from deep to shallow. It then upsamples the deep-shallow fusion feature map corresponding to the previous shallow high-resolution feature map to generate a deep semantic feature map. This allows the deep semantic feature map, containing high-level semantic information, to be passed to the top layer of the hierarchical feature extraction mechanism. Finally, the deep semantic feature map and the shallow high-resolution feature map are convolved to obtain the deep-shallow fusion feature map. This method simultaneously connects and fuses features at different levels laterally and constructs a hierarchical feature extraction mechanism from top to bottom. It achieves an effective combination of shallow high-resolution features and deep semantic features, which helps to obtain a more comprehensive multi-scale global fusion feature representation that captures the complex semantic information of micro-expressions, thereby improving the richness and discriminativeness of facial image visual features.

[0077] Furthermore, based on such Figure 2 The multi-level feature extraction model shown achieves an end-to-end trainable feature extraction model that simultaneously integrates and fuses features from different levels laterally and constructs a top-down hierarchical feature extraction mechanism. It can automatically optimize feature representations according to task requirements while maintaining high computational efficiency. Furthermore, if... Figure 2 The multi-level feature extraction model shown and such Figure 3 The micro-expression recognition model shown can be deployed in robots in home human-computer interaction scenarios to significantly enhance the robot's adaptability to different speakers and complex home scenarios, and effectively improve the accuracy and robustness of expression recognition in home scenarios.

[0078] Based on the above embodiments, as an optional embodiment, determining the multi-scale global fusion feature map of the facial image based on each of the deep and shallow fusion feature maps includes: Based on the deep and shallow fusion feature maps whose micro-expression correlation is greater than a preset correlation threshold, a multi-scale global fusion feature map of the facial image is determined.

[0079] Specifically, such as Figure 2As shown, various depth-blending feature maps of the facial image were obtained. Then, the micro-expression correlation between each shallow and deep fusion feature map and the micro-expression region of the facial image is calculated, and the micro-expression correlation of each shallow and deep fusion feature map is compared with a preset correlation threshold. The shallow and deep fusion feature maps with micro-expression correlation greater than the preset correlation threshold are used as target shallow and deep fusion feature maps to obtain several target shallow and deep fusion feature maps. Then, feature addition and fusion operations are performed on several target shallow and deep fusion feature maps to obtain a multi-scale global fusion feature map of the facial image.

[0080] It should be noted that the micro-expression correlation of the deep and shallow fusion feature maps can be determined based on the correlation between the calculated feature representation and the micro-expression region. The preset correlation threshold can be determined according to the specific circumstances such as the accuracy requirements of micro-expression recognition and the response speed requirements of micro-expression recognition. This application does not impose any restrictions on this.

[0081] By simultaneously fusing features at different levels through lateral connections and constructing a hierarchical feature extraction mechanism from the top down, and by using micro-expression correlation to filter deep and shallow fusion feature maps for determining multi-scale global fusion feature maps of facial images, micro-expression recognition can focus on different facial features in different recognition scenarios. The recognition scenario is determined based on the emotion category.

[0082] For example, when recognizing happy micro-expressions in a happy scene, the robot will focus on the micro-expression areas of the mouth and face to identify the emotion of happiness; when recognizing surprised micro-expressions in a surprised scene, the robot will focus on the micro-expression areas of the eyebrows and forehead to identify the emotion of surprise; when recognizing sad micro-expressions in a sad scene, the robot will focus on the micro-expression area of ​​the mouth to identify the emotion of sadness.

[0083] The micro-expression recognition method provided in this application determines the multi-scale global fusion feature map input to the micro-expression recognition model by selecting deep and shallow fusion feature maps whose micro-expression correlation is greater than a preset correlation threshold. This is equivalent to introducing an adaptive gating mechanism, which adaptively controls the degree to which different scales of the multi-scale global fusion feature map retain or suppress the original information according to the micro-expression correlation, thereby further enhancing the expressive power of the multi-scale global fusion feature map and improving the accuracy and performance of micro-expression recognition.

[0084] Based on the above embodiments, as an optional embodiment, determining the multi-scale global fusion feature map of the facial image based on each of the deep and shallow fusion feature maps includes: Based on the aforementioned deep and shallow fusion feature maps, a first facial feature map is determined; Based on the first facial feature map and its corresponding attention mapping matrix, the attention-weighted second facial feature map is determined. The multi-scale global fusion feature map is determined based on the first facial feature map and the second facial feature map.

[0085] The attention mapping matrix is ​​a matrix consisting of attention weights corresponding to each position in the first facial feature map.

[0086] Specifically, the various deep and shallow fused feature maps output by the multi-level feature extraction model are added and fused to obtain a feature map with dimension [dimensional value missing]. First facial feature map And obtain the first facial feature map. Corresponding, dimension is Attention mapping matrix For example, using convolutional kernels with different weights on the first facial feature map Channel compression is performed to compute the attention mapping matrix using the different compressed feature maps. .

[0087] For the first facial feature map Attention mapping matrix Attention-weighted calculations are performed to obtain the attention-weighted second facial feature map. .

[0088] Finally, the first facial feature map Second facial feature map after attention weighting By performing addition, fusion and other processing, a multi-scale global fusion feature map corresponding to the facial image is obtained.

[0089] Optionally, the formula for calculating the second facial feature map is as follows: ; in, This is the second facial feature map; This is the first facial feature map; This is the attention mapping matrix.

[0090] In another embodiment, determining a first facial feature map based on each deep and shallow fusion feature map includes: determining a first facial feature map based on deep and shallow fusion feature maps whose micro-expression correlation is greater than a preset correlation threshold.

[0091] The micro-expression recognition method provided in this application determines the original first facial feature map by using a fusion feature map of different levels of features fused by lateral connections and top-down extraction of hierarchical features. An adaptive residual attention mechanism is introduced to learn the correlation between features at different positions in the first facial feature map, adaptively focus on key features, and generate an attention map that highlights important features and suppresses irrelevant features. Then, the attention map is combined with the original first facial feature map to weight important regions in the facial feature map, which can enhance the adaptability of multi-scale feature representation and obtain an enhanced multi-scale global fusion feature map, which helps to further improve the accuracy of micro-expression recognition.

[0092] Based on the above embodiments, as an optional embodiment, determining the attention-weighted second facial feature map based on the first facial feature map and its corresponding attention mapping matrix includes: The first facial feature map is compressed using a first convolution kernel with first weights to obtain a first compressed feature map. The first facial feature map is compressed using a second convolution kernel with second weights to obtain a second compressed feature map; the first weights are not equal to the second weights. Based on the similarity between the first compressed feature map and the second compressed feature map, the attention mapping matrix corresponding to the first facial feature map is determined; Based on the element-wise multiplication result of the first facial feature map and the attention mapping matrix, the attention-weighted second facial feature map is determined.

[0093] Specifically, in obtaining the first facial feature map Attention mapping matrix At the same time, on the one hand, the first weight is utilized The first convolution kernel on the first facial feature map Channel compression is performed to obtain the first compressed feature map. On the other hand, determining the first weight Different second weights Using the second weight The first convolution kernel on the first facial feature map Channel compression is performed to obtain the second compressed feature map. Among them, the first compressed feature map Second compression feature map All dimensions are .

[0094] Further calculation of the first compressed feature map Second compression feature map The similarity between them is used to calculate the first facial feature map. The similarity between positions is used to generate a first facial feature map. Attention mapping matrix .

[0095] Further analysis of the first facial feature map Attention mapping matrix Element-wise multiplication is performed, and the attention-weighted second facial feature map is determined based on the result. .

[0096] Optionally, the formula for calculating the attention mapping matrix is ​​as follows: ; in, This is the attention mapping matrix; This is the transpose of the first compressed feature map; This is the second compression feature.

[0097] Optionally, the first and second convolution kernels have the same kernel size, such as 1×1.

[0098] The micro-expression recognition method provided in this application introduces an adaptive residual attention mechanism. It uses convolutional kernels with different weights to compress the first facial feature map into different channels to obtain different compressed feature maps. The correlation between features at different positions in the first facial feature map is learned using different compressed feature maps. An attention mapping matrix is ​​generated based on the similarity between the compressed feature maps to highlight important features and suppress irrelevant features. The attention-weighted facial feature map is determined based on the attention mapping matrix. Finally, the enhanced multi-scale global fusion feature map is determined based on the original facial feature map and the attention-weighted facial feature map, thereby further improving the accuracy of micro-expression recognition.

[0099] Based on the above embodiments, as an optional embodiment, determining the multi-scale global fusion feature map based on the first facial feature map and the second facial feature map includes: Based on the first facial feature map and the second facial feature map, a residual attention feature map is determined; Based on the optical parameter vector of the facial image, the scaling dot product attention calculation results of the second facial feature map and the residual attention feature map, the correlation weight between the second facial feature map and the residual attention feature map is determined; The multi-scale global fusion feature map is determined based on the weighted sum of the correlation weights and the residual attention feature map.

[0100] The optical parameter vector of the facial image is determined based on at least one of the optical parameters of the facial image, such as illumination intensity, color temperature, illuminance uniformity, illumination direction, contrast, and color rendering index.

[0101] It is understood that the determination of the optical parameter vector of a facial image can be obtained by normalizing and concatenating the optical parameter values ​​after obtaining at least one of the aforementioned optical parameter values ​​of the facial image, or it can be determined based on other related calculation methods. This application does not limit this method.

[0102] Specifically, the first facial feature map Second facial feature map The sums are then passed through a convolutional layer to obtain the residual attention feature map. .

[0103] Further utilizing the scaling dot product attention mechanism, the second facial feature map is... As a query vector, the residual attention feature map The second facial feature map is calculated by combining the key and value vectors with the optical parameter vectors of the facial image. and residual attention feature map Correlation weights between each location feature .

[0104] Finally, the relevance weights As weights for the residual attention feature map By performing weighted summation, a new feature representation that integrates global information and the current input context is obtained, namely a multi-scale global fusion feature map. .

[0105] Optionally, the formula for calculating the residual attention feature map is as follows: ; in, This is a residual attention feature map; It is a learnable scaling factor used to scale the effect of attention enhancement; This is the second facial feature map; This is the first facial feature map; for The convolution kernel weights.

[0106] Optionally, the formula for calculating the relevance weight is as follows: ; in, Relevance weights; This is the second facial feature map; This is a vector of optical parameters for the facial image; This is the linear transformation matrix of the second facial feature map; is the linear transformation matrix of the residual attention feature map; The attention dimension is used to scale the dot product to ensure numerical stability. This is a residual attention feature map; This is the softmax activation function, used to normalize the dot product result to the interval [0, 1].

[0107] Optionally, the calculation formula for the multi-scale global fusion feature map is as follows: ; in, This is a multi-scale global fusion feature map; The sequence length of the residual attention feature map; Relevance weights; This is a residual attention feature map; is the linear transformation matrix of the residual attention feature map.

[0108] In low-light and complex environments such as home, facial images captured may exhibit uneven spatial distribution of optical parameters due to the influence of multiple light sources, reflections, and shadows. For example, facial images captured under a desk lamp may show light variations with one side bright and the other dark. This can affect the accuracy of multi-level feature extraction models in locating micro-expression regions and the performance of feature extraction. Consequently, the extracted feature maps will be affected by changes in light, reducing the accuracy and stability of micro-expression recognition and negatively impacting the user experience.

[0109] The micro-expression recognition method provided in this application, by introducing an adaptive residual attention mechanism to process the first facial feature map and the attention-weighted second facial feature map, utilizes a scaled dot product attention mechanism to combine the optical parameter vector of the facial image, the attention-weighted second facial feature map, and the residual attention feature map to calculate the relevance weights. This allows the feature map to learn the relevance of features at each location. The input of the micro-expression recognition model is determined by using the weighted sum of the relevance weights and the residual attention feature map, resulting in a multi-scale global fusion feature map that integrates contextual information and the most globally relevant features. This overcomes the problem of light variation interference affecting the accuracy and stability of expression recognition in home scenes, improving anti-interference ability and adaptability, and ensuring the stability and reliability of recognition results in low-light environments. Furthermore, by simultaneously utilizing a hierarchical feature extraction mechanism and an adaptive residual attention mechanism to determine the input of the micro-expression recognition model, it achieves an effective combination of shallow high-resolution features and deep semantic features. On the other hand, it can adaptively focus on important features and suppress irrelevant influences, achieving efficient modeling of targets at different scales. Finally, it obtains a multi-scale global fusion feature map that integrates multi-scale features, thereby improving the accuracy and stability of micro-expression recognition.

[0110] Based on the above embodiments, as an optional embodiment, determining shallow high-resolution feature maps of different scales of the facial image includes: The initial feature map of the facial image is input into the micro-expression recognition model to obtain the initial micro-expression recognition result output by the micro-expression recognition model; If the confidence level of the initial micro-expression recognition result is less than a preset confidence threshold, then an enlarged region image of the facial image is obtained. The enlarged image of the region is input into a multi-level feature extraction model to obtain shallow high-resolution feature maps of different scales output by the multi-level feature extraction model.

[0111] The initial feature map is a feature representation extracted from facial images using methods such as ASM and AAM that do not involve hierarchical feature extraction mechanisms and adaptive residual attention mechanisms.

[0112] Specifically, before determining whether to use hierarchical feature extraction and / or adaptive residual attention mechanisms to determine the input of the micro-expression recognition model, an initial feature map is first obtained from the facial image. This initial feature map is then input into the micro-expression recognition model to obtain the initial micro-expression recognition result output by the model. The confidence score of the micro-expression category with the highest confidence score is used as the confidence score of the initial micro-expression recognition result. If the confidence score of the initial micro-expression recognition result is less than a preset confidence threshold (e.g., 80%, 90%), it indicates that the micro-expression has not been successfully recognized. In this case, the robot expands the facial region for micro-expression recognition in the facial image pixel by pixel to obtain an expanded facial image. This expanded image is then input into the multi-level feature extraction model to recapture expression information, resulting in shallow high-resolution feature maps at different scales output by the multi-level feature extraction model. This allows the use of hierarchical feature extraction and / or adaptive residual attention mechanisms to obtain the final multi-scale global fusion feature map, enhancing the feature representation capability.

[0113] Understandably, if the confidence level of the initial micro-expression recognition result is greater than or equal to the preset confidence threshold, the initial micro-expression recognition result can be directly used as the final micro-expression recognition result, without the need to extract multi-scale global fusion feature maps from the facial image to recognize micro-expressions.

[0114] It should be noted that the preset reliability threshold can be set according to the actual situation, and this application does not impose any restrictions on it.

[0115] Generally speaking, when recognizing micro-expressions based on facial images, a lot of facial information is involved, but some emotional expressions on the face are not obvious, which can lead to situations where reliable micro-expression recognition results cannot be obtained.

[0116] The micro-expression recognition method provided in this application first performs initial simple feature extraction on the facial image to identify micro-expressions, and then expands the facial recognition area to obtain an enlarged image when a reliable micro-expression region is not identified. The enlarged image is then used to extract shallow high-resolution feature maps at different scales, and finally a multi-scale global fusion feature map is obtained. This achieves a feature extraction and micro-expression recognition scheme that balances recognition efficiency and recognition reliability.

[0117] Figure 4 This is a schematic diagram of the micro-expression recognition device provided in this application, as shown below. Figure 4 As shown, the micro-expression recognition device includes, but is not limited to, a first determining module 401, a second determining module 402, a third determining module 403, and a recognition module 404.

[0118] The first determining module 401 is used to determine shallow high-resolution feature maps of different scales of the facial image.

[0119] The second determining module 402 is used to determine each shallow-deep fusion feature map corresponding to each shallow high-resolution feature map based on each shallow high-resolution feature map and its corresponding deep semantic feature map.

[0120] The third determining module 403 is used to determine the multi-scale global fusion feature map of the facial image based on the deep and shallow fusion feature maps.

[0121] The recognition module 404 is used to input the multi-scale global fusion feature map into the micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model.

[0122] It should be noted that the micro-expression recognition device provided in this application can execute the micro-expression recognition method described in any of the above embodiments during actual operation, which will not be elaborated in this embodiment.

[0123] The micro-expression recognition device provided in this application first determines shallow high-resolution feature maps of different scales from facial images, and then determines deep-shallow fusion feature maps based on each shallow high-resolution feature map and its corresponding deep semantic feature map. This allows for the simultaneous capture of local details and global semantic information of facial images. Furthermore, it determines multi-scale global fusion feature maps of facial images based on each deep-shallow fusion feature map, achieving an effective combination of shallow high-resolution features and deep semantic features. This results in a more comprehensive multi-scale global fusion feature representation that captures the complex semantic information of micro-expressions, enhancing the richness and discriminativeness of facial image visual features, improving the feature representation capability of micro-expression recognition, and thus improving the accuracy performance of micro-expression recognition.

[0124] As an optional embodiment, the second determining module is further configured to: sort the shallow high-resolution feature maps in descending order of scale to obtain a shallow feature map sequence; perform a channel number adjustment operation on the first shallow high-resolution feature map in the shallow feature map sequence to obtain a deep-shallow fusion feature map corresponding to the first shallow high-resolution feature map; take the second shallow high-resolution feature map in the shallow feature map sequence as the current shallow feature map; and determine the deep semantic feature map corresponding to the current shallow feature map based on the upsampled feature map corresponding to the previous shallow high-resolution feature map and the current shallow feature map; the previous shallow high-resolution feature map... The upsampled feature map corresponding to the resolution feature map is obtained by upsampling the deep-shallow fusion feature map corresponding to the previous shallow high-resolution feature map; a convolution operation is performed on the deep semantic feature map corresponding to the current shallow feature map to obtain the deep-shallow fusion feature map corresponding to the current shallow feature map; the next shallow high-resolution feature map of the current shallow feature map is taken as the new current shallow feature map, and the process returns to the step of performing a convolution operation on the upsampled feature map corresponding to the previous shallow high-resolution feature map of the current shallow feature map and the current shallow feature map, until the current shallow feature map is the last shallow high-resolution feature map in the shallow feature map sequence.

[0125] As an optional embodiment, the third determining module further includes: a filtering submodule, used to determine the multi-scale global fusion feature map of the facial image based on the deep and shallow fusion feature maps whose micro-expression correlation is greater than a preset correlation threshold.

[0126] As an optional embodiment, the third determining module further includes: a first fusion module, a weighted calculation module, and a second fusion module; The first fusion module is used to determine a first facial feature map based on the aforementioned deep and shallow fusion feature maps; The weighted calculation module is used to determine the attention-weighted second facial feature map based on the first facial feature map and its corresponding attention mapping matrix. The second fusion module is used to determine the multi-scale global fusion feature map based on the first facial feature map and the second facial feature map.

[0127] As an optional embodiment, the weighted calculation module is further configured to: compress the first facial feature map using a first convolution kernel with a first weight to obtain a first compressed feature map; compress the first facial feature map using a second convolution kernel with a second weight to obtain a second compressed feature map; wherein the first weight is not equal to the second weight; determine the attention mapping matrix corresponding to the first facial feature map based on the similarity between the first compressed feature map and the second compressed feature map; and determine the attention-weighted second facial feature map based on the element-wise multiplication result of the first facial feature map and the attention mapping matrix.

[0128] As an optional embodiment, the second fusion module is further configured to: compress the first facial feature map using a first convolution kernel with first weights to obtain a first compressed feature map; compress the first facial feature map using a second convolution kernel with second weights to obtain a second compressed feature map; wherein the first weights are not equal to the second weights; determine the attention mapping matrix corresponding to the first facial feature map based on the similarity between the first compressed feature map and the second compressed feature map; and determine the attention-weighted second facial feature map based on the element-wise multiplication result of the first facial feature map and the attention mapping matrix.

[0129] As an optional embodiment, the first determining module is further configured to: input the initial feature map of the facial image into the micro-expression recognition model to obtain the initial micro-expression recognition result output by the micro-expression recognition model; if the confidence of the initial micro-expression recognition result is less than a preset confidence threshold, then obtain an enlarged region image of the facial image; input the enlarged region image into a multi-level feature extraction model to obtain shallow high-resolution feature maps of different scales output by the multi-level feature extraction model.

[0130] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the micro-expression recognition method provided in any of the above embodiments. The micro-expression recognition method includes, but is not limited to, the following steps: determining shallow high-resolution feature maps of different scales of a facial image; determining deep-shallow fusion feature maps corresponding to each shallow high-resolution feature map based on each shallow high-resolution feature map and its corresponding deep semantic feature map; determining multi-scale global fusion feature maps of the facial image based on each deep-shallow fusion feature map; and inputting the multi-scale global fusion feature maps into a micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model.

[0131] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the micro-expression recognition method provided in any of the above embodiments. The micro-expression recognition method includes, but is not limited to, the following steps: determining shallow high-resolution feature maps of different scales of a facial image; determining shallow-deep fusion feature maps corresponding to each shallow high-resolution feature map based on each shallow high-resolution feature map and its corresponding deep semantic feature map; determining multi-scale global fusion feature maps of the facial image based on each shallow-deep fusion feature map; and inputting the multi-scale global fusion feature maps into a micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model.

[0133] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the micro-expression recognition method provided in any of the above embodiments. The micro-expression recognition method includes, but is not limited to, the following steps: determining shallow high-resolution feature maps of different scales of a facial image; determining shallow-deep fusion feature maps corresponding to each shallow high-resolution feature map based on each shallow high-resolution feature map and its corresponding deep semantic feature map; determining multi-scale global fusion feature maps of the facial image based on each shallow-deep fusion feature map; and inputting the multi-scale global fusion feature maps into a micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A micro-expression recognition method, characterized in that, include: Determine shallow high-resolution feature maps at different scales of facial images; Based on each of the shallow high-resolution feature maps and their corresponding deep semantic feature maps, determine each deep-shallow fusion feature map corresponding to each of the shallow high-resolution feature maps. Based on the aforementioned deep and shallow fusion feature maps, a multi-scale global fusion feature map of the facial image is determined; The multi-scale global fusion feature map is input into the micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model; The step of determining the multi-scale global fusion feature map of the facial image based on each of the deep and shallow fusion feature maps includes: Based on the aforementioned deep and shallow fusion feature maps, a first facial feature map is determined; Based on the first facial feature map and its corresponding attention mapping matrix, the attention-weighted second facial feature map is determined. The multi-scale global fusion feature map is determined based on the first facial feature map and the second facial feature map; The step of determining the multi-scale global fusion feature map based on the first facial feature map and the second facial feature map includes: Based on the first facial feature map and the second facial feature map, a residual attention feature map is determined; Based on the optical parameter vector of the facial image, the scaling dot product attention calculation results of the second facial feature map and the residual attention feature map, the correlation weight between the second facial feature map and the residual attention feature map is determined; The multi-scale global fusion feature map is determined based on the weighted sum of the correlation weights and the residual attention feature map.

2. The micro-expression recognition method according to claim 1, characterized in that, The step of determining the shallow-south fusion feature map corresponding to each shallow high-resolution feature map and its corresponding deep semantic feature map includes: The shallow high-resolution feature maps are sorted in order of scale from deep to shallow to obtain a shallow feature map sequence. The number of channels is adjusted in the first shallow high-resolution feature map of the shallow feature map sequence to obtain the deep-shallow fusion feature map corresponding to the first shallow high-resolution feature map; The second shallow high-resolution feature map in the shallow feature map sequence is used as the current shallow feature map; Based on the upsampled feature map corresponding to the previous shallow high-resolution feature map and the current shallow feature map, the deep semantic feature map corresponding to the current shallow feature map is determined; the upsampled feature map corresponding to the previous shallow high-resolution feature map is obtained by performing an upsampling operation on the deep-shallow fusion feature map corresponding to the previous shallow high-resolution feature map. Perform a convolution operation on the deep semantic feature map corresponding to the current shallow feature map to obtain the deep-shallow fusion feature map corresponding to the current shallow feature map; The next shallow high-resolution feature map of the current shallow feature map is taken as the new current shallow feature map, and the process returns to the step of performing a convolution operation on the upsampled feature map corresponding to the previous shallow high-resolution feature map of the current shallow feature map and the current shallow feature map, until the current shallow feature map is the last shallow high-resolution feature map in the shallow feature map sequence.

3. The micro-expression recognition method according to claim 1, characterized in that, The step of determining the multi-scale global fusion feature map of the facial image based on the aforementioned deep and shallow fusion feature maps includes: Based on the deep and shallow fusion feature maps whose micro-expression correlation is greater than a preset correlation threshold, a multi-scale global fusion feature map of the facial image is determined.

4. The micro-expression recognition method according to claim 1, characterized in that, The step of determining the attention-weighted second facial feature map based on the first facial feature map and its corresponding attention mapping matrix includes: The first facial feature map is compressed using a first convolution kernel with first weights to obtain a first compressed feature map. The first facial feature map is compressed using a second convolution kernel with second weights to obtain a second compressed feature map; the first weights are not equal to the second weights. Based on the similarity between the first compressed feature map and the second compressed feature map, the attention mapping matrix corresponding to the first facial feature map is determined; Based on the element-wise multiplication result of the first facial feature map and the attention mapping matrix, the attention-weighted second facial feature map is determined.

5. The micro-expression recognition method according to claim 1, characterized in that, The determination of shallow high-resolution feature maps at different scales of the facial image includes: The initial feature map of the facial image is input into the micro-expression recognition model to obtain the initial micro-expression recognition result output by the micro-expression recognition model; If the confidence level of the initial micro-expression recognition result is less than a preset confidence threshold, then an enlarged region image of the facial image is obtained. The enlarged image of the region is input into a multi-level feature extraction model to obtain shallow high-resolution feature maps of different scales output by the multi-level feature extraction model.

6. A micro-expression recognition device, characterized in that, include: The first determining module is used to determine shallow high-resolution feature maps of different scales of the facial image; The second determining module is used to determine each shallow-deep fusion feature map corresponding to each shallow high-resolution feature map based on each shallow high-resolution feature map and its corresponding deep semantic feature map. The third determining module is used to determine the multi-scale global fusion feature map of the facial image based on the deep and shallow fusion feature maps; The recognition module is used to input the multi-scale global fusion feature map into the micro-expression recognition model to obtain the micro-expression recognition result output by the micro-expression recognition model; The third determining module is further configured to determine a first facial feature map based on each of the deep and shallow fusion feature maps; determine an attention-weighted second facial feature map based on the first facial feature map and its corresponding attention mapping matrix; and determine the multi-scale global fusion feature map based on the first facial feature map and the second facial feature map. The third determining module is further configured to determine a residual attention feature map based on the first facial feature map and the second facial feature map; Based on the optical parameter vector of the facial image, the scaling dot product attention calculation results of the second facial feature map and the residual attention feature map, the correlation weight between the second facial feature map and the residual attention feature map is determined; The multi-scale global fusion feature map is determined based on the weighted sum of the correlation weights and the residual attention feature map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the micro-expression recognition method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the micro-expression recognition method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the micro-expression recognition method as described in any one of claims 1 to 5.

Images