Micro-expression recognition method and device applied to emotional intention analysis

Abstract

The present application relates to the technical field of computer vision, and particularly relates to a micro-expression recognition method and device applied to emotion intention analysis. The method comprises: performing video frame extraction on a micro-expression video to be detected to obtain a target video frame; performing time and space analysis on a target intermediate frame and a target vertex frame to obtain micro-expression space-time features; performing dynamic analysis on a target starting frame and the target vertex frame to obtain micro-expression dynamic features; performing feature fusion on the micro-expression space-time features and the micro-expression dynamic features to obtain micro-expression fusion features; and performing expression type recognition on the micro-expression fusion features to obtain a micro-expression recognition result. Time analysis, space analysis and dynamic analysis are performed on micro-expression video frames to improve the richness of the obtained micro-expression features, so that the recognition result obtained by performing expression type recognition according to the micro-expression fusion features has higher accuracy, and the reliability of subsequent emotion intention analysis on a target individual can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for micro-expression recognition applied to emotional intent analysis. Background Technology

[0002] Microexpressions are subtle and fleeting forms of human facial expression. These expressions are extremely brief (typically lasting only 40 to 200 milliseconds), yet they play a crucial role in human emotional expression and social interaction. Microexpressions often appear when individuals attempt to conceal their emotional reactions during social interactions; they are unconscious expressions of emotion. The difficulty in controlling and disguising microexpressions makes them an important basis for understanding the emotional intentions of others. Microexpression analysis has significant applications in lie detection, interrogation, depression treatment, and business negotiations. Currently, optical flow is primarily used for facial microexpression recognition and analysis. This method captures facial movement by analyzing pixel changes between the microexpression initiation frame (the video frame corresponding to the start of the microexpression) and the peak frame (the video frame corresponding to the moment of maximum microexpression intensity), thereby extracting expression features to identify the corresponding expression type.

[0003] However, in practice, it has been found that because micro-expressions are short in duration and correspond to small facial movements, few facial features can be extracted by analyzing pixel changes between the start frame and the vertex frame of a micro-expression. This makes the accuracy of micro-expression recognition low and the reliability of the analysis results of the target individual's emotional intent low.

[0004] Therefore, improving the accuracy of micro-expression recognition, thereby enhancing the reliability of the emotional intent analysis results of target individuals, is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] This invention provides a micro-expression recognition method and apparatus for emotional intent analysis, which can improve the accuracy of micro-expression recognition and thus enhance the reliability of emotional intent analysis results for target individuals.

[0006] To address the aforementioned technical problems, the first aspect of this invention discloses a micro-expression recognition method applied to emotional intent analysis, the method comprising:

[0007] Video frames are extracted from the micro-expression video to be detected to obtain target video frames. The micro-expression video is equipped with annotation information for the start time of the micro-expression and the time when the micro-expression motion intensity is the maximum. The target video frame includes a target start frame, a target middle frame, and a target vertex frame. The target start frame is the video frame corresponding to the start time of the micro-expression. The target vertex frame is the video frame corresponding to the time when the micro-expression motion intensity is the maximum. The target middle frame is at least one video frame corresponding to the middle time after the start time of the micro-expression and before the time when the micro-expression motion intensity is the maximum.

[0008] Temporal and spatial analysis is performed on the target intermediate frame and the target vertex frame to obtain the spatiotemporal features of micro-expressions;

[0009] Dynamic analysis is performed on the target start frame and the target vertex frame to obtain micro-expression dynamic features;

[0010] The spatiotemporal features and dynamic features of micro-expressions are fused to obtain micro-expression fusion features;

[0011] Based on the micro-expression fusion features, the micro-expression type is identified, and the micro-expression recognition result is obtained.

[0012] As an optional implementation, in the first aspect of the present invention, the step of performing temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain micro-expression spatiotemporal features includes:

[0013] Perform motion magnification on the target intermediate frame and the target vertex frame to obtain a motion magnified frame;

[0014] The motion magnification frame is analyzed in time and space using a preset spatiotemporal feature analysis network to obtain the spatiotemporal features of micro-expressions.

[0015] As an optional implementation, in the first aspect of the present invention, performing motion magnification on the target intermediate frame and the target vertex frame to obtain a motion-magnified frame includes:

[0016] Calculate the structural similarity between the target starting frame and the target vertex frame to obtain the target similarity index;

[0017] The difference between the maximum value of the structural similarity range and the target similarity index is calculated to obtain the target similarity difference; wherein, the magnitude of the target similarity difference characterizes the intensity of micro-expression movements;

[0018] Calculate the ratio of the preset magnification control factor to the target similarity difference, and round the ratio according to the preset number of digits to obtain the target magnification factor;

[0019] Motion magnification is performed on the target intermediate frame and the target vertex frame according to the target magnification factor to obtain a motion-magnified frame.

[0020] As an optional implementation, in the first aspect of the present invention, the spatiotemporal feature analysis network is a three-dimensional residual network that integrates a spatial attention mechanism. The spatiotemporal feature analysis network includes a primary feature extraction layer, a first max pooling layer, and at least one spatiotemporal feature analysis unit. Each spatiotemporal feature analysis unit includes a three-dimensional residual module and a second max pooling layer. The three-dimensional residual module is used to extract spatiotemporal features, and the second max pooling layer is used to reduce the data space size of the spatiotemporal features extracted by the three-dimensional residual module.

[0021] The step of performing temporal and spatial analysis on the motion magnification frame according to a preset spatiotemporal feature analysis network to obtain micro-expression spatiotemporal features includes:

[0022] The image information of the motion magnification frame is extracted through the primary feature extraction layer to obtain primary image features;

[0023] The primary image features are pooled using the first max pooling layer to obtain the target image features;

[0024] The spatiotemporal feature analysis of the target image features is performed by all the spatiotemporal feature analysis units to obtain micro-expression spatiotemporal features.

[0025] As an optional implementation, in the first aspect of the present invention, the step of dynamically analyzing the target starting frame and the target vertex frame to obtain micro-expression dynamic features includes:

[0026] Optical flow analysis is performed on the target start frame and the target vertex frame to obtain optical flow fusion features;

[0027] The optical flow fusion features are dynamically analyzed using a preset dynamic analysis network to obtain micro-expression dynamic features.

[0028] As an optional implementation, in the first aspect of the present invention, the step of performing optical flow analysis on the target starting frame and the target vertex frame to obtain optical flow fusion features includes:

[0029] Optical flow features are extracted based on the pixel difference information between the target start frame and the target vertex frame to obtain the horizontal component features and the vertical component features of optical flow.

[0030] The optical strain between the target starting frame and the target vertex frame is calculated based on the optical flow horizontal component characteristics and the optical flow vertical component characteristics to obtain the optical strain characteristics;

[0031] The optical flow horizontal component features, the optical flow vertical component features, and the optical strain features are fused to obtain optical flow fused features.

[0032] As an optional implementation, in the first aspect of the present invention, the dynamic analysis network is a multi-scale convolutional network, which includes a first-scale convolutional unit, a second-scale convolutional unit, and a third max-pooling unit.

[0033] Furthermore, the step of dynamically analyzing the optical flow fusion features according to a preset dynamic analysis network to obtain micro-expression dynamic features includes:

[0034] The optical flow fusion features are dynamically analyzed and processed by the first-scale convolutional unit to obtain the first initial dynamic features.

[0035] The optical flow fusion features are dynamically analyzed and processed by the second-scale convolutional unit to obtain the second initial dynamic features.

[0036] The first initial dynamic feature and the second initial dynamic feature are added together and fused to obtain the initial fused feature;

[0037] The first initial dynamic feature is pooled by the third max pooling unit to obtain the first target dynamic feature;

[0038] The initial fused features are pooled using the third max pooling unit to obtain the second target dynamic features;

[0039] The micro-expression dynamic features include the first target dynamic features and the second target dynamic features.

[0040] As an optional implementation, in the first aspect of the present invention, the feature fusion of the spatiotemporal features of the micro-expressions and the dynamic features of the micro-expressions to obtain micro-expression fusion features includes:

[0041] A first target transformation feature is obtained by performing a fully connected transformation on the first target dynamic feature, and a second target transformation feature is obtained by performing a fully connected transformation on the second target dynamic feature.

[0042] The dynamic transformation features are obtained by summing the first target transformation features and the second target transformation features.

[0043] The first target transformation feature and the second target transformation feature are combined and stacked, and then input into a preset long short-term memory network for feature filtering to obtain the initial filtered features;

[0044] The initial screening features are subjected to a fully connected transformation to obtain the target screening features;

[0045] Perform a fully connected transformation on the spatiotemporal features of the micro-expressions to obtain spatiotemporal transformation features;

[0046] The dynamic transformation features, the target selection features, and the spatiotemporal transformation features are weighted and summed to obtain the micro-expression fusion features.

[0047] A second aspect of the present invention discloses a micro-expression recognition device for emotional intent analysis, the device comprising:

[0048] The video frame extraction module is used to extract video frames from the micro-expression video to be detected, and obtain target video frames. The micro-expression video is configured with annotation information of the micro-expression start time and the time when the micro-expression motion intensity is the maximum. The target video frame includes a target start frame, a target middle frame, and a target vertex frame. The target start frame is the video frame corresponding to the micro-expression start time. The target vertex frame is the video frame corresponding to the time when the micro-expression motion intensity is the maximum. The target middle frame is at least one video frame corresponding to the middle time after the micro-expression start time and before the time when the micro-expression motion intensity is the maximum.

[0049] The spatiotemporal analysis module is used to perform temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain the spatiotemporal features of micro-expressions;

[0050] The dynamic analysis module is used to perform dynamic analysis on the target start frame and the target vertex frame to obtain micro-expression dynamic features;

[0051] The feature fusion module is used to fuse the spatiotemporal features of the micro-expressions and the dynamic features of the micro-expressions to obtain micro-expression fusion features;

[0052] The expression recognition module is used to identify the expression type of micro-expressions based on the micro-expression fusion features, and obtain the micro-expression recognition result.

[0053] As an optional implementation, in a second aspect of the invention, the spatiotemporal analysis module performs temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain micro-expression spatiotemporal features, including:

[0054] Perform motion magnification on the target intermediate frame and the target vertex frame to obtain a motion magnified frame;

[0055] The motion magnification frame is analyzed in time and space using a preset spatiotemporal feature analysis network to obtain the spatiotemporal features of micro-expressions.

[0056] As an optional implementation, in a second aspect of the invention, the spatiotemporal analysis module performs motion magnification operations on the target intermediate frame and the target vertex frame to obtain a motion-magnified frame, including:

[0057] Calculate the structural similarity between the target starting frame and the target vertex frame to obtain the target similarity index;

[0058] The difference between the maximum value of the structural similarity range and the target similarity index is calculated to obtain the target similarity difference; wherein, the magnitude of the target similarity difference characterizes the intensity of micro-expression movements;

[0059] Calculate the ratio of the preset magnification control factor to the target similarity difference, and round the ratio according to the preset number of digits to obtain the target magnification factor;

[0060] Motion magnification is performed on the target intermediate frame and the target vertex frame according to the target magnification factor to obtain a motion-magnified frame.

[0061] As an optional implementation, in a second aspect of the present invention, the spatiotemporal feature analysis network is a three-dimensional residual network that integrates a spatial attention mechanism. The spatiotemporal feature analysis network includes a primary feature extraction layer, a first max pooling layer, and at least one spatiotemporal feature analysis unit. Each spatiotemporal feature analysis unit includes a three-dimensional residual module and a second max pooling layer. The three-dimensional residual module is used to extract spatiotemporal features, and the second max pooling layer is used to reduce the data space size of the spatiotemporal features extracted by the three-dimensional residual module.

[0062] The spatiotemporal analysis module performs temporal and spatial analysis on the motion magnification frame according to a preset spatiotemporal feature analysis network to obtain the spatiotemporal features of micro-expressions. Specific methods for obtaining these features include:

[0063] The image information of the motion magnification frame is extracted through the primary feature extraction layer to obtain primary image features;

[0064] The primary image features are pooled using the first max pooling layer to obtain the target image features;

[0065] The spatiotemporal feature analysis of the target image features is performed by all the spatiotemporal feature analysis units to obtain micro-expression spatiotemporal features.

[0066] As an optional implementation, in a second aspect of the present invention, the dynamic analysis module performs dynamic analysis on the target starting frame and the target vertex frame to obtain micro-expression dynamic features in the following specific ways:

[0067] Optical flow analysis is performed on the target start frame and the target vertex frame to obtain optical flow fusion features;

[0068] The optical flow fusion features are dynamically analyzed using a preset dynamic analysis network to obtain micro-expression dynamic features.

[0069] As an optional implementation, in a second aspect of the present invention, the dynamic analysis module performs optical flow analysis on the target starting frame and the target vertex frame to obtain optical flow fusion features in the following specific ways:

[0070] Optical flow features are extracted based on the pixel difference information between the target start frame and the target vertex frame to obtain the horizontal component features and the vertical component features of optical flow.

[0071] The optical strain between the target starting frame and the target vertex frame is calculated based on the optical flow horizontal component characteristics and the optical flow vertical component characteristics to obtain the optical strain characteristics;

[0072] The optical flow horizontal component features, the optical flow vertical component features, and the optical strain features are fused to obtain optical flow fused features.

[0073] As an optional implementation, in a second aspect of the present invention, the dynamic analysis network is a multi-scale convolutional network, which includes a first-scale convolutional unit, a second-scale convolutional unit, and a third max-pooling unit.

[0074] Furthermore, the specific methods by which the dynamic analysis module performs dynamic analysis on the optical flow fusion features according to a preset dynamic analysis network to obtain the micro-expression dynamic features include:

[0075] The optical flow fusion features are dynamically analyzed and processed by the first-scale convolutional unit to obtain the first initial dynamic features.

[0076] The optical flow fusion features are dynamically analyzed and processed by the second-scale convolutional unit to obtain the second initial dynamic features.

[0077] The first initial dynamic feature and the second initial dynamic feature are added together and fused to obtain the initial fused feature;

[0078] The first initial dynamic feature is pooled by the third max pooling unit to obtain the first target dynamic feature;

[0079] The initial fused features are pooled using the third max pooling unit to obtain the second target dynamic features;

[0080] The micro-expression dynamic features include the first target dynamic features and the second target dynamic features.

[0081] As an optional implementation, in a second aspect of the present invention, the feature fusion module performs feature fusion on the spatiotemporal features of the micro-expression and the dynamic features of the micro-expression to obtain the fused micro-expression features in the following specific ways:

[0082] A first target transformation feature is obtained by performing a fully connected transformation on the first target dynamic feature, and a second target transformation feature is obtained by performing a fully connected transformation on the second target dynamic feature.

[0083] The dynamic transformation features are obtained by summing the first target transformation features and the second target transformation features.

[0084] The first target transformation feature and the second target transformation feature are combined and stacked, and then input into a preset long short-term memory network for feature filtering to obtain the initial filtered features;

[0085] The initial screening features are subjected to a fully connected transformation to obtain the target screening features;

[0086] Perform a fully connected transformation on the spatiotemporal features of the micro-expressions to obtain spatiotemporal transformation features;

[0087] The dynamic transformation features, the target selection features, and the spatiotemporal transformation features are weighted and summed to obtain the micro-expression fusion features.

[0088] A third aspect of the present invention discloses another micro-expression recognition device for emotional intent analysis, the device comprising:

[0089] Memory containing executable program code;

[0090] A processor coupled to the memory;

[0091] The processor calls the executable program code stored in the memory to execute the micro-expression recognition method for emotional intention analysis disclosed in the first aspect of the present invention.

[0092] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked by a processor, are used to execute the micro-expression recognition method for emotional intent analysis disclosed in the first aspect of the present invention.

[0093] Compared with the prior art, the present invention has the following beneficial effects:

[0094] In this invention, in addition to acquiring the starting frame corresponding to the beginning of the micro-expression and the vertex frame corresponding to the moment of maximum motion intensity, intermediate frames corresponding to the intermediate moments between the beginning of the micro-expression and the moment of maximum motion intensity are also acquired. Feature analysis is performed on these video frames to obtain the spatiotemporal and dynamic features of the micro-expression. These spatiotemporal and dynamic features are then fused for expression type recognition, resulting in a micro-expression recognition result. Feature extraction based on three types of video frames—starting frame, intermediate frame, and vertex frame—improves the comprehensiveness and reliability of the acquired micro-expression features. Furthermore, temporal, spatial, and dynamic analysis is performed on the corresponding types of video frames to enhance the richness of the acquired micro-expression features. This results in higher accuracy of the recognition result obtained from expression type recognition based on the fused micro-expression features, thereby improving the reliability of subsequent emotional intent analysis of the target individual. Attached Figure Description

[0095] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is a flowchart illustrating the micro-expression recognition method for emotional intent analysis disclosed in an embodiment of the present invention;

[0097] Figure 2 This is a schematic diagram of the structure of the three-dimensional residual module disclosed in the embodiments of the present invention;

[0098] Figure 3 This is a schematic diagram of the spatiotemporal feature analysis network disclosed in an embodiment of the present invention;

[0099] Figure 4 A schematic diagram of the structure of the dynamic analysis network disclosed in this embodiment of the invention;

[0100] Figure 5 This is a schematic diagram of the micro-expression recognition device for emotional intent analysis disclosed in an embodiment of the present invention;

[0101] Figure 6 This is a schematic diagram of the micro-expression recognition device for emotional intent analysis disclosed in an embodiment of the present invention. Detailed Implementation

[0102] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0103] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0104] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0105] Microexpressions are subtle and fleeting forms of human facial expression. These expressions are extremely brief (typically lasting only 40 to 200 milliseconds), yet they play a crucial role in human emotional expression and social interaction. Microexpressions often appear when individuals attempt to conceal their emotional reactions during social interactions; they are unconscious expressions of emotion. The difficulty in controlling and disguising microexpressions makes them an important basis for understanding the emotional intentions of others. Microexpression analysis has significant applications in lie detection, interrogation, depression treatment, and business negotiations. Currently, optical flow is primarily used for facial microexpression recognition and analysis. This method captures facial movement by analyzing pixel changes between the microexpression initiation frame (the video frame corresponding to the start of the microexpression) and the peak frame (the video frame corresponding to the moment of maximum microexpression intensity), thereby extracting expression features to identify the corresponding expression type.

[0106] However, in practice, it has been found that because micro-expressions are short in duration and correspond to small facial movements, few facial features can be extracted by analyzing pixel changes between the start frame and the vertex frame of a micro-expression. This makes the accuracy of micro-expression recognition low and the reliability of the analysis results of the target individual's emotional intent low.

[0107] Therefore, improving the accuracy of micro-expression recognition, thereby enhancing the reliability of the emotional intent analysis results of target individuals, is a pressing technical problem that needs to be solved.

[0108] To address the aforementioned technical problems, this invention discloses a micro-expression recognition method and apparatus for emotion intention analysis, aiming to improve the accuracy of micro-expression recognition and thus enhance the reliability of emotion intention analysis results for target individuals. Detailed descriptions follow.

[0109] Example 1

[0110] Please see Figure 1 , Figure 1 This is a flowchart illustrating the micro-expression recognition method for emotional intent analysis disclosed in an embodiment of the present invention. Figure 1 The method shown can be applied to a micro-expression recognition device, which can identify the expression type of a micro-expression video to be detected. Furthermore, this micro-expression recognition device can be integrated into a computer vision network system or exist independently of it; this embodiment of the invention does not impose limitations. Figure 1 As shown, the micro-expression recognition method for emotional intent analysis disclosed in this embodiment of the invention includes, but is not limited to, the following operations:

[0111] 101. Extract video frames from the micro-expression video to be detected to obtain the target video frames.

[0112] In this embodiment of the invention, the micro-expression video is configured with annotation information of the micro-expression start time and the time when the micro-expression motion intensity is the maximum. The target video frame includes a target start frame, a target middle frame, and a target vertex frame. The target start frame is the video frame corresponding to the micro-expression start time, the target vertex frame is the video frame corresponding to the time when the micro-expression motion intensity is the maximum, and the target middle frame is at least one video frame corresponding to the middle time after the micro-expression start time and before the time when the micro-expression motion intensity is the maximum.

[0113] Most existing micro-expression recognition schemes focus on analyzing motion information between the start frame and the vertex frame to capture the core dynamic changes of micro-expressions. However, the transient nature of micro-expressions means that other frames also contain crucial change information for expression recognition throughout the entire dynamic process from start to finish. Considering only the dynamic information between the start frame and the vertex frame ignores the important information carried by other frames in the micro-expression video sequence. The micro-expression recognition method for emotional intent analysis in this application extracts features based on three types of video frames: the start frame, intermediate frames, and vertex frames, which improves the comprehensiveness and reliability of the acquired micro-expression features.

[0114] 102. Perform temporal and spatial analysis on the target intermediate frames and target vertex frames to obtain the spatiotemporal features of micro-expressions.

[0115] In this embodiment of the invention, temporal and spatial analysis is performed on the target intermediate frame and the target vertex frame to analyze the global and local features of all video frames in the micro-expression video sequence at both temporal and spatial scales.

[0116] It should be noted that the target intermediate frames are preferably the preceding video frames adjacent to the target vertex frame. When there are multiple target intermediate frames, multiple preceding consecutive video frames adjacent to the target vertex frame can be selected. For example, seven preceding consecutive video frames of the target vertex frame can be selected as target intermediate frames, and temporal and spatial analysis can be performed on eight video frames to obtain the spatiotemporal features of micro-expressions.

[0117] 103. Perform dynamic analysis on the target's starting frame and target vertex frame to obtain micro-expression dynamic features.

[0118] In this embodiment of the invention, the subtle movements of the face are analyzed based on the pixel changes between the target start frame and the target vertex frame, thereby extracting micro-expression dynamic features.

[0119] 104. Perform feature fusion on the spatiotemporal features and dynamic features of micro-expressions to obtain micro-expression fusion features.

[0120] In this embodiment of the invention, after obtaining the spatiotemporal features and dynamic features of micro-expressions, the two are fused to obtain micro-expression fusion features containing temporal, spatial, and dynamic features, which contain richer facial expression information than the features obtained by the traditional optical flow method.

[0121] 105. Based on the micro-expression fusion features, perform expression type identification on micro-expressions to obtain micro-expression recognition results.

[0122] As can be seen, in this embodiment of the invention, in addition to obtaining the starting frame corresponding to the start time of the micro-expression and the vertex frame corresponding to the moment of maximum motion intensity, the intermediate frame corresponding to the intermediate moment between the start time of the micro-expression and the moment of maximum motion intensity is also obtained. Feature analysis is performed on these video frames to obtain the spatiotemporal features and dynamic features of the micro-expression. The spatiotemporal features and dynamic features are then fused for expression type recognition, thereby obtaining the micro-expression recognition result. Feature extraction based on the three types of video frames—start frame, intermediate frame, and vertex frame—improves the comprehensiveness and reliability of the obtained micro-expression features. Furthermore, performing temporal, spatial, and dynamic analysis on the corresponding types of video frames enhances the richness of the obtained micro-expression features, resulting in higher accuracy of the recognition result obtained from expression type recognition based on the fused micro-expression features. This improves the reliability of subsequent emotional intent analysis of the target individual.

[0123] In an optional embodiment, performing temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain the spatiotemporal features of micro-expressions may include:

[0124] Perform motion magnification on the target intermediate frame and the target vertex frame to obtain the motion magnified frame;

[0125] Based on the preset spatiotemporal feature analysis network, the motion magnification frame is analyzed in time and space to obtain the spatiotemporal features of micro-expressions.

[0126] In this embodiment of the invention, because the facial image of micro-expressions has a small range of motion and the motion region in the micro-expression video frame is not obvious, it is difficult to identify the movement of micro-expressions. Direct micro-expression feature analysis makes it difficult to correctly extract the spatiotemporal features of micro-expressions. By performing motion magnification on the target intermediate frame and the target vertex frame, the motion region in the micro-expression video frame image is highlighted, which facilitates subsequent temporal and spatial analysis to obtain the spatiotemporal features of micro-expressions.

[0127] In an optional embodiment, performing motion magnification on the target intermediate frame and the target vertex frame to obtain a motion-magnified frame may include:

[0128] Calculate the structural similarity between the target's starting frame and the target's vertex frames to obtain the target similarity index;

[0129] The difference between the maximum value of the structural similarity range and the target similarity index is calculated to obtain the target similarity difference; where the magnitude of the target similarity difference characterizes the intensity of micro-expression movements.

[0130] Calculate the ratio of the preset magnification control factor to the target similarity difference, and round the ratio according to the preset number of digits to obtain the target magnification factor;

[0131] Motion magnification is performed on the target's intermediate frame and target's vertex frame according to the target magnification factor to obtain the motion magnified frame.

[0132] In this embodiment of the invention, since the intensity of micro-expression motion varies among different individuals and different expression types, excessive magnification may lead to significant distortion of the micro-expression image, while insufficient magnification may fail to highlight the moving regions in the micro-expression video frame. Therefore, a fixed magnification is difficult to adapt to micro-expression recognition tasks in different scenarios. This invention measures the motion intensity of micro-expressions by calculating the structural similarity between the starting frame and the vertex frame, thereby adaptively selecting an appropriate magnification to enlarge the micro-expression video frame. This reduces the possibility of image distortion while appropriately highlighting the moving regions in the micro-expression video frame, thus facilitating the subsequent extraction of micro-expression spatiotemporal features.

[0133] Structural similarity is an indicator that measures the visual similarity between two images. It evaluates the loss or change in image quality from three aspects: brightness, contrast, and structure.

[0134] In another optional embodiment, the target similarity index is calculated using the following formula:

[0135]

[0136] Where ssim represents the target similarity index, a represents the image of the target's starting frame, b represents the image of the target's vertex frame, and μ a μ represents the average pixel value of image a. b This represents the average pixel value of image b. This represents the pixel variance of image a. σ represents the pixel variance of image b. ab Let c1 represent the pixel covariance between images a and b, where c1 = (k1L). 2 c2 = (k2L) 2 All are stability variables, where L is the range of pixel values ​​(in an 8-bit image, the value of L is 255), the default value of k1 is 0.01, and the default value of k2 is 0.03.

[0137] The value of ssim ranges from 0 to 1, and a higher ssim value indicates a greater similarity between the two images. The formula for calculating the target magnification is as follows:

[0138]

[0139] Where α represents the target magnification factor, λ represents the preset magnification control factor, [*] represents the rounding operation, and ssim represents the target similarity index between the target start frame and the target vertex frame.

[0140] In another optional embodiment, the spatiotemporal feature analysis network is a three-dimensional residual network that integrates a spatial attention mechanism. The spatiotemporal feature analysis network includes a primary feature extraction layer, a first max pooling layer, and at least one spatiotemporal feature analysis unit. Each spatiotemporal feature analysis unit includes a three-dimensional residual module and a second max pooling layer. The three-dimensional residual module is used to extract spatiotemporal features, and the second max pooling layer is used to reduce the data space size of the spatiotemporal features extracted by the three-dimensional residual module.

[0141] Based on a pre-defined spatiotemporal feature analysis network, temporal and spatial analysis is performed on the magnified motion frames to obtain the spatiotemporal features of micro-expressions, which may include:

[0142] The image information of the motion magnification frame is extracted through the primary feature extraction layer to obtain the primary image features;

[0143] The primary image features are pooled using the first max pooling layer to obtain the target image features;

[0144] Spatiotemporal feature analysis is performed on the target image features by all spatiotemporal feature analysis units to obtain micro-expression spatiotemporal features.

[0145] In this embodiment of the invention, primary features of micro-expression video frames are extracted through a primary feature extraction layer and a first max pooling layer to reduce the spatial size of the data. Then, spatiotemporal features are extracted through one or more spatiotemporal feature analysis units to obtain the spatiotemporal features of micro-expressions.

[0146] In another optional embodiment, the 3D residual module integrates a Spatial Attention Module (SAM). The main working principle of the Spatial Attention Module is as follows: First, average pooling and max pooling operations are performed on the input data to extract global and local features. Then, the results of the two pooling operations are concatenated according to the channel dimension. The concatenated result is then convolved through a 3D convolutional layer to fuse the global and local features together. Finally, the convolution result is converted into a value between 0 and 1 using the Sigmoid function to obtain the spatial attention weights. The spatial attention weights are then multiplied element-wise with the input data to obtain the output features of the Spatial Attention Module.

[0147] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of the three-dimensional residual module disclosed in an embodiment of the present invention. The three-dimensional residual module includes a temporal feature extraction layer, a spatial feature extraction layer, a spatiotemporal feature fusion layer, a spatial attention module, and two 1*1*1 convolutional layers. The temporal feature extraction layer is a 3*1*1 convolutional layer used to perform temporal convolution operations to extract temporal features; the spatial feature extraction layer is a 1*3*3 convolutional layer used to perform spatial convolution operations to extract spatial features; the spatiotemporal feature fusion layer is used to sum the temporal and spatial features; both 1*1*1 convolutional layers are used to adjust the data channel dimensions. The output of the three-dimensional residual module can be represented as:

[0148] h n+1 =h n +SAM(S(h n )+T(h n ))

[0149] Among them, h n h represents the input to the 3D residual module. n+1 The output of the 3D residual module is represented by SAM(), which represents the output obtained by the spatial attention module. S() represents the spatial convolution operation, and T() represents the temporal convolution operation.

[0150] Please see Figure 3 , Figure 3 This is a schematic diagram of the spatiotemporal feature analysis network disclosed in an embodiment of the present invention. The primary feature extraction layer is a 1*5*5 convolutional layer, the first max pooling layer is a 1*3*3 max pooling layer, and the second max pooling layer is a 2*2*2 max pooling layer. Figure 3 As shown, the spatiotemporal feature analysis network includes three spatiotemporal feature analysis units, with a channel adjustment layer between adjacent units. This channel adjustment layer is a 1x1x1 convolutional layer used to adjust the channel dimensions of the data. After a motion-enlarged frame is input into the network, it first passes through a 1x5x5 convolutional layer, followed by a 1x3x3 max-pooling layer to extract image features and reduce the spatial size of the data. Then, three 3D residual modules and three 2x2x2 max-pooling layers are stacked to extract spatiotemporal features, and a 1x1x1 convolutional layer is used to adjust the channel dimensions, ultimately yielding the micro-expression spatiotemporal features.

[0151] In another optional embodiment, dynamic analysis of the target start frame and target vertex frame to obtain micro-expression dynamic features may include:

[0152] Optical flow analysis is performed on the target start frame and the target vertex frame to obtain optical flow fusion features;

[0153] Dynamic analysis of optical flow fusion features is performed based on a pre-defined dynamic analysis network to obtain micro-expression dynamic features.

[0154] In this embodiment of the invention, optical flow analysis and dynamic analysis networks are combined to extract the dynamic features of micro-expressions, thereby obtaining the features of micro-expressions at more scales.

[0155] In another optional embodiment, optical flow analysis is performed on the target start frame and the target vertex frame to obtain optical flow fusion features, which may include:

[0156] Optical flow features are extracted based on the pixel difference information between the target start frame and the target vertex frame to obtain the horizontal component features and the vertical component features of optical flow.

[0157] The optical strain between the target's starting frame and the target's vertex frame is calculated based on the characteristics of the horizontal and vertical components of the optical flow, thus obtaining the optical strain characteristics.

[0158] The optical flow horizontal component features, optical flow vertical component features, and optical strain features are fused to obtain the optical flow fused features.

[0159] Specifically, for a pixel I(x,y,t) in a micro-expression video frame image sequence with horizontal coordinate x, vertical coordinate y, and time coordinate t, its optical flow horizontal component feature u and optical flow vertical component feature v satisfy the following relationship:

[0160] I x u+I y v+I t =0

[0161] Among them, I x Let I be the horizontal component of the optical flow of pixel I. y Let I be the vertical component of the optical flow of pixel I. t The optical flow time component of pixel I.

[0162] The optical flow algorithm is used to extract motion information from the image and then calculate the optical strain feature ε.

[0163]

[0164] Where, ε xx ε yy All are normal strain components, ε xy ε yx All are tangential strain components.

[0165] By fusing the horizontal component feature u, the vertical component feature v, and the optical strain feature ε, we obtain the optical flow fusion feature O = [u, v, ε] ∈ R.

[0166] In another optional embodiment, the dynamic analysis network is a multi-scale convolutional network, which includes a first-scale convolutional unit, a second-scale convolutional unit, and a third max-pooling unit.

[0167] Furthermore, by dynamically analyzing the optical flow fusion features using a pre-defined dynamic analysis network, the resulting micro-expression dynamic features may include:

[0168] The optical flow fusion features are dynamically analyzed and processed by the first-scale convolutional unit to obtain the first initial dynamic features.

[0169] The optical flow fusion features are dynamically analyzed and processed by the second-scale convolutional unit to obtain the second initial dynamic features.

[0170] The first initial dynamic feature and the second initial dynamic feature are added together and fused to obtain the initial fused feature;

[0171] The first initial dynamic feature is pooled by the third max pooling unit to obtain the first target dynamic feature;

[0172] The initial fused features are pooled using the third max pooling unit to obtain the second target dynamic features;

[0173] Among them, micro-expression dynamic features include first target dynamic features and second target dynamic features.

[0174] Please see Figure 4 , Figure 4 A schematic diagram of the structure of the dynamic analysis network disclosed in this embodiment of the invention. (See diagram below.) Figure 4 As shown, the first-scale convolutional unit includes five 5x5 convolutional layers and two 2x2 max-pooling layers; the second-scale convolutional unit includes one 5x5 convolutional layer, two 2x2 max-pooling layers, and four 3x3 convolutional layers; and the third max-pooling unit includes two 2x2 max-pooling layers.

[0175] In the initial stage, both the first-scale and second-scale convolutional units analyze and process the optical flow fusion features through a 5x5 convolutional layer. The first-scale convolutional unit then continues feature extraction using a 5x5 convolutional layer, while the second-scale convolutional unit uses a 3x3 convolutional layer, thus enabling multi-scale acquisition of micro-expression dynamic features. Furthermore, based on the principle of residual connections, the first and second initial dynamic features are added and fused to extract high-dimensional abstract features of micro-expressions. This results in the final micro-expression dynamic features containing more micro-expression information, which is beneficial for improving the accuracy of micro-expression recognition.

[0176] In another optional embodiment, feature fusion is performed on the spatiotemporal features and dynamic features of micro-expressions to obtain micro-expression fusion features, which may include:

[0177] The first target dynamic features are transformed by performing a fully connected transformation, and the second target dynamic features are transformed by performing a fully connected transformation.

[0178] The dynamic transformation features are obtained by summing the transformation features of the first target and the transformation features of the second target.

[0179] The first target transformation feature and the second target transformation feature are combined and stacked and then input into a preset long short-term memory network for feature filtering to obtain the initial filtered features;

[0180] The initial screening features are transformed using a fully connected method to obtain the target screening features;

[0181] A fully connected transformation is performed on the spatiotemporal features of micro-expressions to obtain spatiotemporal transformation features;

[0182] The micro-expression fusion features are obtained by weighted summation of dynamic transformation features, target selection features, and spatiotemporal transformation features.

[0183] In this embodiment of the invention, fully connected transformations are performed on the first target dynamic features and the second target dynamic features, respectively. The transformed features are summed to obtain dynamic transformation features. The stacked transformed features are then input into a preset long short-term memory (LSTM) network for filtering to obtain initial filtered features. The LSM network can further extract the dynamic features of micro-expressions. Fully connected transformations are performed on the initial filtered features to adjust them to the same dimension as the dynamic transformation features, resulting in target filtered features. Fully connected transformations are performed on the spatiotemporal features of micro-expressions to adjust them to the same dimension as the dynamic transformation features, resulting in spatiotemporal transformation features. Finally, the dynamic transformation features, target filtered features, and spatiotemporal transformation features are weighted and summed to obtain micro-expression fusion features containing rich micro-expression information.

[0184] Example 2

[0185] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a micro-expression recognition device for emotional intent analysis disclosed in an embodiment of the present invention. Figure 5 The micro-expression recognition device shown is used to implement the micro-expression recognition method for emotional intent analysis described in Embodiment 1. Furthermore, this micro-expression recognition device can be integrated into a computer vision network system or exist independently of the computer vision network system; this embodiment of the invention does not impose any limitations. Figure 5 As shown, the micro-expression recognition device for emotional intent analysis disclosed in this embodiment of the invention includes, but is not limited to, the following modules:

[0186] The video frame extraction module 201 is used to extract video frames from the micro-expression video to be detected to obtain target video frames. The micro-expression video is configured with annotation information of the micro-expression start time and the time when the micro-expression motion intensity is the maximum. The target video frame includes a target start frame, a target middle frame, and a target vertex frame. The target start frame is the video frame corresponding to the micro-expression start time, the target vertex frame is the video frame corresponding to the time when the micro-expression motion intensity is the maximum, and the target middle frame is at least one video frame corresponding to the middle time after the micro-expression start time and before the time when the micro-expression motion intensity is the maximum.

[0187] The spatiotemporal analysis module 202 is used to perform temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain the spatiotemporal features of micro-expressions;

[0188] The dynamic analysis module 203 is used to perform dynamic analysis on the target start frame and the target vertex frame to obtain the dynamic features of micro-expressions;

[0189] The feature fusion module 204 is used to fuse the spatiotemporal features and dynamic features of micro-expressions to obtain micro-expression fused features.

[0190] The expression recognition module 205 is used to identify the expression type of micro-expressions based on the micro-expression fusion features and obtain the micro-expression recognition result.

[0191] In existing micro-expression recognition schemes, most efforts focus on analyzing motion information between the start frame and the vertex frame to capture the core dynamic changes of micro-expressions. However, the transient nature of micro-expressions means that other frames also contain crucial change information for expression recognition throughout the entire dynamic process from start to finish. Considering only the dynamic information between the start frame and the vertex frame ignores the important information carried by other frames in the micro-expression video sequence. The micro-expression recognition device of this application extracts features based on three types of video frames: the start frame, the intermediate frame, and the vertex frame, which improves the comprehensiveness and reliability of the acquired micro-expression features. Temporal and spatial analysis is performed on the target video frame to analyze the global and local features of all video frames in the micro-expression video sequence at both temporal and spatial scales. The subtle facial movements are analyzed based on the pixel changes between the target start frame and the target vertex frame to extract the micro-expression dynamic features. After obtaining the spatiotemporal and dynamic features of the micro-expression, the two are fused to obtain a micro-expression fusion feature containing temporal, spatial, and dynamic features, which contains richer facial expression information than the features obtained by the traditional optical flow method.

[0192] As can be seen, in this embodiment of the invention, in addition to obtaining the starting frame corresponding to the start time of the micro-expression and the vertex frame corresponding to the moment of maximum motion intensity, the intermediate frame corresponding to the intermediate moment between the start time of the micro-expression and the moment of maximum motion intensity is also obtained. Feature analysis is performed on these video frames to obtain the spatiotemporal features and dynamic features of the micro-expression. The spatiotemporal features and dynamic features are then fused for expression type recognition, thereby obtaining the micro-expression recognition result. Feature extraction based on the three types of video frames—start frame, intermediate frame, and vertex frame—improves the comprehensiveness and reliability of the obtained micro-expression features. Furthermore, performing temporal, spatial, and dynamic analysis on the corresponding types of video frames enhances the richness of the obtained micro-expression features, resulting in higher accuracy of the recognition result obtained from expression type recognition based on the fused micro-expression features. This improves the reliability of subsequent emotional intent analysis of the target individual.

[0193] In an optional embodiment, the spatiotemporal analysis module 202 performs temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain the spatiotemporal features of micro-expressions. The specific methods include:

[0194] Perform motion magnification on the target intermediate frame and the target vertex frame to obtain the motion magnified frame;

[0195] Based on the preset spatiotemporal feature analysis network, the motion magnification frame is analyzed in time and space to obtain the spatiotemporal features of micro-expressions.

[0196] In this embodiment of the invention, because the facial image of micro-expressions has a small range of motion and the motion region in the micro-expression video frame is not obvious, it is difficult to identify the movement of micro-expressions. Direct micro-expression feature analysis makes it difficult to correctly extract the spatiotemporal features of micro-expressions. By performing motion magnification on the target video frame, the motion region in the micro-expression video frame image is highlighted, which facilitates subsequent temporal and spatial analysis to obtain the spatiotemporal features of micro-expressions.

[0197] In an optional embodiment, the spatiotemporal analysis module 202 performs motion magnification operations on the target intermediate frame and the target vertex frame to obtain the motion magnified frame. The specific methods include:

[0198] Calculate the structural similarity between the target's starting frame and the target's vertex frames to obtain the target similarity index;

[0199] The difference between the maximum value of the structural similarity range and the target similarity index is calculated to obtain the target similarity difference; where the magnitude of the target similarity difference characterizes the intensity of micro-expression movements.

[0200] Calculate the ratio of the preset magnification control factor to the target similarity difference, and round the ratio according to the preset number of digits to obtain the target magnification factor;

[0201] Motion magnification is performed on the target's intermediate frame and target's vertex frame according to the target magnification factor to obtain the motion magnified frame.

[0202] In this embodiment of the invention, since the intensity of micro-expression motion varies among different individuals and different expression types, excessive magnification may lead to significant distortion of the micro-expression image, while insufficient magnification may fail to highlight the moving regions in the micro-expression video frame. Therefore, a fixed magnification is difficult to adapt to micro-expression recognition tasks in different scenarios. This invention measures the motion intensity of micro-expressions by calculating the structural similarity between the starting frame and the vertex frame, thereby adaptively selecting an appropriate magnification to enlarge the micro-expression video frame. This reduces the possibility of image distortion while appropriately highlighting the moving regions in the micro-expression video frame, thus facilitating the subsequent extraction of micro-expression spatiotemporal features.

[0203] Structural similarity is an indicator that measures the visual similarity between two images. It evaluates the loss or change in image quality from three aspects: brightness, contrast, and structure.

[0204] In another optional embodiment, the target similarity index is calculated using the following formula:

[0205]

[0206] Where ssim represents the target similarity index, a represents the image of the target's starting frame, b represents the image of the target's vertex frame, and μ a μ represents the average pixel value of image a. b This represents the average pixel value of image b. This represents the pixel variance of image a. σ represents the pixel variance of image b. ab Let c1 represent the pixel covariance between images a and b, where c1 = (k1L). 2 c2 = (k2L) 2 All are stability variables, where L is the range of pixel values ​​(in an 8-bit image, the value of L is 255), the default value of k1 is 0.01, and the default value of k2 is 0.03.

[0207] The value of ssim ranges from 0 to 1, and a higher ssim value indicates a greater similarity between the two images. The formula for calculating the target magnification is as follows:

[0208]

[0209] Where α represents the target magnification factor, λ represents the preset magnification control factor, [*] represents the rounding operation, and ssim represents the target similarity index between the target start frame and the target vertex frame.

[0210] In another optional embodiment, the spatiotemporal feature analysis network is a three-dimensional residual network that integrates a spatial attention mechanism. The spatiotemporal feature analysis network includes a primary feature extraction layer, a first max pooling layer, and at least one spatiotemporal feature analysis unit. Each spatiotemporal feature analysis unit includes a three-dimensional residual module and a second max pooling layer. The three-dimensional residual module is used to extract spatiotemporal features, and the second max pooling layer is used to reduce the data space size of the spatiotemporal features extracted by the three-dimensional residual module.

[0211] The spatiotemporal analysis module 202 performs temporal and spatial analysis on the motion magnification frame according to the preset spatiotemporal feature analysis network, and obtains the specific methods of micro-expression spatiotemporal features, including:

[0212] The image information of the motion magnification frame is extracted through the primary feature extraction layer to obtain the primary image features;

[0213] The primary image features are pooled using the first max pooling layer to obtain the target image features;

[0214] Spatiotemporal feature analysis is performed on the target image features by all spatiotemporal feature analysis units to obtain micro-expression spatiotemporal features.

[0215] In this embodiment of the invention, primary features of micro-expression video frames are extracted through a primary feature extraction layer and a first max pooling layer to reduce the spatial size of the data. Then, spatiotemporal features are extracted through one or more spatiotemporal feature analysis units to obtain the spatiotemporal features of micro-expressions.

[0216] In another optional embodiment, the 3D residual module integrates a Spatial Attention Module (SAM). The main working principle of the Spatial Attention Module is as follows: First, average pooling and max pooling operations are performed on the input data to extract global and local features. Then, the results of the two pooling operations are concatenated according to the channel dimension. The concatenated result is then convolved through a 3D convolutional layer to fuse the global and local features together. Finally, the convolution result is converted into a value between 0 and 1 using the Sigmoid function to obtain the spatial attention weights. The spatial attention weights are then multiplied element-wise with the input data to obtain the output features of the Spatial Attention Module.

[0217] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of the three-dimensional residual module disclosed in an embodiment of the present invention. The three-dimensional residual module includes a temporal feature extraction layer, a spatial feature extraction layer, a spatiotemporal feature fusion layer, a spatial attention module, and two 1*1*1 convolutional layers. The temporal feature extraction layer is a 3*1*1 convolutional layer used to perform temporal convolution operations to extract temporal features; the spatial feature extraction layer is a 1*3*3 convolutional layer used to perform spatial convolution operations to extract spatial features; the spatiotemporal feature fusion layer is used to sum the temporal and spatial features; both 1*1*1 convolutional layers are used to adjust the data channel dimensions. The output of the three-dimensional residual module can be represented as:

[0218] h n+1 =h n +SAM(S(h n )+T(h n ))

[0219] Among them, h n h represents the input to the 3D residual module. n+1 The output of the 3D residual module is represented by SAM(), which represents the output obtained by the spatial attention module. S() represents the spatial convolution operation, and T() represents the temporal convolution operation.

[0220] Please see Figure 3 , Figure 3This is a schematic diagram of the spatiotemporal feature analysis network disclosed in an embodiment of the present invention. The primary feature extraction layer is a 1*5*5 convolutional layer, the first max pooling layer is a 1*3*3 max pooling layer, and the second max pooling layer is a 2*2*2 max pooling layer. Figure 3 As shown, the spatiotemporal feature analysis network includes three spatiotemporal feature analysis units, with a channel adjustment layer between adjacent units. This channel adjustment layer is a 1x1x1 convolutional layer used to adjust the channel dimensions of the data. After a motion-enlarged frame is input into the network, it first passes through a 1x5x5 convolutional layer, followed by a 1x3x3 max-pooling layer to extract image features and reduce the spatial size of the data. Then, three 3D residual modules and three 2x2x2 max-pooling layers are stacked to extract spatiotemporal features, and a 1x1x1 convolutional layer is used to adjust the channel dimensions, ultimately yielding the micro-expression spatiotemporal features.

[0221] In another optional embodiment, the dynamic analysis module 203 performs dynamic analysis on the target start frame and the target vertex frame to obtain the micro-expression dynamic features in the following specific ways:

[0222] Optical flow analysis is performed on the target start frame and the target vertex frame to obtain optical flow fusion features;

[0223] Dynamic analysis of optical flow fusion features is performed based on a pre-defined dynamic analysis network to obtain micro-expression dynamic features.

[0224] As an optional implementation, in the second aspect of the present invention, the dynamic analysis module 203 performs optical flow analysis on the target start frame and the target vertex frame to obtain optical flow fusion features in the following specific ways:

[0225] Optical flow features are extracted based on the pixel difference information between the target start frame and the target vertex frame to obtain the horizontal component features and the vertical component features of optical flow.

[0226] The optical strain between the target's starting frame and the target's vertex frame is calculated based on the characteristics of the horizontal and vertical components of the optical flow, thus obtaining the optical strain characteristics.

[0227] The optical flow horizontal component features, optical flow vertical component features, and optical strain features are fused to obtain the optical flow fused features.

[0228] Specifically, for a pixel I(x,y,t) in a micro-expression video frame image sequence with horizontal coordinate x, vertical coordinate y, and time coordinate t, its optical flow horizontal component feature u and optical flow vertical component feature v satisfy the following relationship:

[0229] I x u+Iy v+I t =0

[0230] Among them, I x Let I be the horizontal component of the optical flow of pixel I. y Let I be the vertical component of the optical flow of pixel I. t The optical flow time component of pixel I.

[0231] The optical flow algorithm is used to extract motion information from the image and then calculate the optical strain feature ε.

[0232]

[0233] Where, ε xx ε yy All are normal strain components, ε xy ε yx All are tangential strain components.

[0234] By fusing the horizontal component feature u, the vertical component feature v, and the optical strain feature ε, we obtain the optical flow fusion feature O = [u, v, ε] ∈ R.

[0235] In another optional embodiment, the dynamic analysis network is a multi-scale convolutional network, which includes a first-scale convolutional unit, a second-scale convolutional unit, and a third max-pooling unit.

[0236] Furthermore, the dynamic analysis module 203 performs dynamic analysis on the optical flow fusion features according to a preset dynamic analysis network to obtain the specific methods of micro-expression dynamic features, including:

[0237] The optical flow fusion features are dynamically analyzed and processed by the first-scale convolutional unit to obtain the first initial dynamic features.

[0238] The optical flow fusion features are dynamically analyzed and processed by the second-scale convolutional unit to obtain the second initial dynamic features.

[0239] The first initial dynamic feature and the second initial dynamic feature are added together and fused to obtain the initial fused feature;

[0240] The first initial dynamic feature is pooled by the third max pooling unit to obtain the first target dynamic feature;

[0241] The initial fused features are pooled using the third max pooling unit to obtain the second target dynamic features;

[0242] Among them, micro-expression dynamic features include first target dynamic features and second target dynamic features.

[0243] Please see Figure 4 , Figure 4 A schematic diagram of the structure of the dynamic analysis network disclosed in this embodiment of the invention. (See diagram below.) Figure 4 As shown, the first-scale convolutional unit includes five 5x5 convolutional layers and two 2x2 max-pooling layers; the second-scale convolutional unit includes one 5x5 convolutional layer, two 2x2 max-pooling layers, and four 3x3 convolutional layers; and the third max-pooling unit includes two 2x2 max-pooling layers.

[0244] In the initial stage, both the first-scale and second-scale convolutional units analyze and process the optical flow fusion features through a 5x5 convolutional layer. The first-scale convolutional unit then continues feature extraction using a 5x5 convolutional layer, while the second-scale convolutional unit uses a 3x3 convolutional layer, thus enabling multi-scale acquisition of micro-expression dynamic features. Furthermore, based on the principle of residual connections, the first and second initial dynamic features are added and fused to extract high-dimensional abstract features of micro-expressions. This results in the final micro-expression dynamic features containing more micro-expression information, which is beneficial for improving the accuracy of micro-expression recognition.

[0245] In another optional embodiment, the feature fusion module 204 performs feature fusion on the spatiotemporal features and dynamic features of micro-expressions to obtain the fused micro-expression features. The specific methods include:

[0246] The first target dynamic features are transformed by performing a fully connected transformation, and the second target dynamic features are transformed by performing a fully connected transformation.

[0247] The dynamic transformation features are obtained by summing the transformation features of the first target and the transformation features of the second target.

[0248] The first target transformation feature and the second target transformation feature are combined and stacked and then input into a preset long short-term memory network for feature filtering to obtain the initial filtered features;

[0249] The initial screening features are transformed using a fully connected method to obtain the target screening features;

[0250] A fully connected transformation is performed on the spatiotemporal features of micro-expressions to obtain spatiotemporal transformation features;

[0251] The micro-expression fusion features are obtained by weighted summation of dynamic transformation features, target selection features, and spatiotemporal transformation features.

[0252] In this embodiment of the invention, fully connected transformations are performed on the first target dynamic features and the second target dynamic features, respectively. The transformed features are summed to obtain dynamic transformation features. The stacked transformed features are then input into a preset long short-term memory (LSTM) network for filtering to obtain initial filtered features. The LSM network can further extract the dynamic features of micro-expressions. Fully connected transformations are performed on the initial filtered features to adjust them to the same dimension as the dynamic transformation features, resulting in target filtered features. Fully connected transformations are performed on the spatiotemporal features of micro-expressions to adjust them to the same dimension as the dynamic transformation features, resulting in spatiotemporal transformation features. Finally, the dynamic transformation features, target filtered features, and spatiotemporal transformation features are weighted and summed to obtain micro-expression fusion features containing rich micro-expression information.

[0253] Example 3

[0254] Please see Figure 6 , Figure 6 This is a schematic diagram of another micro-expression recognition device for emotional intent analysis disclosed in an embodiment of the present invention. Figure 6 The micro-expression recognition device shown is used to implement the micro-expression recognition method for emotional intent analysis described in Embodiment 1. Furthermore, this micro-expression recognition device can be integrated into a computer vision network system or exist independently of the computer vision network system; this embodiment of the invention does not impose any limitations. Figure 6 As shown, another micro-expression recognition device for emotional intent analysis disclosed in this embodiment of the invention includes, but is not limited to, the following modules:

[0255] Memory 301 storing executable program code;

[0256] Processor 302 coupled to memory 301;

[0257] The processor 302 calls the executable program code stored in the memory 301 to execute some or all of the steps in the micro-expression recognition method for emotional intent analysis described in Embodiment 1 of the present invention.

[0258] Example 4

[0259] This invention discloses a computer storage medium storing computer instructions. When the computer instructions are invoked by a processor, they are used to execute some or all of the steps in the micro-expression recognition method for emotional intent analysis described in Embodiment 1 of this invention.

[0260] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0261] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0262] Finally, it should be noted that the micro-expression recognition method and apparatus for emotional intent analysis disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A micro-expression recognition method applied to emotional intent analysis, characterized in that, The method includes: Video frames are extracted from the micro-expression video to be detected to obtain target video frames. The micro-expression video is equipped with annotation information for the start time of the micro-expression and the time when the micro-expression motion intensity is the maximum. The target video frame includes a target start frame, a target middle frame, and a target vertex frame. The target start frame is the video frame corresponding to the start time of the micro-expression. The target vertex frame is the video frame corresponding to the time when the micro-expression motion intensity is the maximum. The target middle frame is at least one video frame corresponding to the middle time after the start time of the micro-expression and before the time when the micro-expression motion intensity is the maximum. Temporal and spatial analysis is performed on the target intermediate frame and the target vertex frame to obtain the spatiotemporal features of micro-expressions; Dynamic analysis is performed on the target start frame and the target vertex frame to obtain micro-expression dynamic features; The spatiotemporal features and dynamic features of micro-expressions are fused to obtain micro-expression fusion features; Based on the micro-expression fusion features, the micro-expression type is identified to obtain the micro-expression recognition result; The step of performing temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain the spatiotemporal features of micro-expressions includes: Perform motion magnification on the target intermediate frame and the target vertex frame to obtain a motion magnified frame; The motion magnification frame is analyzed in time and space according to a preset spatiotemporal feature analysis network to obtain the spatiotemporal features of micro-expressions; The dynamic analysis of the target starting frame and the target vertex frame to obtain micro-expression dynamic features includes: Optical flow analysis is performed on the target start frame and the target vertex frame to obtain optical flow fusion features; The optical flow fusion features are dynamically analyzed based on a preset dynamic analysis network to obtain micro-expression dynamic features; The dynamic analysis network is a multi-scale convolutional network, which includes a first-scale convolutional unit, a second-scale convolutional unit, and a third max-pooling unit. Furthermore, the step of dynamically analyzing the optical flow fusion features according to a preset dynamic analysis network to obtain micro-expression dynamic features includes: The optical flow fusion features are dynamically analyzed and processed by the first-scale convolutional unit to obtain the first initial dynamic features. The optical flow fusion features are dynamically analyzed and processed by the second-scale convolutional unit to obtain the second initial dynamic features. The first initial dynamic feature and the second initial dynamic feature are added together and fused to obtain the initial fused feature; The first initial dynamic feature is pooled by the third max pooling unit to obtain the first target dynamic feature; The initial fused features are pooled using the third max pooling unit to obtain the second target dynamic features; The micro-expression dynamic features include the first target dynamic features and the second target dynamic features.

2. The micro-expression recognition method for emotional intent analysis according to claim 1, characterized in that, The step of performing motion magnification on the target intermediate frame and the target vertex frame to obtain a motion magnified frame includes: Calculate the structural similarity between the target starting frame and the target vertex frame to obtain the target similarity index; The difference between the maximum value of the structural similarity range and the target similarity index is calculated to obtain the target similarity difference; wherein, the magnitude of the target similarity difference characterizes the intensity of micro-expression movements; Calculate the ratio of the preset magnification control factor to the target similarity difference, and round the ratio according to the preset number of digits to obtain the target magnification factor; Motion magnification is performed on the target intermediate frame and the target vertex frame according to the target magnification factor to obtain a motion-magnified frame.

3. The micro-expression recognition method for emotional intent analysis according to claim 1, characterized in that, The spatiotemporal feature analysis network is a three-dimensional residual network that integrates a spatial attention mechanism. The spatiotemporal feature analysis network includes a primary feature extraction layer, a first max pooling layer, and at least one spatiotemporal feature analysis unit. Each spatiotemporal feature analysis unit includes a three-dimensional residual module and a second max pooling layer. The three-dimensional residual module is used to extract spatiotemporal features, and the second max pooling layer is used to reduce the data space size of the spatiotemporal features extracted by the three-dimensional residual module. The step of performing temporal and spatial analysis on the motion magnification frame according to a preset spatiotemporal feature analysis network to obtain micro-expression spatiotemporal features includes: The image information of the motion magnification frame is extracted through the primary feature extraction layer to obtain primary image features; The primary image features are pooled using the first max pooling layer to obtain the target image features; The spatiotemporal feature analysis of the target image features is performed by all the spatiotemporal feature analysis units to obtain micro-expression spatiotemporal features.

4. The micro-expression recognition method for emotional intent analysis according to any one of claims 1 to 3, characterized in that, The optical flow analysis of the target starting frame and the target vertex frame to obtain optical flow fusion features includes: Optical flow features are extracted based on the pixel difference information between the target start frame and the target vertex frame to obtain the horizontal component features and the vertical component features of optical flow. The optical strain between the target starting frame and the target vertex frame is calculated based on the optical flow horizontal component characteristics and the optical flow vertical component characteristics to obtain the optical strain characteristics; The optical flow horizontal component features, the optical flow vertical component features, and the optical strain features are fused to obtain optical flow fused features.

5. The micro-expression recognition method for emotional intent analysis according to any one of claims 1 to 3, characterized in that, The process of fusing the spatiotemporal features and dynamic features of micro-expressions to obtain micro-expression fusion features includes: A first target transformation feature is obtained by performing a fully connected transformation on the first target dynamic feature, and a second target transformation feature is obtained by performing a fully connected transformation on the second target dynamic feature. The dynamic transformation features are obtained by summing the first target transformation features and the second target transformation features. The first target transformation feature and the second target transformation feature are combined and stacked, and then input into a preset long short-term memory network for feature filtering to obtain the initial filtered features; The initial screening features are subjected to a fully connected transformation to obtain the target screening features; Perform a fully connected transformation on the spatiotemporal features of the micro-expressions to obtain spatiotemporal transformation features; The dynamic transformation features, the target selection features, and the spatiotemporal transformation features are weighted and summed to obtain the micro-expression fusion features.

6. A micro-expression recognition device for emotional intent analysis, characterized in that, The device includes: The video frame extraction module is used to extract video frames from the micro-expression video to be detected, and obtain target video frames. The micro-expression video is configured with annotation information of the micro-expression start time and the time when the micro-expression motion intensity is the maximum. The target video frame includes a target start frame, a target middle frame, and a target vertex frame. The target start frame is the video frame corresponding to the micro-expression start time. The target vertex frame is the video frame corresponding to the time when the micro-expression motion intensity is the maximum. The target middle frame is at least one video frame corresponding to the middle time after the micro-expression start time and before the time when the micro-expression motion intensity is the maximum. The spatiotemporal analysis module is used to perform temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain the spatiotemporal features of micro-expressions; The dynamic analysis module is used to perform dynamic analysis on the target start frame and the target vertex frame to obtain micro-expression dynamic features; The feature fusion module is used to fuse the spatiotemporal features of the micro-expressions and the dynamic features of the micro-expressions to obtain micro-expression fusion features; The expression recognition module is used to identify the expression type of micro-expressions based on the micro-expression fusion features, and obtain the micro-expression recognition result; The spatiotemporal analysis module performs temporal and spatial analysis on the target intermediate frame and the target vertex frame to obtain micro-expression spatiotemporal features, including: Perform motion magnification on the target intermediate frame and the target vertex frame to obtain a motion magnified frame; The motion magnification frame is analyzed in time and space according to a preset spatiotemporal feature analysis network to obtain the spatiotemporal features of micro-expressions; The dynamic analysis module performs dynamic analysis on the target start frame and the target vertex frame to obtain the dynamic features of micro-expressions. Specific methods for obtaining these features include: Optical flow analysis is performed on the target start frame and the target vertex frame to obtain optical flow fusion features; The optical flow fusion features are dynamically analyzed based on a preset dynamic analysis network to obtain micro-expression dynamic features; The dynamic analysis network is a multi-scale convolutional network, which includes a first-scale convolutional unit, a second-scale convolutional unit, and a third max-pooling unit. Furthermore, the dynamic analysis module performs dynamic analysis on the optical flow fusion features according to a preset dynamic analysis network to obtain the micro-expression dynamic features in the following specific ways: The optical flow fusion features are dynamically analyzed and processed by the first-scale convolutional unit to obtain the first initial dynamic features. The optical flow fusion features are dynamically analyzed and processed by the second-scale convolutional unit to obtain the second initial dynamic features. The first initial dynamic feature and the second initial dynamic feature are added together and fused to obtain the initial fused feature; The first initial dynamic feature is pooled by the third max pooling unit to obtain the first target dynamic feature; The initial fused features are pooled using the third max pooling unit to obtain the second target dynamic features; The micro-expression dynamic features include the first target dynamic features and the second target dynamic features.

7. A micro-expression recognition device for emotional intent analysis, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the micro-expression recognition method for emotional intention analysis as described in any one of claims 1 to 5.

Images