A micro-expression recognition method and device, electronic equipment and storage medium

Abstract

The application provides a micro-expression recognition method and device, electronic equipment and storage medium, and relates to the field of micro-expression recognition. The method comprises the following steps: acquiring a video frame sequence and event stream data obtained by shooting a face target within a preset time period; the event stream data is collected by a dynamic visual sensor; performing feature extraction on the video frame sequence to obtain video frame sequence features; performing feature extraction on the event stream data to obtain event stream data features; fusing the video frame sequence features and the event stream data features to obtain multi-modal fusion features; performing micro-expression recognition by using the multi-modal fusion features to obtain a micro-expression category of the face target; the event stream data with extremely high time resolution can effectively supplement the video frame sequence, and the video frame sequence features of the video frame sequence and the event stream data features of the event stream data can be fused for micro-expression recognition, so that the recognition accuracy of the micro-expression of the face target can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of micro-expression recognition, and particularly relates to a micro-expression recognition method and device, electronic equipment and a storage medium. BACKGROUND

[0002] A micro-expression refers to a facial expression with short change time and small change amplitude. In related technologies, due to the low frame rate of an image acquisition device, it is difficult to effectively capture the subtle changes of a micro-expression, and thus it is impossible to use images acquired by the image acquisition device to recognize the type of the micro-expression, which brings difficulties to the recognition of the type of the micro-expression. SUMMARY

[0003] The present application aims to provide a micro-expression recognition method, device, electronic equipment and storage medium, which can effectively supplement a video frame sequence by using event stream data with extremely high time resolution, and can recognize a micro-expression by fusing video frame sequence features of the video frame sequence and event stream data of the event stream data, so as to improve the recognition accuracy of a facial target micro-expression.

[0004] To solve the above technical problems, the present application provides a micro-expression recognition method, comprising:

[0005] acquiring a video frame sequence and event stream data obtained by shooting a facial target in a preset time period, wherein the event stream data is acquired by a dynamic visual sensor;

[0006] extracting features of the video frame sequence to obtain video frame sequence features;

[0007] extracting features of the event stream data to obtain event stream data features;

[0008] fusing the video frame sequence features and the event stream data features to obtain multi-modal fusion features;

[0009] recognizing a micro-expression by using the multi-modal fusion features to obtain a micro-expression category of the facial target.

[0010] Optionally, the step of extracting features of the video frame sequence to obtain video frame sequence features comprises:

[0011] extracting picture features of each image in the video frame sequence to obtain a picture feature sequence;

[0012] extracting facial features of each image in the video frame sequence to obtain a facial feature sequence;

[0013] The picture feature sequence is processed by using a transformer model to perform multi-head attention mechanism processing, micro-expression enhancement processing, and multi-layer perception processing, to obtain the video frame sequence features; the micro-expression enhancement processing is attention mechanism processing performed on the picture features sequence after multi-head attention mechanism processing, by using the human face feature sequence.

[0014] Optionally, the human face feature extraction on each image in the video frame sequence comprises:

[0015] The human face target detection is performed on each image in the video frame sequence, to obtain a human face region in the image.

[0016] The human face feature extraction is performed on the human face region in each image, to obtain a human face feature sequence corresponding to the video frame sequence.

[0017] Optionally, before the human face feature extraction on the human face region in each image, the method further comprises:

[0018] The key points are used to mark the positions of the five facial features in the human face region.

[0019] The human face region is scaled according to the positions of the five facial features and a preset relative position relationship between the five facial features and the human face.

[0020] Correspondingly, the human face feature extraction on the human face region in each image comprises:

[0021] The human face feature extraction is performed on the scaled human face region.

[0022] Optionally, after the human face feature extraction on the human face region in each image, the method further comprises:

[0023] When the image contains multiple human face regions, the human face features of the multiple human face regions are averaged and fused.

[0024] Optionally, the human face feature extraction on each image in the video frame sequence comprises:

[0025] The human face feature extraction is performed on each human face region in the image by using a deep residual network, to obtain a human face feature sequence corresponding to the video frame sequence.

[0026] Optionally, the converter model is composed of converter units, and each converter unit is composed of a first residual structure, a second residual structure and a third residual structure in series; the first residual structure is a residual structure of multi-head attention, the second residual structure is a residual structure of a micro-expression enhancement structure, and the third residual structure is a residual structure of a multi-layer perception; the micro-expression enhancement structure is configured to take an output vector sequence of the first residual structure as a query vector sequence, map a human face feature in the human face feature sequence to a key vector sequence and a value vector sequence through two full connection layers with different weights, and process the key vector sequence, the value vector sequence and the query vector sequence through an attention mechanism to obtain an enhanced vector sequence to be input to the third residual structure.

[0027] Optionally, the converter model is composed of a preset number of converter units in series.

[0028] Optionally, the picture feature extraction on each image in the video frame sequence to obtain the picture feature sequence comprises:

[0029] The picture feature extraction on each image in the video frame sequence is performed by using an effective network model to obtain the picture feature sequence.

[0030] Optionally, the feature extraction on the event stream data to obtain the event stream data feature comprises:

[0031] The first absolute time in the event stream data is used to adjust all absolute times in the event stream data to relative times.

[0032] The adjusted event stream data is converted into point cloud data; the coordinates of each point in the point cloud data are composed of a pixel point coordinate and a relative time corresponding to the point, and the feature of the point is the polarity of the event stream data corresponding to the point.

[0033] The feature extraction on the point cloud data to obtain the event stream data feature.

[0034] Optionally, before the feature extraction on the point cloud data to obtain the event stream data feature, the method further comprises:

[0035] For each point in the point cloud data, it is determined whether other points in a neighborhood range of the point have the polarity;

[0036] If not, the point is subjected to noise reduction processing;

[0037] Correspondingly, the feature extraction on the point cloud data to obtain the event stream data feature comprises:

[0038] The point cloud data after the noise reduction processing is used for feature extraction to obtain the event stream data features.

[0039] Optionally, the feature extraction on the point cloud data to obtain the event stream data features comprises:

[0040] The point cloud data is encoded to obtain point cloud encoding data;

[0041] The point cloud encoding data is subjected to time series compression processing and structure keeping processing by using an input space keeping structure to obtain first output data;

[0042] The first output data is subjected to feature extraction to obtain first feature data;

[0043] The first feature data is subjected to time series compression processing and structure keeping processing by using a feature space keeping structure to obtain second output data;

[0044] The second output data is subjected to feature extraction to obtain the event stream data features.

[0045] Optionally, the time series compression processing and structure keeping processing on the point cloud encoding data by using the input space keeping structure to obtain the first output data comprises:

[0046] The point cloud encoding data is subjected to dimension transformation to obtain dimension-transformed input data;

[0047] The input data is subjected to multiple convolution processing to obtain a first feature map;

[0048] The first feature map is subjected to time series compression processing to obtain a second feature map;

[0049] The second feature map is subjected to dimension reduction processing by multiple fully connected layers to obtain a third feature map;

[0050] The third feature map is subjected to matrix addition with a same-size diagonal matrix, and the addition result is subjected to deformation processing to obtain a transformation matrix;

[0051] The transformation matrix is subjected to batch matrix multiplication processing with the dimension-transformed input data to obtain the first output data.

[0052] Optionally, the feature extraction on the first output data to obtain the first feature data comprises:

[0053] The first output data is subjected to dimension transformation to obtain a fourth feature map;

[0054] The fourth feature map is subjected to convolution processing by using a one-dimensional convolution kernel to obtain the first feature data.

[0055] Optionally, the video frame sequence and the event stream data are collected by a dynamic active vision sensor.

[0056] Optionally, the video frame sequence and the event stream data photographed by the face target in a preset time period are obtained, comprising:

[0057] The video frame sequence collected by the image acquisition device is obtained.

[0058] The event stream data collected by the dynamic vision sensor is obtained.

[0059] The video frame sequence and the event stream data are time-aligned and space-aligned.

[0060] Optionally, the micro-expression recognition is performed by using the multi-modal fusion feature to obtain a micro-expression category of the face target, comprising:

[0061] The multi-modal fusion feature is reduced in dimension by using a full connection layer to obtain a reduced dimension feature.

[0062] The reduced dimension feature is normalized and exponentiated to obtain a probability value of the face target belonging to each micro-expression category.

[0063] The micro-expression category corresponding to the face target is determined according to the probability value.

[0064] Optionally, the video frame sequence feature and the event stream data feature are fused to obtain a multi-modal fusion feature, comprising:

[0065] A first pass coefficient is generated by using the event stream data feature, and the video frame sequence feature is weighted and scaled by using the first pass coefficient to obtain a video frame fusion feature.

[0066] A second pass coefficient is generated by using the video frame sequence feature, and the event stream data feature is weighted and scaled by using the second pass coefficient to obtain an event stream fusion feature.

[0067] The video frame fusion feature and the event stream fusion feature are spliced to obtain the multi-modal fusion feature.

[0068] Optionally, the first pass coefficient is generated by using the event stream data feature, comprising:

[0069] The event stream data feature is one-dimensionally convoluted to obtain a mapping feature.

[0070] The mapping feature is maximum-pooled to obtain a maximum-pooled feature.

[0071] The maximum pooling feature is processed by using an S-shaped function to obtain the first passing coefficient.

[0072] The application further provides a micro-expression recognition device, comprising:

[0073] An acquisition module is configured to acquire a video frame sequence and event stream data obtained by photographing a face target within a preset time period, wherein the event stream data is collected by a dynamic visual sensor;

[0074] A video frame sequence processing module is configured to perform feature extraction on the video frame sequence to obtain video frame sequence features.

[0075] An event stream data processing module is configured to perform feature extraction on the event stream data to obtain event stream data features.

[0076] A feature fusion module is configured to fuse the video frame sequence features and the event stream data features to obtain multi-modal fusion features.

[0077] An output module is configured to perform micro-expression recognition by using the multi-modal fusion features to obtain a micro-expression category of the face target.

[0078] The application further provides an electronic device, comprising:

[0079] A memory is configured to store a computer program.

[0080] A processor is configured to implement the micro-expression recognition method as described above when executing the computer program.

[0081] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores computer executable instructions, and the computer executable instructions are loaded and executed by a processor to implement the micro-expression recognition method as described above.

[0082] The application provides a micro-expression recognition method, comprising: acquiring a video frame sequence and event stream data obtained by photographing a face target within a preset time period, wherein the event stream data is collected by a dynamic visual sensor; performing feature extraction on the video frame sequence to obtain video frame sequence features; performing feature extraction on the event stream data to obtain event stream data features; fusing the video frame sequence features and the event stream data features to obtain multi-modal fusion features; and performing micro-expression recognition by using the multi-modal fusion features to obtain a micro-expression category of the face target.

[0083] The beneficial effects of the present application are that: the present application can first obtain a video frame sequence and event stream data of a face target shot within a preset time period, wherein the event stream data is collected by a dynamic visual sensor, the time resolution of the dynamic visual sensor is extremely high, and then the event stream data can be used to collect subtle expression changes in the face target, and the video frame sequence can be effectively supplemented; then, the present application can extract features from the video frame sequence to obtain video frame sequence features, and extract features from the event stream data to obtain event stream data features, so that the video frame sequence features and the event stream data features can be fused to obtain multi-modal fusion features, and micro-expression recognition can be performed based on the multi-modal fusion features, so that the recognition accuracy of the face target micro-expression can be significantly improved.

[0084] The present application also provides a micro-expression recognition device, an electronic device and a computer readable storage medium, which have the above beneficial effects. BRIEF DESCRIPTION OF DRAWINGS

[0085] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0086] Figure 1 A flowchart of a micro-expression recognition method provided by an embodiment of the present application;

[0087] Figure 2 A schematic diagram of a micro-expression recognition model provided by an embodiment of the present application;

[0088] Figure 3 A schematic diagram of an output module provided by an embodiment of the present application;

[0089] Figure 4 A flowchart of face feature extraction provided by an embodiment of the present application;

[0090] Figure 5 A schematic diagram of a converter model provided by an embodiment of the present application;

[0091] Figure 6 A schematic diagram of a micro-expression enhancement structure provided by an embodiment of the present application;

[0092] Figure 7 A schematic diagram of a video frame sequence processing module provided by an embodiment of the present application;

[0093] Figure 8A schematic diagram of an input space maintaining structure provided by an embodiment of the present application;

[0094] Figure 9 A flow chart of a feature extraction process provided by an embodiment of the present application;

[0095] Figure 10 A schematic diagram of an event stream data processing module provided by an embodiment of the present application;

[0096] Figure 11 A flow chart of a coefficient calculation provided by an embodiment of the present application;

[0097] Figure 12 A schematic diagram of a feature fusion module provided by an embodiment of the present application;

[0098] Figure 13 A structural block diagram of a micro-expression recognition device provided by an embodiment of the present application;

[0099] Figure 14 A structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0100] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0101] Micro-expression refers to facial expression with short change time and small change amplitude. In the related art, due to the low frame rate of the image acquisition device, it is difficult to effectively capture the subtle change of micro-expression, and thus the image acquired by the image acquisition device cannot be used for micro-expression type recognition, which brings difficulty to the recognition of micro-expression type. Therefore, the present application can provide a micro-expression recognition method, which can effectively supplement the video frame sequence by using event stream data with extremely high time resolution, and can fuse the video frame sequence features of the video frame sequence and the event stream data of the event stream data for micro-expression recognition, so as to improve the recognition accuracy of the facial target micro-expression.

[0102] It should be noted that the embodiments of the present application are not limited to the hardware devices for executing the method, for example, the hardware devices can be personal computers, mobile devices (mobile phones, tablet computers) and servers, etc., which can be set according to actual application requirements.

[0103] Reference is made to Figure 1 , Figure 1A flowchart of a micro-expression recognition method provided by an embodiment of the present application, which can include:

[0104] S100, acquiring a video frame sequence and event stream data obtained by shooting a human face target in a preset time period; the event stream data is collected by a dynamic vision sensor.

[0105] The dynamic vision sensor (DVS, Dynamic Vision Sensor) can also be referred to as an event camera. Unlike image acquisition devices, each pixel on the dynamic vision sensor can work independently and collect event data independently. Event data represents whether there is a change in illumination at the pixel point at the current time, which generally includes pixel point coordinates, absolute time and polarity. The polarity represents the direction of the change in illumination, for example, a positive polarity indicates that the illumination intensity is enhanced, and a negative polarity indicates that the illumination intensity is weakened, that is, when there is a polarity, it means that the pixel point has collected an illumination change event. Since the dynamic vision sensor does not collect event data in units of frames, but almost real-time time data, the time resolution of the event stream data collected by the dynamic vision sensor is extremely high, generally 1 μs. In comparison, the frame rate of a general image acquisition device can only be adjusted between 25-200 frames / second. Obviously, in the same time period, the dynamic vision sensor can collect more data, which is very beneficial to the collection of micro-expressions.

[0106] The feature pattern of micro-expression recognition is relatively weak, that is, the external observable expression change caused is very weak, and another feature is that the duration of micro-expression is particularly short, generally only 1 / 25-1 / 2 second. If the shortest duration of micro-expression is calculated, for a general image acquisition device, in the case of a frame rate of 25, only one video frame picture is output; and in the case of a frame rate of 200, the video frame sequence contains only 8 pictures. This greatly limits the recognition of micro-expression at the input end. Since the event stream data only records the position and direction of the change in space, there is no concept of frame, so if each data collection is expanded into a picture by filling, the conversion frame rate of the event stream data will be 10 6 This ultra-high time resolution will be able to capture more micro-expression features, thereby improving the accuracy of micro-expression recognition.

[0107] It should be noted that the embodiment of the present application performs micro-expression recognition based on the video frame sequence and the event stream data collected within a preset time period. The embodiment of the present application does not limit the length of the preset time period, which can be set according to actual application requirements. The embodiment of the present application also does not limit the number of images contained in the video frame sequence and the number of event data contained in the event stream data, which can be set according to actual application requirements. In addition, the video frame sequence and the event stream data can be collected by the same device or different devices. In the case of collecting by the same device, the video frame sequence and the event stream data are both collected by a dynamic active vision sensor (DAVIS, Dynamic and Active Vision Sensor), which can simultaneously collect images and event data on the same sensor, and the image collection frame rate is 40 frames per second. In the case of collecting by different devices, the video frame sequence can be collected by an image collection device, and the event stream data can be collected by a dynamic vision sensor. Since the viewing angle and collection time of the image collection device and the dynamic vision sensor may be different, after collecting the video frame sequence and the event stream data by different devices, the video frame sequence and the event stream data need to be time-aligned and space-aligned, wherein the time alignment refers to aligning the collection time of the video frame sequence with the collection time of the event stream data, and the space alignment refers to aligning the image content of the video frame sequence with the content of the event stream data.

[0108] Based on this, the video frame sequence and the event stream data obtained by photographing the face target within a preset time period can include:

[0109] S101: Obtain a video frame sequence collected by an image collection device.

[0110] S102: Obtain event stream data collected by a dynamic vision sensor.

[0111] S103: Time-align and space-align the video frame sequence and the event stream data.

[0112] S200, feature extraction is performed on the video frame sequence to obtain video frame sequence features.

[0113] In the embodiment of the present application, the video frame sequence features are serialized from the features of all images in the video frame sequence. In other words, the embodiment of the present application performs feature extraction on the video frame sequence, that is, feature extraction is performed on each image in the video frame sequence. It should be noted that the embodiment of the present application does not limit the specific way of performing feature extraction on the video frame sequence, which can be set according to actual application requirements.

[0114] S300, feature extraction is performed on the event stream data to obtain event stream data features.

[0115] Since the light on the face will change accordingly when the face changes expression, the event stream data contains rich facial expression information. The embodiment of the present application will extract features from the event stream data to further extract facial expression features from the perspective of event stream to facilitate micro-expression recognition. It should be noted that the embodiment of the present application does not limit the specific way of extracting features from the event stream data, which can be set according to actual application requirements.

[0116] It is worth noting that step S200 and step S300 can be executed in parallel.

[0117] S400, fuse the video frame sequence feature and the event stream data feature to obtain a multi-modal fusion feature.

[0118] After obtaining the video frame sequence feature and the event stream data feature, the embodiment of the present application will fuse the two to identify micro-expression based on features of different modalities. It should be noted that the embodiment of the present application does not limit the specific way of fusing the video frame sequence feature and the event stream data feature, for example, it can be simply spliced, or it can be weighted fusion, which can be set according to actual application requirements.

[0119] S500, using the multi-modal fusion feature to identify micro-expression to obtain the micro-expression category of the face target.

[0120] Specifically, the embodiment of the present application can first reduce the dimension of the multi-modal fusion feature through the full connection layer, and normalize the dimension-reduced feature to obtain the probability value of the face target belonging to each micro-expression category, so as to determine the micro-expression category corresponding to the face target according to the probability value.

[0121] Therefore, using the multi-modal fusion feature to identify micro-expression to obtain the micro-expression category of the face target can include:

[0122] S501, using the full connection layer to reduce the dimension of the multi-modal fusion feature to obtain a dimension-reduced feature;

[0123] S502, normalizing the dimension-reduced feature to obtain the probability value of the face target belonging to each micro-expression category;

[0124] S503, determining the micro-expression category corresponding to the face target according to the probability value.

[0125] It should be noted that the embodiment of the present application does not limit the specific micro-expression category, which can include happiness, anger, sadness, etc., which can be set according to actual application requirements. The embodiment of the present application does not limit the specific process of full connection layer processing and normalization processing, which can refer to related technologies.

[0126] Of course, for the convenience of the user to view, after obtaining the micro-expression category corresponding to the face target, the micro-expression category can be output to the display device.

[0127] For the convenience of understanding, please refer to Figure 2 , Figure 2 A schematic diagram of a micro-expression recognition model provided by an embodiment of the present application. The micro-expression recognition model specifically includes a video frame processing module, an event stream data processing module, a feature fusion module, and an output module, which are respectively used to execute steps S200, S300, S400, and S500. Please refer to Figure 3 , Figure 3 A schematic diagram of an output module provided by an embodiment of the present application. The module can reduce the dimensionality of the multi-modal fusion feature through multiple fully connected layers, and then utilize the output layer to perform normalization exponential processing on the reduced dimensionality feature to obtain the probability value of the face target belonging to each micro-expression category, so as to determine the micro-expression category corresponding to the face target according to the probability value. It should be pointed out that the above-mentioned model is trained, that is, the weight in all calculation modules is determined. In order to determine this weight, a supervised training process is needed to determine the weight among all data modules, and this process is obtained by training on a standard data set using a cross-entropy loss function.

[0128] Based on the above embodiment, the present application can first obtain the video frame sequence and event stream data of the face target shot within a preset time period, wherein the event stream data is collected by a dynamic visual sensor, and the time resolution of the dynamic visual sensor is extremely high, so that the event stream data can be used to collect the subtle expression changes in the face target, and the video frame sequence can be effectively supplemented; then, the present application can extract features from the video frame sequence to obtain video frame sequence features, and extract features from the event stream data to obtain event stream data features, so that the video frame sequence features and the event stream data features can be fused to obtain multi-modal fusion features, and micro-expression recognition can be performed based on the multi-modal fusion features, so as to significantly improve the recognition accuracy of the micro-expression of the face target.

[0129] Based on the above embodiment, the specific working process of the video frame sequence processing module will be described in detail. In one possible case, the video frame sequence is extracted to obtain the video frame sequence feature, which can include:

[0130] S201, picture feature extraction is performed on each image in the video frame sequence to obtain a picture feature sequence.

[0131] The embodiment of the present application can first extract features of complete picture content contained in each image in the video frame sequence to obtain a picture feature sequence. It should be noted that the embodiment of the present application does not limit how to extract picture features, and a suitable neural network model can be selected for extraction according to actual needs. For example, considering that the efficient network model (EfficientNet) is relatively lightweight and can better extract picture features while saving computing power, the efficient network model can be used to extract picture features of each image in the video frame sequence to obtain the picture feature sequence.

[0132] S202, face feature extraction is performed on each image in the video frame sequence to obtain a face feature sequence.

[0133] In addition to extracting features of complete picture content, considering that the change range of micro-expression is small, the embodiment of the present application also extracts face features of each image to remove changes irrelevant to expression caused by face movement, focuses on the face producing micro-expression, and avoids negative effects of background irrelevant information. Specifically, the embodiment of the present application can first perform face recognition on each image to determine a face region where the face is located; then, feature extraction can be performed on the face region to obtain the required face features, and the face features of all images in the video frame sequence can be integrated into a face feature sequence.

[0134] Based on this, face feature extraction is performed on each image in the video frame sequence to obtain a face feature sequence, which can include:

[0135] Step 11: For each image in the video frame sequence, face target detection is performed on the image to obtain a face region in the image;

[0136] Step 12: Face feature extraction is performed on the face region in each image to obtain a face feature sequence corresponding to the video frame sequence.

[0137] It should be noted that the embodiment of the present application does not limit how to perform face target detection, and related technologies of face target detection can be referred to, for example, the Openface tool can be used for face target extraction. The embodiment of the present application also does not limit how to extract face features. Considering that the feature extraction effect of the deep residual network (ResNet) is good, the embodiment of the present application can use the deep residual network to extract face features of the face region in each image to obtain a face feature sequence corresponding to the video frame sequence.

[0138] Further, considering that the face target is easy to move, which causes the size of the face region to change. To avoid the negative influence caused by the size change of the face target, the face region can be scaled to adjust the face region in each image to the same or similar size. Specifically, the face region is first marked with the key points to mark the position of the facial features; then, the face region can be scaled according to the position of the facial features and the preset relative position relationship between the facial features and the face. The face region is adjusted based on the position of the facial features and the preset relative position relationship between the facial features and the face, which can obtain a better face region adjustment effect.

[0139] Based on this, before the face feature extraction of the face region in each image, it can also include:

[0140] Step 21: In the face region, the position of the facial features is marked with the key points;

[0141] Step 22: The face region is scaled according to the position of the facial features and the preset relative position relationship between the facial features and the face.

[0142] Correspondingly, the face feature extraction of the face region in each image can include:

[0143] Step 23: The face feature extraction is performed on the scaled face region.

[0144] For easy understanding, please refer to Figure 4 , Figure 4 A flowchart of face feature extraction is provided by the embodiment of the present application. First, the face detection is performed on the video frame sequence to determine the face region in each image in the video frame sequence; then the face registration is performed to adjust the size of each face region to a uniform size; finally, the face feature extraction is performed on the face region to obtain the face feature sequence corresponding to the video frame sequence.

[0145] Further, since the present application mainly aims at the micro-expression recognition of a single face target, multiple face targets inevitably appear in the image collection area, which causes multiple face regions in the same image and affects the subsequent recognition. To avoid the influence of multiple face targets on the micro-expression recognition, when it is determined that the image contains multiple face regions, the face feature of the multiple face regions can be averaged and fused to fuse the face features of the multiple face regions into a face feature, so as to avoid the influence of multiple face targets on the recognition.

[0146] Based on this, after the face feature extraction of the face region in each image, it can also include:

[0147] Step 31: When the image contains multiple face regions, the face features of the multiple face regions are averaged and fused.

[0148] S203, the picture feature sequence is processed by the transformer model to obtain a video frame sequence feature, and the picture feature sequence is processed by a multi-head attention mechanism, micro-expression enhancement processing and multi-layer perception processing.

[0149] The picture feature sequence and the face feature sequence are processed by the transformer model to further improve the attention of the face expression feature. Specifically, the transformer model processes the picture feature sequence in sequence by a multi-head attention mechanism, micro-expression enhancement processing and multi-layer perception processing, wherein the micro-expression enhancement processing is attention mechanism processing using the face feature sequence and the picture feature sequence after the multi-head attention mechanism processing, thereby further improving the extraction of the face expression feature.

[0150] To achieve the above effects, the transformer model can be improved to additionally add a residual structure for micro-expression enhancement. Specifically, the transformer model is composed of transformer units, and each transformer unit is composed of a first residual structure, a second residual structure and a third residual structure in series. The first residual structure is a multi-head attention residual structure, the second residual structure is a micro-expression enhancement structure residual structure, and the third residual structure is a multi-layer perception residual structure. The micro-expression enhancement structure uses the output vector sequence of the first residual structure as the query vector sequence, maps the face features in the face feature sequence to the key vector sequence and the value vector sequence through two different weight fully connected layers, and processes the key vector sequence, the value vector sequence and the query vector sequence by attention mechanism to obtain an enhanced vector sequence input to the third residual structure. For better understanding, please refer to Figure 5 , Figure 5 A schematic diagram of the transformer model provided by the embodiment of the present application is shown in the figure, wherein the left side is the transformer unit included in the general transformer model, and the right side is the transformer unit provided by the embodiment of the present application. It should be pointed out that the first residual structure is composed of a multi-head attention mechanism module and a summation and normalization module, the second residual structure is composed of a micro-expression enhancement structure and a summation and normalization module, and the third residual structure is composed of a multi-layer perception and a summation and normalization module. The input of the micro-expression enhancement structure is the output of the first residual structure and the face feature sequence, and the output of the first residual structure and the face feature sequence can be processed based on the mutual attention mechanism. Please refer to Figure 6 , Figure 6A schematic diagram of a micro-expression enhancement structure provided by an embodiment of the present application is shown in FIG. 1. The structure has two inputs, one of which is a face feature, and the other of which is an output result of a first residual structure. The face feature vector is first passed through a feature mapping layer composed of fully connected layers with different weights to obtain a key vector and a value vector, respectively. The output result of the first residual structure is taken as a query vector to perform vector multiplication with the key vector to obtain a calculation weight, and then the weight is multiplied with the value vector to obtain an enhanced vector output. It should be noted that the above is an example of a vector to illustrate the specific calculation process, and in the actual calculation process, a sequence composed of multiple vectors is used, so the output is also a sequence of enhanced vectors.

[0151] Further, to improve the feature extraction capability of the converter model, the converter model can also be composed of a preset number of converter units in series, for example, composed of 6 converter units in series.

[0152] Further, for ease of understanding, please refer to Figure 7 , Figure 7 A schematic diagram of a video frame sequence processing module provided by an embodiment of the present application is shown in FIG. 2. The module includes an image feature extraction module, a micro-expression enhancement module, and a converter model. The image feature extraction module is used to generate an image feature sequence, the micro-expression enhancement module is used to extract a face feature sequence corresponding to the video frame sequence, and the converter model is used to process the image feature sequence and the face feature sequence to obtain a video frame sequence feature.

[0153] Based on the above embodiment, the specific working process of the event stream data processing module is described in detail below. In one possible case, the feature extraction on the event stream data to obtain the event stream data feature can include:

[0154] S301, adjusting all absolute times in the event stream data to relative times by using a first absolute time in the event stream data.

[0155] The embodiment of the present application first needs to adjust the time information in the event stream data, because the time contained in the original event stream data is absolute time, and the time easily affects the accuracy of micro-expression recognition. Therefore, the embodiment of the present application adjusts all absolute times in the event stream data to relative times by using a first absolute time in the event stream data, i.e., the start time of the event data stream, to avoid the influence of different time records on the recognition result.

[0156] S302, converting the adjusted event stream data into point cloud data; the coordinates of each point in the point cloud data are composed of the pixel point coordinates corresponding to the point and the relative time corresponding to the event stream data of the point, and the feature of the point is the polarity of the event stream data corresponding to the point.

[0157] The recording manner of the event stream data is a four-dimensional vector, and the four dimensions are respectively two coordinates of a corresponding pixel point in a two-dimensional space, a time of event occurrence, and a polarity of event occurrence. Such a recording manner is similar to a three-dimensional point cloud recording manner. Using a point cloud type coding manner has the advantages of fast coding speed and easy processing, and therefore, the embodiment of the present application can convert the event stream data into point cloud data, and feature extraction is performed by using the point cloud data. Specifically, the embodiment of the present application can set the pixel point coordinates and the relative time in the event stream data as three-dimensional coordinates of a point in a point cloud space, and set the polarity as a feature of the point. In this way, all information in the event stream data can be completely converted into the point cloud, and the two dimensions representing space and the one dimension representing time can be regarded as an unfolded three-dimensional space, and it is ensured that the point cloud is in order in the time dimension.

[0158] Further, after the conversion of the point cloud data is completed, considering that noise information can still be included in the point cloud data, the present application can also perform noise reduction on the point cloud data before performing feature extraction on the point cloud data. Specifically, considering that the change of light intensity is usually a regional change, that is, the points at which events occur are usually adjacent in the space-time angle, for each point in the point cloud data, it can be determined whether other points in the neighborhood range of the point have polarity, wherein the neighborhood range is constituted by other points adjacent to the point. If none of the other points in the neighborhood range has polarity, it indicates that none of the other points around the point has generated an event, and therefore, the point belongs to noise and needs to be removed. Conversely, if some or all of the points in the neighborhood range have polarity, it indicates that the point does not belong to noise and does not need to be removed.

[0159] Based on this, before performing feature extraction on the point cloud data to obtain event stream data features, the following can also be included:

[0160] Step 41: For each point in the point cloud data, it is determined whether other points in the neighborhood range of the point have polarity; if yes, it is determined whether other points in the neighborhood range of the next point have polarity; if no, step 42 is entered;

[0161] Step 42: The point is subjected to noise reduction processing;

[0162] Correspondingly, the feature extraction on the point cloud data to obtain event stream data features can include:

[0163] Step 43: The feature extraction is performed by using the point cloud data subjected to noise reduction processing to obtain event stream data features.

[0164] S303, feature extraction is performed on the point cloud data to obtain event stream data features.

[0165] In the feature extraction, the point cloud data needs to be encoded first to obtain point cloud encoded data. The encoding can be performed by using a PointNet model, which is a general encoder for point cloud data. After the encoding, the point cloud encoded data can be subjected to feature extraction, which is performed by using a spatial preserving structure and a feature extraction module each time. The spatial preserving structure can perform time series compression and structure preserving on the point cloud encoded data, and the feature extraction module extracts features from the output of the spatial preserving structure. Further, to achieve better feature extraction effect, the point cloud data is subjected to two-stage feature extraction.

[0166] Based on this, the feature extraction on the point cloud data to obtain the event stream data feature can include:

[0167] Step 51: encoding the point cloud data to obtain point cloud encoded data;

[0168] Step 52: performing time series compression and structure preserving on the point cloud encoded data by using an input spatial preserving structure to obtain first output data;

[0169] Step 53: extracting features from the first output data to obtain first feature data;

[0170] Step 54: performing time series compression and structure preserving on the first feature data by using a feature spatial preserving structure to obtain second output data;

[0171] Step 55: extracting features from the second output data to obtain event stream data features.

[0172] It should be noted that the input spatial preserving structure and the feature spatial preserving structure are similar in structure, and the difference between them is only in the feature dimension processed. In addition, the process of extracting features from the first output data is the same as the process of extracting features from the second output data.

[0173] The specific process of time series compression and structure preserving will be introduced below. In one possible case, the time series compression and structure preserving on the point cloud encoded data by using the input spatial preserving structure to obtain the first output data can include:

[0174] Step 61: performing dimension transformation on the point cloud encoded data as input data to obtain dimension-transformed input data;

[0175] Step 62: performing multiple convolution operations on the input data to obtain a first feature map;

[0176] Step 63: performing time series compression on the first feature map to obtain a second feature map;

[0177] Step 64: dimension reduction processing is performed on the second feature map through a plurality of fully connected layers to obtain a third feature map;

[0178] Step 65: matrix addition is performed on the third feature map and a same-size diagonal matrix, and the addition result is deformed to obtain a transformation matrix;

[0179] Step 66: batch matrix multiplication processing is performed on the transformation matrix and the input data after dimension conversion to obtain first output data.

[0180] Specifically, the data processing process of the input space maintaining structure is as shown in Figure 8 First, 4 in the input data represents the feature dimension of the input data, B represents the batch size, and N represents the number of event stream data. In this module, the purpose of structure maintenance is to maintain the structural characteristics of the 4-dimensional data. The input data is converted into BxNx4 data through dimension conversion, and is converted into Bx4x4 data through a series of operations, and the two are obtained through batch matrix multiplication (BMM) to obtain the final output. The detailed operation of the other path includes changing the feature dimension of the input data to obtain a first feature map through a plurality of one-dimensional convolution processing, then performing time series compression processing on the first feature map through a recurrent neural network (RNN) to obtain a second feature map, and then changing the dimension of the second feature map to Bx16 through a plurality of fully connected layers, and adding the stretched matrix of the same size 4x4 diagonal matrix, and finally deforming to obtain one of the inputs of the batch matrix multiplication operation, that is, the transformation matrix of Bx4x4.

[0181] The specific process of feature extraction will be introduced below. In one possible case, the first output data is extracted to obtain first feature data, which can include:

[0182] Step 71: dimension conversion is performed on the first output data to obtain a fourth feature map;

[0183] Step 72: one-dimensional convolution kernel is used to perform convolution processing on the fourth feature map to obtain first feature data.

[0184] Specifically, the one-stage feature extraction operation process is as shown in Figure 9 , which mainly converts the dimension of the input three-dimensional tensor through one-dimensional convolution, and the dimension of the output feature map is Bx64xN. Such output is directly input into a structure similar to the input space maintaining structure. The subsequent two-stage feature extraction is similar to the one-stage feature extraction, and finally the output of BxNx64 is obtained.

[0185] Further, for the convenience of understanding, please refer to Figure 10 , Figure 10 is a schematic diagram of an event stream data processing module provided by an embodiment of the present application. As can be seen, first, the event stream data can be subjected to three kinds of preprocessing, namely, time relativization processing, point cloud conversion processing and noise reduction processing; then the point cloud data can be encoded to obtain point cloud encoded data; further, the point cloud encoded data can be subjected to twice feature extraction by using an input space maintaining structure, a one-stage feature extraction module, a feature space maintaining structure and a two-stage feature extraction module to obtain event stream data features.

[0186] Based on the above embodiment, the specific working process of the feature fusion module will be described in detail below. In one possible case, the video frame sequence features and the event stream data features are fused to obtain multi-modal fusion features, which can include:

[0187] S501, using the event stream data features to generate a first passing coefficient, and using the first passing coefficient to perform weighted scaling processing on the video frame sequence features to obtain video frame fusion features.

[0188] The embodiment of the present application specifically adopts a weighted fusion manner to fuse the video frame sequence features and the event stream data features. Specifically, for the video frame sequence features, the embodiment of the present application can use the event stream data features to generate a first passing coefficient, which is the weight of the video frame sequence features, for performing weight scaling on the video frame sequence features. Then, the embodiment of the present application can use the first passing coefficient to perform weighted scaling processing on the video frame sequence features to obtain video frame fusion features, so that the video frame sequence features can be weighted based on the event stream data features. Similarly, for the event stream data features, the embodiment of the present application can use the video stream sequence features to generate a second passing coefficient, which is the weight of the event stream data features, for performing weight scaling on the event stream data features. Then, the embodiment of the present application can use the second passing coefficient to perform weighted scaling processing on the event stream data features to obtain event stream fusion features, so that the event stream data features can be weighted based on the video frame sequence features.

[0189] It should be pointed out that the first passing coefficient and the second passing coefficient are generated in the same way. The generation of the first passing coefficient using the event stream data features will be described in detail below. In one possible case, the first passing coefficient is generated using the event stream data features, which can include:

[0190] Step 81: performing one-dimensional convolution processing on the event stream data features to obtain mapping features;

[0191] Step 82: performing maximum pooling processing on the mapping features to obtain maximum pooling features;

[0192] Step 83: the maximum pooling features are processed by using an S-shaped function to obtain the first passing coefficient.

[0193] Specifically, the embodiment of the present application sets a video gating unit to generate the first passing coefficient. The unit first maps data with a dimension of BxNx64 to BxKx64 through one-dimensional convolution with a kernel of 1, then performs a maximum pooling operation on the 64 dimensions to obtain BxK, and finally obtains the passing coefficient through a Sigmoid function. The process is as shown in Figure 11 The passing coefficient is used to weight and fuse the video frame compression features to obtain video frame fusion features. The same method can be used to obtain event stream fusion features. It should be noted that the dimension of the fusion features does not change, only the two kinds of data are weighted, and the subsequent vector splicing operation makes the dimension of the output features Bx(1024+64).

[0194] S502, generating a second passing coefficient using the video frame sequence features, and performing weighted scaling processing on the event stream data features using the second passing coefficient to obtain event stream fusion features.

[0195] S503, splicing the video frame fusion features and the event stream fusion features to obtain multi-modal fusion features.

[0196] For ease of understanding, please refer to Figure 12 , Figure 12 is a schematic diagram of a feature fusion module provided by the embodiment of the present application. The module includes two branches. The first branch can use a video gating unit to convert the event stream data features into a first passing coefficient, and use the first passing coefficient to weight the video frame sequence features to obtain video frame fusion features. The second branch can use an event gating unit to convert the video stream sequence features into a second passing coefficient, and use the second passing coefficient to weight the event stream data features to obtain event stream fusion features. Finally, the video frame fusion features and the event stream fusion features can be spliced to obtain multi-modal fusion features.

[0197] The micro-expression recognition device, electronic equipment and computer readable storage medium provided by the embodiment of the present application will be introduced below. The micro-expression recognition device, electronic equipment and computer readable storage medium described below can be mutually corresponding to the micro-expression recognition method described above.

[0198] For ease of understanding, please refer to Figure 13 , Figure 13 is a structural block diagram of a micro-expression recognition device provided by the embodiment of the present application. The device can include:

[0199] The acquisition module 1301 is configured to acquire a video frame sequence and event stream data obtained by shooting a face target in a preset time period. The event stream data is collected by a dynamic visual sensor.

[0200] The video frame sequence processing module 1302 is configured to perform feature extraction on the video frame sequence to obtain video frame sequence features.

[0201] The event stream data processing module 1303 is configured to perform feature extraction on the event stream data to obtain event stream data features.

[0202] The feature fusion module 1304 is configured to fuse the video frame sequence features and the event stream data features to obtain multi-modal fusion features.

[0203] The output module 1305 is configured to perform micro-expression recognition using the multi-modal fusion features to obtain a micro-expression category of the face target.

[0204] Optionally, the video frame sequence processing module 1302 can include:

[0205] The picture feature extraction submodule is configured to perform picture feature extraction on each image in the video frame sequence to obtain a picture feature sequence.

[0206] The face feature extraction submodule is configured to perform face feature extraction on each image in the video frame sequence to obtain a face feature sequence.

[0207] The converter model module is configured to perform multi-head attention mechanism processing, micro-expression enhancement processing, and multi-layer perception processing on the picture feature sequence order using a converter model to obtain the video frame sequence features. The micro-expression enhancement processing is attention mechanism processing performed on the picture feature sequence after the multi-head attention mechanism processing is completed using the face feature sequence.

[0208] Optionally, the face feature extraction submodule can include:

[0209] The face detection unit is configured to perform face target detection on each image in the video frame sequence to obtain a face region in the image.

[0210] The face feature extraction unit is configured to perform face feature extraction on the face region in each image to obtain a face feature sequence corresponding to the video frame sequence.

[0211] Optionally, the face feature extraction submodule can further include:

[0212] The face registration unit is configured to mark the positions of the five features in the face region using key points, and scale the face region according to the positions of the five features and a preset relative positional relationship between the five features and the face.

[0213] Correspondingly, the face feature extraction unit is specifically configured to:

[0214] perform face feature extraction on the scaled face region.

[0215] Optionally, the face feature extraction sub-module can further include:

[0216] a face feature fusion processing unit, configured to perform average fusion processing on face features of multiple face regions when the image contains the multiple face regions.

[0217] Optionally, the face feature extraction unit is specifically configured to:

[0218] extract face features of the face regions in each image by using a deep residual network to obtain a face feature sequence corresponding to the video frame sequence.

[0219] Optionally, the converter model is composed of converter units, and each converter unit is composed of a first residual structure, a second residual structure, and a third residual structure connected in series; the first residual structure is a residual structure of multi-head attention, the second residual structure is a residual structure of a micro-expression enhancement structure, and the third residual structure is a residual structure of a multi-layer perception machine; the micro-expression enhancement structure is configured to take an output vector sequence of the first residual structure as a query vector sequence, map face features in the face feature sequence to a key vector sequence and a value vector sequence through two full connection layers with different weights, and process the key vector sequence, the value vector sequence, and the query vector sequence by using an attention mechanism to obtain an enhanced vector sequence to be input to the third residual structure.

[0220] Optionally, the converter model is composed of a preset number of converter units connected in series.

[0221] Optionally, the picture feature extraction sub-module is specifically configured to:

[0222] extract picture features of each image in the video frame sequence by using an effective network model to obtain a picture feature sequence.

[0223] Optionally, the event stream data processing module 1303 can include:

[0224] a time relativization processing sub-module, configured to adjust all absolute times in the event stream data to relative times by using a first absolute time in the event stream data;

[0225] a point cloud conversion sub-module, configured to convert the adjusted event stream data into point cloud data; coordinates of each point in the point cloud data are composed of a pixel point coordinate and a relative time corresponding to the point corresponding to the event stream data, and a feature of the point is a polarity of the event stream data corresponding to the point;

[0226] a point cloud data feature extraction sub-module, configured to extract features of the point cloud data to obtain event stream data features.

[0227] Optionally, the event stream data processing module 1303 can further include:

[0228] The noise reduction module is configured to determine, for each point in the point cloud data, whether other points in a neighborhood range of the point have a polarity; if not, the point is subjected to noise reduction processing.

[0229] Correspondingly, the point cloud data feature extraction sub-module is specifically configured to:

[0230] The event stream data feature is obtained by performing feature extraction on the point cloud data subjected to the noise reduction processing.

[0231] Optionally, the point cloud data feature extraction sub-module can include:

[0232] The encoding unit is configured to perform encoding processing on the point cloud data to obtain point cloud encoding data.

[0233] The input space maintaining structure is configured to perform time series compression processing and structure maintaining processing on the point cloud encoding data to obtain first output data.

[0234] The first feature extraction unit is configured to perform feature extraction on the first output data to obtain first feature data.

[0235] The feature space maintaining structure is configured to perform time series compression processing and structure maintaining processing on the first feature data to obtain second output data.

[0236] The second feature extraction unit is configured to perform feature extraction on the second output data to obtain event stream data feature.

[0237] Optionally, the input space maintaining structure is specifically configured to:

[0238] The point cloud encoding data is subjected to dimension transformation to obtain dimension-transformed input data; the input data is subjected to multiple convolution processing to obtain a first feature map; the first feature map is subjected to time series compression processing to obtain a second feature map; the second feature map is subjected to dimension reduction processing through multiple fully connected layers to obtain a third feature map; the third feature map and a same-size diagonal matrix are subjected to matrix addition, and the addition result is subjected to deformation processing to obtain a transformation matrix; the transformation matrix and the dimension-transformed input data are subjected to batch matrix multiplication processing to obtain the first output data.

[0239] Optionally, the first feature extraction unit is specifically configured to:

[0240] The first output data is subjected to dimension transformation to obtain a fourth feature map; the fourth feature map is subjected to convolution processing through a one-dimensional convolution kernel to obtain the first feature data.

[0241] Optionally, the video frame sequence and the event stream data are both collected by a dynamic active vision sensor.

[0242] Optionally, the obtaining module 1301 comprises:

[0243] The first obtaining sub-module is configured to obtain a video frame sequence captured by the image capture device.

[0244] The second obtaining sub-module is configured to obtain event stream data captured by the dynamic vision sensor.

[0245] The alignment sub-module is configured to perform time alignment and space alignment on the video frame sequence and the event stream data.

[0246] Optionally, the output module 1305 can comprise:

[0247] The fully connected layer is configured to reduce dimensionality of the multi-module fusion feature to obtain a reduced dimensionality feature.

[0248] The normalization exponential processing module is configured to perform normalization exponential processing on the reduced dimensionality feature to obtain probability values of the face target belonging to each micro-expression category.

[0249] The output module is configured to determine a micro-expression category corresponding to the face target according to the probability values.

[0250] Optionally, the feature fusion module 1304 can comprise:

[0251] The first fusion sub-module is configured to generate a first passing coefficient using the event stream data feature, and perform weighted scaling processing on the video frame sequence feature using the first passing coefficient to obtain a video frame fusion feature.

[0252] The second fusion sub-module is configured to generate a second passing coefficient using the video frame sequence feature, and perform weighted scaling processing on the event stream data feature using the second passing coefficient to obtain an event stream fusion feature.

[0253] The splicing sub-module is configured to perform splicing processing on the video frame fusion feature and the event stream fusion feature to obtain a multi-modal fusion feature.

[0254] Optionally, the first fusion sub-module is specifically configured to:

[0255] perform one-dimensional convolution processing on the event stream data feature to obtain a mapping feature;

[0256] perform maximum pooling processing on the mapping feature to obtain a maximum pooling feature;

[0257] perform processing on the maximum pooling feature using an S-shaped function to obtain the first passing coefficient.

[0258] Please refer to Figure 14 , Figure 14The structural block diagram of the electronic device provided by the embodiment of the present application, the embodiment of the present application provides an electronic device 140, comprising a processor 141 and a memory 142;The memory 142 is used to save the computer program;The processor 141 is used to execute the micro-expression recognition method provided by the preceding embodiment when executing the computer program.

[0259] The specific process of the above micro-expression recognition method can refer to the corresponding content provided in the preceding embodiment, which will not be repeated here.

[0260] And the memory 142 as the carrier of the resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, and the storage mode can be temporary storage or permanent storage.

[0261] In addition, the electronic device 140 further comprises a power supply 143, a communication interface 144, an input / output interface 145 and a communication bus 146;The power supply 143 is used to provide working voltage for each hardware device on the electronic device 140;The communication interface 144 can create a data transmission channel between the electronic device 140 and the external device, and the communication protocol followed is any communication protocol applicable to the technical solution of the present application, which will not be specifically limited here;The input / output interface 145 is used to obtain external input data or output data to the outside world, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited here.

[0262] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the micro-expression recognition method of any of the above embodiments.

[0263] Since the embodiment of the computer readable storage medium part corresponds to the embodiment of the micro-expression recognition method part, the embodiment of the storage medium part is described in the embodiment of the micro-expression recognition method part, which will not be repeated here.

[0264] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other.For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0265] Those skilled in the art will further appreciate that the functionality of the various examples described herein— including the steps of the methods described herein— can be implemented by software programs executed by one or more processors. Specifically, such programs can be used to cause a processor or processors to perform a series of operations to achieve the functionality described herein. These programs can be created using a high level programming language and / or implemented using a programming language that can be translated to machine language by an assembler or compiler. Software programs can be stored in any

[0266] Steps of methods or algorithms described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), non-volatile memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0267] The above provides a micro-expression recognition method, device, electronic equipment and storage medium. The principle and implementation mode of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and its core idea. It should be pointed out that for those skilled in the art, without departing from the principle of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A micro-expression recognition method, characterized in that, The method comprises the following steps: acquiring a video frame sequence and event stream data obtained by shooting a face target within a preset time period, wherein the event stream data is collected by a dynamic visual sensor; extracting features of the video frame sequence to obtain video frame sequence features; extracting features of the event stream data to obtain event stream data features; fusing the video frame sequence features and the event stream data features to obtain multi-modal fusion features; performing micro-expression recognition by using the multi-modal fusion features to obtain a micro-expression category of the face target; the step of extracting features of the event stream data to obtain event stream data features comprises the following steps: adjusting all absolute times in the event stream data to relative times by using a first absolute time in the event stream data; converting the adjusted event stream data into point cloud data, wherein the coordinates of each point in the point cloud data are composed of pixel point coordinates and relative times corresponding to the point, and the feature of the point is the polarity of the event stream data corresponding to the point; extracting features of the point cloud data to obtain the event stream data features; the step of extracting features of the point cloud data to obtain the event stream data features comprises the following steps: performing encoding processing on the point cloud data to obtain point cloud encoding data; performing time series compression processing and structure maintaining processing on the point cloud encoding data by using an input space maintaining structure to obtain first output data; extracting features of the first output data to obtain first feature data; performing time series compression processing and structure maintaining processing on the first feature data by using a feature space maintaining structure to obtain second output data; extracting features of the second output data to obtain the event stream data features.

2. The micro-expression recognition method of claim 1, wherein, the step of extracting features of the video frame sequence to obtain video frame sequence features comprises the following steps: extracting picture features of each image in the video frame sequence to obtain a picture feature sequence; extracting face features of each image in the video frame sequence to obtain a face feature sequence; performing multi-head attention mechanism processing, micro-expression enhancement processing and multi-layer perception processing on the picture feature sequence in sequence by using a converter model to obtain the video frame sequence features, wherein the micro-expression enhancement processing is attention mechanism processing performed on the picture feature sequence after multi-head attention mechanism processing by using the face feature sequence.

3. The micro-expression recognition method of claim 2, wherein, the step of extracting face features of each image in the video frame sequence to obtain a face feature sequence comprises the following steps: detecting a face target in each image in the video frame sequence to obtain a face region in the image; extracting face features of the face region in each image to obtain a face feature sequence corresponding to the video frame sequence.

4. The micro-expression recognition method of claim 3, wherein, before the step of extracting face features of the face region in each image, the method further comprises the following steps: annotating positions of five organs in the face region by using key points; scaling the face region according to the positions of the five organs and a preset relative position relationship between the five organs and the face; correspondingly, the step of extracting face features of the face region in each image comprises the following steps: Face feature extraction is performed on the scaled face region.

5. The micro-expression recognition method of claim 3, wherein, After face feature extraction is performed on the face region in each of the images, the method further includes: When the image contains multiple face regions, performing average fusion processing on face features of the multiple face regions.

6. The micro-expression recognition method of claim 3, wherein, The face feature extraction on the face region in each of the images obtains a face feature sequence corresponding to the video frame sequence. The face feature extraction on the face region in each of the images is performed by using a deep residual network to obtain a face feature sequence corresponding to the video frame sequence.

7. The micro-expression recognition method of claim 2, wherein, The converter model is composed of converter units, and each converter unit is composed of a first residual structure, a second residual structure, and a third residual structure connected in series. The first residual structure is a multi-head attention residual structure, the second residual structure is a micro-expression enhancement residual structure, and the third residual structure is a multi-layer perceptron residual structure. The micro-expression enhancement structure is configured to take an output vector sequence of the first residual structure as a query vector sequence, map face features in the face feature sequence to a key vector sequence and a value vector sequence through two full connection layers with different weights, and process the key vector sequence, the value vector sequence, and the query vector sequence through an attention mechanism to obtain an enhanced vector sequence to be input to the third residual structure.

8. The micro-expression recognition method of claim 7, wherein, The converter model is composed of a preset number of converter units connected in series.

9. The micro-expression recognition method of claim 2, wherein, The face feature extraction on the face region in each of the images obtains a face feature sequence corresponding to the video frame sequence. The face feature extraction on the face region in each of the images is performed by using a deep residual network to obtain a face feature sequence corresponding to the video frame sequence.

10. The micro-expression recognition method of claim 1, wherein, Before the feature extraction on the point cloud data to obtain the event stream data feature, the method further includes: For each point in the point cloud data, determining whether other points in a neighborhood range of the point have the polarity; If not, performing noise reduction processing on the point; Correspondingly, the feature extraction on the point cloud data to obtain the event stream data feature includes: Performing feature extraction on the noise-reduced point cloud data to obtain the event stream data feature.

11. The micro-expression recognition method of claim 1, wherein, The time series compression processing and structure preservation processing on the point cloud encoding data by using the input space preservation structure to obtain first output data includes: Performing dimension transformation on the point cloud encoding data as input data to obtain dimension-transformed input data; Performing multiple convolution processing on the input data to obtain a first feature map; Performing time series compression processing on the first feature map to obtain a second feature map; Performing dimension reduction processing on the second feature map through multiple full connection layers to obtain a third feature map; Performing matrix addition on the third feature map and a same-size diagonal matrix, and performing deformation processing on the addition result to obtain a transformation matrix; Performing batch matrix multiplication processing on the transformation matrix and the dimension-transformed input data to obtain the first output data.

12. The micro-expression recognition method of claim 1, wherein, The feature extraction on the first output data to obtain first feature data includes: Performing dimension transformation on the first output data to obtain a fourth feature map; Convolution is performed on the fourth feature map by using a one-dimensional convolution kernel to obtain the first feature data.

13. The micro-expression recognition method of claim 1, wherein, The video frame sequence and the event stream data are collected by a dynamic active vision sensor.

14. The micro-expression recognition method of claim 1, wherein, The video frame sequence and the event stream data obtained by photographing the face target within a preset time period are acquired. The video frame sequence collected by the image acquisition device is acquired. The event stream data collected by the dynamic vision sensor is acquired. The video frame sequence and the event stream data are time-aligned and space-aligned.

15. The micro-expression recognition method of claim 1, wherein, The micro-expression recognition is performed by using the multi-modal fusion feature to obtain the micro-expression category of the face target. The multi-modal fusion feature is obtained by using a fully connected layer to reduce the dimension of the multi-modal fusion feature. The reduced dimension feature is normalized and exponentiated to obtain a probability value of the face target belonging to each micro-expression category. The micro-expression category corresponding to the face target is determined according to the probability value.

16. The micro-expression recognition method of any one of claims 1 to 15, characterized in that, The video frame sequence feature and the event stream data feature are fused to obtain a multi-modal fusion feature, including: A first pass coefficient is generated by using the event stream data feature, and the video frame sequence feature is weighted and scaled by using the first pass coefficient to obtain a video frame fusion feature. A second pass coefficient is generated by using the video frame sequence feature, and the event stream data feature is weighted and scaled by using the second pass coefficient to obtain an event stream fusion feature. The video frame fusion feature and the event stream fusion feature are spliced to obtain the multi-modal fusion feature.

17. The micro-expression recognition method of claim 16, wherein, The first pass coefficient is generated by using the event stream data feature, including: The event stream data feature is one-dimensionally convoluted to obtain a mapping feature. The mapping feature is maximum-pooled to obtain a maximum-pooled feature. The maximum-pooled feature is processed by using an S-shaped function to obtain the first pass coefficient.

18. A micro-expression recognition apparatus, characterized by, It includes: An acquisition module is configured to acquire a video frame sequence and event stream data obtained by photographing a face target within a preset time period, wherein the event stream data is collected by a dynamic vision sensor. A video frame sequence processing module is configured to perform feature extraction on the video frame sequence to obtain a video frame sequence feature. An event stream data processing module is configured to perform feature extraction on the event stream data to obtain an event stream data feature. A feature fusion module is configured to fuse the video frame sequence feature and the event stream data feature to obtain a multi-modal fusion feature. An output module is configured to perform micro-expression recognition by using the multi-modal fusion feature to obtain a micro-expression category of the face target. The event stream data processing module includes: A time relativization processing submodule is configured to adjust all absolute times in the event stream data to relative times by using a first absolute time in the event stream data. A point cloud conversion submodule is configured to convert the adjusted event stream data into point cloud data, wherein the coordinates of each point in the point cloud data are composed of pixel point coordinates and relative times corresponding to the point in the event stream data corresponding to the point, and the feature of the point is the polarity of the event stream data corresponding to the point. The point cloud data feature extraction submodule is configured to extract features from the point cloud data to obtain the event stream data features. The point cloud data feature extraction submodule includes: An encoding unit configured to encode the point cloud data to obtain point cloud encoded data; An input space preserving structure configured to perform time series compression and structure preserving on the point cloud encoded data using the input space preserving structure to obtain first output data; A first feature extraction unit configured to extract features from the first output data to obtain first feature data; A feature space preserving structure configured to perform time series compression and structure preserving on the first feature data using the feature space preserving structure to obtain second output data; A second feature extraction unit configured to extract features from the second output data to obtain the event stream data features.

19. An electronic device, comprising: The computer readable storage medium stores a computer program. The processor is configured to implement the micro-expression recognition method according to any one of claims 1 to 17 when executing the computer program. The computer readable storage medium stores computer executable instructions, and the computer executable instructions are loaded and executed by the processor to implement the micro-expression recognition method according to any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that, ​

Images