An automatic micro-expression and macro-expression localization method based on video-level tags

Abstract

This invention discloses an automatic micro-expression and macro-expression localization method based on video-level labeling. First, optical flow is generated based on the video image. The video image and optical flow are segmented into uniform-length segments, and input features are generated. Then, a core salient complementarity module is used to capture the similarity between the image and the optical flow, achieving salient modal consistency. Modality-related attention is generated using the features after modality complementarity, and the average attention of the two modalities is calculated. Next, temporal distribution differences are used to achieve video distribution consistency for micro-expressions and macro-expressions, and differences in facial muscle movement duration are used to achieve label duration consistency for micro-expressions and macro-expressions. Finally, the similarity of features corresponding to the same label is used to achieve segment feature consistency for micro-expressions and macro-expressions. This invention's method achieves fine-grained frame-level micro-expression and macro-expression localization based on video-level weak labeling, solving the problems of time-consuming and labor-intensive frame-level annotation of video micro-expressions and macro-expressions, as well as the need to incorporate prior knowledge.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning technology, and specifically relates to a method for automatic localization of micro-expressions and macro-expressions based on video-level tags. Background Art

[0002] Facial expressions play a crucial role in conveying emotions, and their analysis in undisturbed long videos involves two stages: localization and recognition. Localization primarily determines the time period in which an expression (micro-expression or macro-expression) occurs, while recognition mainly identifies the expression type (e.g., happiness, sadness). Micro-expressions are subtle, unconscious facial movements that typically occur when attempting to hide or suppress true emotions. Compared to macro-expressions, micro-expressions are characterized by short duration, low movement intensity, and localized movement range. Micro-expressions reflect changes in true emotions; therefore, real-time detection and analysis of micro-expressions from videos is of significant value in high-risk or special environments (such as medical diagnosis, public safety, and business negotiations).

[0003] However, current facial expression localization methods rely on detailed frame-level annotations in long videos, requiring significant human effort, which makes them difficult to implement in large-scale applications. Furthermore, current methods generally rely on optical flow or image processing for localization, without fully integrating features from both, resulting in low accuracy. Additionally, due to the numerous differences between micro-expressions and macro-expressions, current methods do not adequately exploit these prior differences to enhance the model's learning ability. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes an automatic localization method for micro-expressions and macro-expressions based on video-level tags. This method captures the correlation between images and optical flow through a core significant complementary module, thereby achieving significant modal consistency between the two. Furthermore, it fully integrates the prior differences between the two (including frequency, duration, and features) into the model training, further optimizing the localization results and resolving the issues of asynchronous features between optical flow and images, as well as the integration of prior information.

[0005] The technical solution adopted in this invention is: an automatic micro-expression and macro-expression localization method based on video-level tags, the specific steps of which are as follows:

[0006] S1. Generate optical flow based on video images, segment the video images and optical flow into segments of uniform length, and use these segments to generate input features;

[0007] S2. Use the core significant complementary module to capture the temporal information similarity between the image and the optical flow and thereby achieve significant modal consistency to obtain spatial enhancement features and temporal enhancement features;

[0008] S3. Use the spatial enhancement features and temporal enhancement features obtained in step S2 after modal complementation to generate modal-related attention and calculate the average attention of the two modalities;

[0009] S4. Use temporal distribution differences to achieve consistency in video distribution of micro-expressions and macro-expressions, and use differences in the duration of facial muscle movements to achieve consistency in the duration of labels for micro-expressions and macro-expressions.

[0010] S5. Use the similarity of features corresponding to the same label to achieve the consistency of fragment features of micro-expressions and macro-expressions, and complete the automatic localization of micro-expressions and macro-expressions.

[0011] Furthermore, step S1 is specifically as follows:

[0012] S11. Generate optical flow based on video images, and segment the optical flow and video images into a set of segments of uniform length along the time dimension in a non-overlapping manner;

[0013] S12. Use the I3D model to extract features from all segments and use them as input.

[0014] Furthermore, step S2 is specifically as follows:

[0015] S21. Considering the temporal inconsistency of the two modal information, select one mode of the image and optical flow features obtained in step S1 as the main mode and the other as the auxiliary mode to construct a core significantly complementary module.

[0016] S22. Based on the core significant complementary module, the input auxiliary modal features are convolved once and copied three times. Two copies are used to calculate the feature similarity at each time point. The calculated similarity is then multiplied by the remaining feature. The result of the multiplication is the modal significant feature, which can achieve modal significant consistency between the image and optical flow features.

[0017] S23. The modal saliency consistency from step S22 is fused into the corresponding image and optical flow respectively, that is, the auxiliary modal saliency features extracted in step S22 are fused with the main modal features to obtain spatial enhancement features and temporal enhancement features.

[0018] Furthermore, step S3 is specifically as follows:

[0019] S31. Generate modality-related attention from the spatial enhancement features and temporal enhancement features obtained in step S2;

[0020] S32. Average the modal-related attention generated in step S31 along the time dimension to obtain the multimodal fusion average attention for each segment.

[0021] Furthermore, step S4 is specifically as follows:

[0022] S41. Concatenate the spatial augmentation features and temporal augmentation features obtained in step S2 along the feature dimensions to form the final overall feature;

[0023] S42. Using the overall features obtained in step S41, generate the final segment-level class probability matrix using multiple convolutional layers;

[0024] S43. Set up three branches to process the category probability matrix respectively, namely the category probability matrix used directly, the category probability matrix after average attention enhancement, and the category probability matrix after label duration consistency enhancement.

[0025] The average attention is multiplied by the category probability matrix along the time dimension to obtain the category probability matrix after attention enhancement. Label duration consistency is calculated by taking the mean of the average attention in the specified neighborhood, and using this mean to calculate the difference between two adjacent neighborhoods. This difference is then used to filter out potential micro-expression fragments, retaining only the expression and background fragments.

[0026] S44. For the three branches of step S43, the video distribution consistency is used to process the class probability matrix, that is, along the time dimension, different first K values ​​are taken according to different categories, and then the average value of the different K values ​​corresponding to different categories is calculated, and the final class probability is obtained based on the softmax function.

[0027] S45. Set different video-level labels for the three branches respectively, that is, the label of the first branch includes foreground and background, the label of the second branch does not include background, and the label of the third branch does not include micro-expressions.

[0028] S46. Match the class probabilities of the three branches obtained in step S44 with the labels corresponding to step S45, and calculate the final cross-entropy loss.

[0029] Furthermore, step S5 is specifically as follows:

[0030] S51. Based on video-level tags, obtain two videos v1 and v2 that have at least some of the same tags;

[0031] S52. Take the index of the top K average attention A1 values ​​in video v1, and use this index to filter the overall feature after the fusion of spatial and temporal features corresponding to video v1 and the segment-level category probability matrix generated from the overall feature to obtain candidate feature f1 and candidate probability matrix u1.

[0032] S53. Fuse the feature f1 and the category probability matrix u1 to obtain the features fu1 belonging to different categories;

[0033] S54. Calculate the cosine similarity s1 between feature fu1 and the feature corresponding to video v2 along the time dimension;

[0034] S55. Following the selected features, each segment on video v1 can obtain the maximum similarity ms1 corresponding to s1;

[0035] S56. Based on the similarity calculated in steps S54 and S55, the video-level tags y1 and y2 corresponding to videos v1 and v2 are fused, and the final feature consistency cs1 of video v1 segment is calculated. The calculation method is as follows:

[0036]

[0037] Where n represents the number of expression categories, T represents the number of segments along the time dimension, and A 1j This represents the j-th attention value generated along the time-correlated T dimension of video v1, in milliseconds. 1ji This represents the maximum similarity of the i-th class corresponding to the j-th segment along the time-related T-dimensional axis.

[0038] Similarly, repeat step S56 to calculate the feature consistency cs2 of video v2 segments.

[0039] S57. Calculate the final fragment feature consistency loss using the consistency results from step S56. The calculation method is as follows:

[0040]

[0041] Where, N s N represents the total number of valid tag pairs contained in all the filtered video pairs, d represents the number of filtered video pairs, and N represents the total number of valid tag pairs contained in all the filtered video pairs. p This represents the total number of all filtered video pairs.

[0042] The beneficial effects of this invention are as follows: The method of this invention first generates optical flow based on video images, segments the video images and optical flow into segments of uniform length and generates input features, then uses a core salient complementarity module to capture the similarity between the images and optical flow and achieve salient modal consistency, uses the features after modal complementarity to generate modality-related attention and calculate the average attention of the two modalities, then uses temporal distribution differences to achieve video distribution consistency of micro-expressions and macro-expressions and uses the differences in the movement duration of facial muscles to achieve label duration consistency of micro-expressions and macro-expressions, and finally uses the similarity of features corresponding to the same label to achieve segment feature consistency of micro-expressions and macro-expressions. The purpose of this invention is to alleviate the time-consuming and labor-intensive problems of existing annotation methods by using video-level labels to achieve weakly supervised micro-expression and macro-expression localization. It proposes a core salient complementary module to capture the similarity between images and optical flow, thereby achieving significant modal consistency between the two. Utilizing the prior differences between the two, including frequency, duration, and features, it fully integrates them into the training process to further optimize the localization results. Based on video-level weak labels, it achieves fine-grained frame-level micro-expression and macro-expression localization, solving the problems of time-consuming and labor-intensive frame-level annotation of video micro-expressions and macro-expressions, as well as the integration of prior knowledge. Attached Figure Description

[0043] Figure 1 This is a flowchart of an automatic micro-expression and macro-expression localization method based on video-level tags according to the present invention.

[0044] Figure 2 This is a schematic diagram of the training and testing phase of the method of the present invention in an embodiment of the present invention. Detailed Implementation

[0045] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] like Figure 1 The flowchart of the automatic micro-expression and macro-expression localization method based on video-level tags of the present invention is shown below. The specific steps are as follows:

[0047] S1. Generate optical flow based on video images, segment the video images and optical flow into segments of uniform length, and use these segments to generate input features;

[0048] S2. Use the core significant complementary module to capture the temporal information similarity between the image and the optical flow and thereby achieve significant modal consistency to obtain spatial enhancement features and temporal enhancement features;

[0049] S3. Use the spatial enhancement features and temporal enhancement features obtained in step S2 after modal complementation to generate modal-related attention and calculate the average attention of the two modalities;

[0050] S4. Use temporal distribution differences to achieve consistency in video distribution of micro-expressions and macro-expressions, and use differences in the duration of facial muscle movements to achieve consistency in the duration of labels for micro-expressions and macro-expressions.

[0051] S5. Use the similarity of features corresponding to the same label to achieve the consistency of fragment features of micro-expressions and macro-expressions, and complete the automatic localization of micro-expressions and macro-expressions.

[0052] In this embodiment, step S1 is specifically as follows:

[0053] S11. Generate optical flow based on video images, and segment the optical flow and video images into a set of segments of uniform length along the time dimension in a non-overlapping manner;

[0054] S12. Use the I3D model to extract features from all segments and use them as input.

[0055] In this embodiment, step S2 is specifically as follows:

[0056] S21. Considering the temporal inconsistency of the two modal information, select one mode of the image and optical flow features obtained in step S1 as the main mode and the other as the auxiliary mode to construct a core significantly complementary module.

[0057] S22. Based on the core significant complementary module, the input auxiliary modal features are convolved once and copied three times. Two copies are used to calculate the feature similarity at each time point. The calculated similarity is then multiplied by the remaining feature. The result of the multiplication is the modal significant feature, which can achieve modal significant consistency between the image and optical flow features.

[0058] S23. The modal saliency consistency from step S22 is fused into the corresponding image and optical flow respectively, that is, the auxiliary modal saliency features extracted in step S22 are fused with the main modal features to obtain spatial enhancement features and temporal enhancement features.

[0059] In this embodiment, step S3 is specifically as follows:

[0060] S31. Generate modality-related attention from the spatial enhancement features and temporal enhancement features obtained in step S2;

[0061] S32. Average the modal-related attention generated in step S31 along the time dimension to obtain the multimodal fusion average attention for each segment.

[0062] like Figure 2 As shown, in this embodiment, step S4 is specifically as follows:

[0063] S41. Concatenate the spatial augmentation features and temporal augmentation features obtained in step S2 along the feature dimensions to form the final overall feature;

[0064] S42. Using the overall features obtained in step S41, generate the final segment-level class probability matrix using multiple convolutional layers;

[0065] S43. Set up three branches to process the category probability matrix respectively, namely the category probability matrix used directly, the category probability matrix after average attention enhancement, and the category probability matrix after label duration consistency enhancement.

[0066] The average attention is multiplied by the category probability matrix along the time dimension to obtain the category probability matrix after attention enhancement. The label duration consistency is calculated by taking the mean of the average attention in a specified neighborhood (which is related to the maximum number of segments that micro-expressions can cover), and using this mean to calculate the difference between two adjacent neighborhoods. This difference is then used to filter out potential micro-expression segments, retaining only the segments of expression and background.

[0067] S44. For the three branches of step S43, the video distribution consistency is used to process the class probability matrix, that is, along the time dimension, different first K values ​​are taken according to different categories, and then the average value of the different K values ​​corresponding to different categories is calculated, and the final class probability is obtained based on the softmax function.

[0068] S45. Set different video-level labels for the three branches respectively, that is, the label of the first branch includes foreground and background, the label of the second branch does not include background, and the label of the third branch does not include micro-expressions.

[0069] S46. Match the class probabilities of the three branches obtained in step S44 with the labels corresponding to step S45, and calculate the final cross-entropy loss.

[0070] like Figure 2 As shown, in this embodiment, step S5 is specifically as follows:

[0071] S51. Based on video-level tags, obtain two videos v1 and v2 that have at least some of the same tags;

[0072] S52. Take the index of the top K average attention A1 values ​​in video v1, and use this index to filter the overall feature after the fusion of spatial and temporal features of video v1 (from step S41) and the segment-level category probability matrix generated by the overall feature (from step S42) to obtain candidate feature f1 and candidate probability matrix u1.

[0073] S53. Fuse the feature f1 and the category probability matrix u1 to obtain the features fu1 belonging to different categories;

[0074] S54. Calculate the cosine similarity s1 between feature fu1 and the feature corresponding to video v2 along the time dimension;

[0075] S55. Following the selected features, each segment on video v1 can obtain the maximum similarity ms1 corresponding to s1;

[0076] S56. Based on the similarity calculated in steps S54 and S55, the video-level tags y1 and y2 corresponding to videos v1 and v2 are fused, and the final feature consistency cs1 of video v1 segment is calculated. The calculation method is as follows:

[0077]

[0078] Where n represents the number of expression categories, T represents the number of segments along the time dimension, and A 1j This represents the j-th attention value generated along the time-correlated T dimension of video v1, in milliseconds. 1ji This represents the maximum similarity of the j-th segment corresponding to the i-th class along the time-related T dimension (from step S55).

[0079] Similarly, repeat step S56 to calculate the feature consistency cs2 of video v2 segments.

[0080] S57. Calculate the final fragment feature consistency loss using the consistency results from step S56. The calculation method is as follows:

[0081]

[0082] Where, N s N represents the total number of valid tag pairs contained in all the filtered video pairs, d represents the number of filtered video pairs, and N represents the total number of valid tag pairs contained in all the filtered video pairs. p This represents the total number of all filtered video pairs.

[0083] In this embodiment, the method of the present invention first uses an I3D model trained on Kinetics to pre-train a model to extract features from the segments. During model training, the method of the present invention only provides video-level labels, that is, it does not tell the time point and frequency of the category, but only whether the category exists in the video. The purpose is to alleviate the problem of time-consuming and laborious annotation, and to achieve weakly supervised micro-expression and macro-expression localization in long videos. Figure 2 As shown, the method of this invention proposes a core significantly complementary module to capture the correlation between image and optical flow, thereby achieving significant modal consistency between the two. Furthermore, this invention utilizes the prior differences between the two, including frequency, duration, and features, and fully integrates them into the training process, enabling more refined localization.

[0084] In summary, the purpose of this invention is to alleviate the time-consuming and labor-intensive problems of existing annotation methods by using video-level labels to achieve weakly supervised micro-expression and macro-expression localization. It proposes a core salient complementary module to capture the similarity between images and optical flow, thereby achieving significant modal consistency between the two. Utilizing the prior differences between the two, including frequency, duration, and features, it fully integrates them into the training process to further optimize the localization results. Based on video-level weak labels, it achieves fine-grained frame-level micro-expression and macro-expression localization, solving the problems of time-consuming and labor-intensive frame-level annotation of video micro-expressions and macro-expressions, as well as the integration of prior knowledge.

[0085] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for automatic localization of micro-expressions and macro-expressions based on video-level tags, the specific steps of which are as follows: S1. Generate optical flow based on video images, segment the video images and optical flow into segments of uniform length, and use these segments to generate input features; S2. Use the core significant complementary module to capture the temporal information similarity between the image and the optical flow and thereby achieve significant modal consistency to obtain spatial enhancement features and temporal enhancement features; S21. Considering the temporal inconsistency of the two modal information, select one mode of the image and optical flow features obtained in step S1 as the main mode and the other as the auxiliary mode to construct a core significantly complementary module. S22. Based on the core significant complementary module, the input auxiliary modal features are convolved once and copied three times. Two copies are used to calculate the feature similarity at each time point. The calculated similarity is then multiplied by the remaining feature. The result of the multiplication is the modal significant feature, which can achieve modal significant consistency between the image and optical flow features. S23. The modal saliency consistency from step S22 is fused into the corresponding image and optical flow respectively, that is, the auxiliary modal saliency features extracted in step S22 are fused with the main modal features to obtain spatial enhancement features and temporal enhancement features; S3. Use the spatial enhancement features and temporal enhancement features obtained in step S2 after modal complementation to generate modal-related attention and calculate the average attention of the two modalities; S4. Use temporal distribution differences to achieve consistency in video distribution of micro-expressions and macro-expressions, and use differences in the duration of facial muscle movements to achieve consistency in the duration of labels for micro-expressions and macro-expressions. S5. Use the similarity of features corresponding to the same label to achieve the consistency of fragment features of micro-expressions and macro-expressions, and complete the automatic localization of micro-expressions and macro-expressions.

2. The method for automatic localization of micro-expressions and macro-expressions based on video-level tags according to claim 1, characterized in that, The specific steps of S1 are as follows: S11. Generate optical flow based on video images, and segment the optical flow and video images into a set of segments of uniform length along the time dimension in a non-overlapping manner; S12. Use the I3D model to extract features from all segments and use them as input.

3. The method for automatic localization of micro-expressions and macro-expressions based on video-level tags according to claim 1, characterized in that, Step S3 is as follows: S31. Generate modality-related attention from the spatial enhancement features and temporal enhancement features obtained in step S2; S32. Average the modal-related attention generated in step S31 along the time dimension to obtain the multimodal fusion average attention for each segment.

4. The method for automatic localization of micro-expressions and macro-expressions based on video-level tags according to claim 1, characterized in that, Step S4 is as follows: S41. Concatenate the spatial augmentation features and temporal augmentation features obtained in step S2 along the feature dimensions to form the final overall feature; S42. Using the overall features obtained in step S41, generate the final segment-level class probability matrix using multiple convolutional layers; S43. Set up three branches to process the category probability matrix respectively, namely the category probability matrix used directly, the category probability matrix after average attention enhancement, and the category probability matrix after label duration consistency enhancement. Among them, the average attention is multiplied by the category probability matrix along the time dimension to obtain the category probability matrix after the average attention enhancement; the label duration consistency is to calculate the mean of the average attention in the specified neighborhood, and use this mean to calculate the difference between two adjacent neighborhoods, filter this difference to remove potential micro-expression fragments, and only retain the expression and background fragments. S44. For the three branches of step S43, the video distribution consistency is used to process the class probability matrix, that is, along the time dimension, different first K values ​​are taken according to different categories, and then the average value of the different K values ​​corresponding to different categories is calculated, and the final class probability is obtained based on the softmax function. S45. Set different video-level labels for the three branches respectively, that is, the label of the first branch includes foreground and background, the label of the second branch does not include background, and the label of the third branch does not include micro-expressions. S46. Match the class probabilities of the three branches obtained in step S44 with the labels corresponding to step S45, and calculate the final cross-entropy loss.

5. The method for automatic localization of micro-expressions and macro-expressions based on video-level tags according to claim 1, characterized in that, Step S5 is as follows: S51. Based on video-level tags, obtain two videos v1 and v2 that have at least some of the same tags; S52. Take the index of the top K average attention A1 values ​​in video v1, and use this index to filter the overall feature after the fusion of spatial and temporal features corresponding to video v1 and the segment-level category probability matrix generated from the overall feature to obtain candidate feature f1 and candidate probability matrix u1. S53. Fuse the feature f1 and the category probability matrix u1 to obtain the features fu1 belonging to different categories; S54. Calculate the cosine similarity s1 between feature fu1 and the feature corresponding to video v2 along the time dimension; S55. Following the selected features, each segment on video v1 can obtain the maximum similarity ms1 corresponding to s1; S56. Based on the similarity calculated in steps S54 and S55, the video-level tags y1 and y2 corresponding to videos v1 and v2 are fused, and the final feature consistency cs1 of video v1 segment is calculated. The calculation method is as follows: Where n represents the number of expression categories, T represents the number of segments along the time dimension, and A 1j This represents the j-th attention value generated along the time-correlated T dimension of video v1, in milliseconds. 1ji This represents the maximum similarity of the i-th class corresponding to the j-th segment along the time-related T-dimensional axis; Similarly, repeat step S56 to calculate the feature consistency cs2 of video segment v2; S57. Calculate the final fragment feature consistency loss using the consistency results from step S56. The calculation method is as follows: Where, N s N represents the total number of valid tag pairs contained in all the filtered video pairs, d represents the number of filtered video pairs, and N represents the total number of valid tag pairs contained in all the filtered video pairs. p This represents the total number of all filtered video pairs.

Images