Macro-micro expression interval positioning method based on meta-learning

Abstract

The invention relates to the technical field of video action detection, and provides a macro-micro expression interval positioning method based on meta-learning, which comprises the following steps: performing face alignment and image size unified processing on an input face expression video frame sequence; performing down-sampling on the processed facial expression video frame sequence based on a pre-trained video understanding model, and extracting time sequence features; dividing and sampling meta-learning tasks according to preset attribute dimensions on the basis of the time sequence characteristics, and constructing a time sequence positioning model at the same time; performing meta-learning training on the time sequence positioning model according to the meta-learning task to obtain a meta-basic model with a generalized spatial-temporal feature representation capability; and carrying out adaptive fine tuning on the element basic model, and using the fine-tuned model to position macro expression and micro expression intervals of the target domain facial expression video. The method can be more easily deployed in various real and complex new scenes, so that the progress of the expression analysis related technology from the laboratory to the practical application is accelerated.

Description

Technical Field

[0001] This invention relates to the field of video action detection technology, and more specifically, to a method for temporal location and intensity classification of facial expression intervals generated in a video, particularly a macro-micro facial expression interval localization method based on meta-learning. Background Technology

[0002] Facial expression analysis technology has demonstrated wide application value in recent years in various fields such as emotion recognition, psychological state assessment, fatigue detection, and human-computer interaction. Within the field of computer vision, this technology mainly covers three key tasks: expression classification and recognition, facial action unit (AU) detection, and expression interval localization. Expression recognition aims to distinguish different categories of emotional expressions; AU detection focuses on the quantification of local muscle movements and is often used as a basic feature or auxiliary information for high-level semantic analysis; while expression interval localization is used to automatically determine the start and end times of expressions and their peak intensity from videos, belonging to the mid-to-upper-level tasks of structured analysis of spatiotemporal sequences.

[0003] In many practical applications, identifying "when an expression occurs" is more valuable than identifying "what kind of expression it is." For example, in judicial interrogation, quickly pinpointing the timing of a suspect's micro-expressions can provide crucial clues for interrogators; in clinical psychological assessment, capturing the intervals of specific emotional responses helps to more objectively assess a patient's condition; and in driver fatigue monitoring or human-computer interaction scenarios, real-time detection of expression intervals is a prerequisite for timely feedback. Furthermore, building expression recognition models typically requires extracting effective segments of macro- and micro-expressions from long videos. Because micro-expressions are extremely short-lived and have very subtle amplitudes, manual annotation is not only inefficient but also heavily reliant on professional experience, becoming a bottleneck restricting data expansion and technological development. Therefore, developing a high-precision automatic macro-micro-expression interval localization method can not only improve the efficiency of practical applications but also significantly accelerate the construction of expression datasets.

[0004] Early facial expression localization methods primarily relied on manually designed features, such as optical flow, texture variations, or facial keypoint motion trajectories, combined with threshold segmentation or rule inference to achieve localization. These methods demanded high levels of prior knowledge, had limited manual feature representation capabilities, and struggled to adapt to varying acquisition conditions and individual differences. In recent years, with the widespread application of deep learning in image and video analysis, researchers have begun using general neural network models (such as LSTM, dual-stream I3D, and VideoMAE) to automatically extract spatiotemporal features of facial expressions and achieve more accurate localization. However, these data-driven methods generally depend on large-scale, high-quality labeled samples. Due to privacy concerns, limited acquisition environments, the scarcity of micro-expression samples, difficulties in manual annotation, and low annotation consistency, existing databases are generally small. Simultaneously, significant differences in facial expressions exist among individuals, and external variables such as scene lighting, shooting angle, and occlusion conditions further exacerbate the complexity of data distribution. Therefore, most current models exhibit poor generalization ability in cross-identity and cross-scene testing, severely hindering the practical application of the technology.

[0005] In the field of macro-micro expression localization, most methods either focus on the extraction methods and quality of features or on building more novel temporal localization models, lacking a systematic solution for the model's cross-domain adaptability. However, real-world applications are complex and ever-changing, requiring localization systems to maintain stable performance when facing new users and new environments.

[0006] In recent years, meta-learning, as a technological paradigm capable of improving generalization ability with few samples, has shown potential in various video understanding tasks. Its core lies in learning a rapidly adaptable prior representation from diverse samples, enabling the model to adapt to new tasks with minimal sample size. However, according to research, there are no precedents for using meta-learning in the specific field of macro-micro-expression localization. Therefore, inventing a meta-learning-based macro-micro-expression localization technology would be highly significant for promoting the application of related technologies in complex real-world scenarios. Summary of the Invention

[0007] This invention addresses the shortcomings of existing technologies by developing a meta-learning-based macro-micro expression interval localization method. This method addresses the issues of poor accuracy and weak generalization in macro-micro expression interval localization models when faced with diverse real-world scenarios and unseen samples. Using popular video understanding models as feature extraction tools, meta-learning is employed to perform meta-basic training on the interval localization model. Through rapid fine-tuning training under specific environments and sample sets, a localization model with good adaptability and high generalization is obtained, enabling accurate localization of macro and micro expression generation intervals in facial expression videos.

[0008] To achieve the above objectives, this invention proposes a macro-micro expression interval localization method based on meta-learning, comprising the following steps:

[0009] Step 1: Face video preprocessing: Perform face alignment and image size unification on the input facial expression video frame sequence;

[0010] Step 2: Temporal Feature Extraction: Using a pre-trained video understanding model, downsample the video frame sequence processed in Step 1 to extract frame-level or segment-level temporal features;

[0011] Step 3: Meta-learning task construction and model building: Based on the temporal features extracted in Step 2, meta-learning tasks are divided and sampled according to the preset attribute dimensions, and a temporal localization model is constructed at the same time.

[0012] Step 4: Meta-training: The temporal localization model is trained using the meta-learning task sampled in Step 3, enabling the model to acquire generalized spatiotemporal feature representation capabilities.

[0013] Step 5: Rapid Fine-tuning and Localization: The meta-basic model trained in Step 4 is rapidly and adaptively fine-tuned using a small number of samples from the target domain scene, and the fine-tuned model is used to locate the macro-expression and micro-expression ranges of facial expression videos in the target domain.

[0014] Furthermore, the face alignment and image resolution unification processing in step one specifically includes:

[0015] Use the facial landmark detection tool library to detect the key points of the template face and the face in the first frame of the input video;

[0016] Based on the facial key points, the transformation matrix is ​​calculated to align the template face with the first frame image;

[0017] To prevent inter-frame jitter during alignment, all subsequent frames of the video sequence are aligned using the affine transformation matrix calculated from the first frame.

[0018] After alignment, the image is cropped by extending outwards by several pixels based on the outermost key points to ensure that the edge of the key area is included, and finally the image is scaled to the same size.

[0019] Furthermore, the temporal feature extraction in step two specifically includes:

[0020] Define the length T of the downsampled feature sequence, and the window size w and stride s for feature extraction; then the formula for calculating the length t of the required input video segment is: t = w + (T - 1) * s;

[0021] For videos with different frame rates, frame extraction can be used to unify them to a certain frame rate, or the values ​​of w and s can be scaled proportionally to obtain features with similar temporal representation ranges.

[0022] A video segment of length t is extracted from the original video using a sliding window method, with a extraction step size of t / 2;

[0023] Feature extraction employs, but is not limited to, the following two methods based on the input modality:

[0024] a. When the input is RGB modality, a pre-trained encoder of a single-modal video understanding model is used for processing. The input tensor shape is [t, 3, w, H, W], and the output is a feature of [t, D].

[0025] b. When the input is a dual-modal video understanding model of RGB and optical flow, it is used to process the data, extracting RGB and optical flow features separately, extracting dense optical flow in adjacent frames, and subtracting the average optical flow within pixels around the nose center keypoint to eliminate errors caused by facial or image jitter. Simultaneously, the last frame of the RGB sequence is removed to maintain frame consistency. The input shape is RGB: [T, 3, w, H, W], optical flow: [T, 2, w, H, W], and the output is features of [T, D]. Finally, the dual-stream features are concatenated along the time dimension T to obtain a fused feature of [T, 2D].

[0026] Furthermore, the specific method for constructing the meta-learning task in step (3) is as follows:

[0027] The samples to be divided are pre-extracted features, and n non-overlapping tasks are sampled from the feature samples. To perform meta-learning, each task is actually a set of sample features, and each task's sample set is divided into a support set. and query set .

[0028] The preset attribute dimensions include one or more of the following division methods:

[0029] a. Divide by duration of facial expression interval: Divide the samples into different task sets according to the duration of their facial expressions;

[0030] b. Segment by face race or age: Divide the samples into different task sets according to the race or age group of the faces;

[0031] c. Divide by expression intensity category distribution: Divide the samples into different task sets according to the intensity category distribution of micro-expressions / macro-expressions or a mixture of both.

[0032] 5. The method according to claim 1, characterized in that the meta-learning training in step (4) specifically includes the following process:

[0033] The parameters of the initial model are This model is used as the target model in the outer meta-learning loop, and it does not participate in the gradient update of the inner loop. This model is copied n times as the inner loop model for n tasks. In the inner loop, for each task... The model performs k gradient updates on the support set for this task:

[0034]

[0035] in, For the first The task model The parameters are updated by the subgradient. The learning rate for the inner loop can be different for each task.

[0036] Using query sets To test the model and calculate The corresponding gradients. The gradients of the query set do not participate in the update of the inner loop model; instead, the sum of the gradients of the query sets from n tasks is used to update the target model of the outer loop.

[0037]

[0038]

[0039] in, This is the sum of the gradients of all task query sets on the corresponding model. The learning rate for the outer loop target model. For the first The parameters of a task model after k gradient descent operations.

[0040] Furthermore, the structure of the temporal localization model includes:

[0041] a. One-dimensional convolutional mapping layer: used to map input features to a fixed dimension, the size of which is consistent with the dimension of the feature embedding layer;

[0042] b. Multi-scale Transformer encoder: It consists of multiple layers, including non-downsampling layers that do not perform downsampling and downsampling layers that perform double downsampling, used to extract multi-scale temporal feature representations;

[0043] c. Classification Head and Boundary Regression Head: These are two one-dimensional convolutional sub-networks that share underlying weights. They are used to predict the probability of each time point belonging to an expression interval and the distance from that point to the interval boundary, respectively.

[0044] Furthermore, the non-downsampling layer in the multi-scale Transformer encoder sequentially includes: a feature embedding layer, a multi-head self-attention mechanism layer, and a feature pyramid network (FPN) layer; the downsampling layer adds a downsampling module to the non-downsampling layer structure.

[0045] Furthermore, the inputs for the classification head and boundary regression head corresponding to layer l are [batch, T / 2]. l - 1 D e ], where D e The embedding dimension of the encoder; the output is [batch, T / 2]. l - 1 [2], where the 2D of the classification head represents the probability of each macro-micro expression interval on the current layer time coordinate, and the regression head represents the distance from each point on the current layer time coordinate to the boundary.

[0046] Furthermore, the rapid adaptive fine-tuning in step (5) refers to updating the weights of the meta-basic model with a small number of iterations using labeled samples from the target domain via gradient descent.

[0047] Furthermore, during model training and testing, the labels are transformed according to the following rules:

[0048] Category label definition: Micro-expression category label is 0, macro-expression category label is 1;

[0049] Range label conversion:

[0050] The initial label is the frame-level interval [t1, t2];

[0051] After feature extraction, the feature is converted into an interval [T1, T2] on the feature time axis. The conversion formula is as follows:

[0052]

[0053]

[0054] When calculating the loss within the model, it is converted into feature points T. f Distance label to interval boundary [T] f - T1, T f -T2], and only if feature point T f Only when the value is located within the interval [T1, T2] will it participate in the calculation of the boundary regression loss.

[0055] Furthermore, the criterion for judging the model prediction results is: when the IOU between the predicted interval and the true interval is >= 0.5, the positioning is judged as successful.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] The core advantage of this invention lies in its innovative introduction of meta-learning training. By simulating "variable tasks" (such as task sets divided by duration, ethnicity, and age), the model is no longer learning a single data distribution, but rather acquiring the essential ability to "quickly adapt to a new task." This makes the resulting meta-based model itself an excellent "fast learner," with powerful built-in cross-domain generalization prior knowledge. In practical applications, it only requires brief fine-tuning with a very small number of samples from the target scenario to quickly adapt and demonstrate excellent performance. This advantage effectively solves the biggest pain point in practical applications—the scenario migration problem—significantly reducing deployment costs and data dependencies.

[0058] This invention does not limit itself to traditional feature extraction methods, but fully draws on and utilizes the latest achievements in the field of computer vision. It flexibly selects currently high-performance pre-trained models for different input modalities: different feature extraction models are adopted for different input modalities.

[0059] This invention successfully developed a high-precision, highly generalizable macro-micro expression interval localization solution, providing a reliable technical tool for related academic research and practical applications. This method effectively solves the core problems of poor generalization and low accuracy in existing technologies. Through the combination of meta-learning and advanced video understanding models, the trained localization model can accurately determine and locate the start and end intervals of macro and micro expressions in continuous video streams, with high-quality and highly reliable output results. It provides a powerful data preprocessing tool for fields requiring detailed facial expression analysis, such as psychology, medical clinical auxiliary diagnosis, and security anti-counterfeiting, automatically generating well-labeled sample libraries, greatly promoting the efficiency and depth of related research. Simultaneously, this method possesses strong generalization capabilities, making it easier to deploy in various real and complex new scenarios, thereby accelerating the process of moving expression analysis technologies from the laboratory to practical applications. Attached Figure Description

[0060] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0061] Figure 1 This is a flowchart of the overall process of macro-micro expression range localization based on meta-learning in this invention;

[0062] Figure 2 This is a visual representation of time-dimension downsampling in the feature pre-extraction process in this embodiment of the invention;

[0063] Figure 3This is a schematic diagram illustrating the training details of the meta-basic model, taking RGB input + VideoMAEv2 model feature extraction as an example in an embodiment of the present invention. Detailed Implementation

[0064] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0065] This embodiment proposes a macro-micro expression interval localization method based on meta-learning, such as... Figure 1 As shown, it includes the following steps:

[0066] (1) For the sequence of human facial expression video frames captured by the video acquisition device, perform necessary face alignment (registration) and image resolution unification.

[0067] (2) For the registered face video frame sequence, different video understanding models are used to extract frame-level / segment-level downsampling temporal features according to different input modalities.

[0068] (3) For the extracted spatial temporal features, a meta-learning task sampling method is designed according to the different skin color, race and scene of the face, and a temporal localization model is constructed at the same time.

[0069] (4) Use the sampling meta-learning task to perform meta-training of the temporal localization model, so that the model can learn to extract spatiotemporal feature representations with generalization ability.

[0070] (5) Use a small number of samples in the target domain to quickly adapt the model to new scenarios.

[0071] The features and specific steps of the face registration method and size uniformity requirements involved in step (1) are as follows:

[0072] ① Use face detection tool libraries (including but not limited to Dlib, Openface, and Retinaface) to detect facial landmarks in template faces and the first frame of each video.

[0073] ② Determine whether to remove some points of the face outline according to the specific alignment requirements, and use all / remaining key points to calculate the affine matrix for face alignment;

[0074] ③ Since key point detection always has slight jitter, performing key point detection and transformation frame by frame will cause "sub-pixel jitter" between adjacent frames. Therefore, it is stipulated that the remaining subsequent frames reuse the affine matrix of the first frame and the template face to achieve alignment stability.

[0075] ④ After alignment, crop the image by 5-10 pixels along the top, bottom, left, and right boundaries of the key points to ensure that the edges of the key parts are not lost while cutting off the excess parts. Correct each frame to the same size and normalize it.

[0076] Furthermore, regarding the video understanding model downsampling feature extraction involved in step (2), the model, parameter selection range, and specific process are as follows:

[0077] ① Given a specified feature length T for downsampling, extract one D-dimensional feature per w frames, with a sampling step size of s. Then, the frame-level segment length t required for sampling features of length T is t = w + (T - 1) * s. A video of length t is slidably extracted from the original video, with a sampling step size of t / 2.

[0078] ② For videos with different frame rates, frame extraction can be used to unify them to a certain frame rate, or the values ​​of w and s can be scaled proportionally to obtain features with similar temporal representation ranges.

[0079] ③ Depending on the input modality, there are two technical approaches:

[0080] a. If a single modal input (RGB frame sequence) is used, a pre-trained encoder of a single modal input model such as VideoMAEv1 / v2 is used for feature extraction. This model does not require optical flow modality. Its input tensor shape is [t,3, w, H, W] and its output is [t, D].

[0081] b. If a dual-modal input of RGB + optical flow is used, a dual-stream video understanding model is employed for processing. RGB and optical flow features are extracted separately. Dense optical flow is extracted for each adjacent frame, and the average optical flow within pixels surrounding the nose center keypoint is subtracted to eliminate errors caused by facial or image jitter. Simultaneously, the last frame of the RGB sequence is removed to maintain frame consistency. The input shape is RGB: [T, 3, w, H, W], optical flow: [T, 2, w, H, W], and the output is features of [T, D]. Finally, the dual-stream features are concatenated along the time dimension T to obtain a fused feature of [T, 2D].

[0082] The meta-learning method, the design of meta-learning task sampling, the adaptive framework adopted, and the structure of the localization model used in steps (3) and (4) have the following characteristics and processes.

[0083] 1) The data samples are divided into two parts: one part is used for training the basic model of meta-learning, and the other part is used for rapid adaptive fine-tuning in specific scenarios. For this invention, the samples to be divided are pre-extracted features, and n non-overlapping tasks are sampled from the feature samples. To perform meta-learning, each task is actually a set of sample features, and each task's sample set is divided into a support set. and query set .

[0084] The training process for meta-learning is as follows: the initial model parameters are... This model is used as the target model in the outer meta-learning loop, and it does not participate in the gradient update of the inner loop. This model is copied n times as the inner loop model for n tasks. In the inner loop, for each task... The model performs k gradient updates on the support set for this task:

[0085]

[0086] in, For the first The task model The parameters are updated by the subgradient. The learning rate for the inner loop can be different for each task.

[0087] Then, use query sets. To test the model and calculate The corresponding gradients. The gradients of the query set do not participate in the update of the inner loop model; instead, the sum of the gradients of the query sets from n tasks is used to update the target model of the outer loop.

[0088]

[0089]

[0090] in, This is the sum of the gradients of all task query sets on the corresponding model. The learning rate for the outer loop target model. For the first The parameters of a task model after k gradient descent operations.

[0091] 2) Furthermore, for the meta-learning task required in the meta-learning training in 1) above, its construction method can be selected as follows:

[0092] a. Divide the task sets according to the duration of different facial expression intervals. The duration of facial expressions is a fundamental characteristic, and their temporal characteristics and motion patterns differ significantly, ranging from fleeting micro-expressions (usually less than 0.5 seconds) to macro-expressions (usually lasting several seconds). Dividing them according to duration allows the model to learn to understand and process facial expression dynamics at different time scales.

[0093] b. Divide the task sets according to different ethnic facial features or ages. By explicitly using different demographic attributes (such as ethnicity, age, and gender) as the basis for task division, the model is forced to learn more intrinsic and generalized facial expression semantics that transcend these surface attributes. This classification method can alleviate the dataset bias problem that is common in face analysis.

[0094] c. Divide the task sets according to the intensity category distribution of facial expression intervals. This method focuses on the intensity distribution of facial expressions and can be divided into samples with only micro-expression intervals, macro-expression intervals, and samples with both, to enhance adaptability to different intensity distributions.

[0095] 3) The specific architectural details and explanations regarding the characteristics of the positioning model structure and the positioning process are as follows:

[0096] a. The model consists of a one-dimensional convolutional mapping layer, a multi-scale Transformer encoder, a classification head, and a boundary regression head.

[0097] b. The mapping layer mainly maps input features of different dimensions (the extracted feature dimensions are generally different depending on the selected feature extraction model or the model size) to a fixed dimension, which can improve the versatility of the model.

[0098] c. The multi-scale Transformer encoder consists of a non-downsampling layer and a downsampling layer. The non-downsampling layer does not change the size of the time dimension, while the downsampling layer downsamples the time dimension by a factor of two to output multi-scale temporal features. The non-downsampling layer consists of a feature embedding layer, a multi-head self-attention layer, and an FPN layer; the downsampling layer adds a downsampling module to this structure.

[0099] d. The classification head and regression head are 1D convolutional subnetworks with shared weights. The input to the classification head and boundary regression head corresponding to layer l is [batch, T / 2]. l - 1 D e ], where D e The embedding dimension of the encoder; the output is [batch, T / 2]. l - 1 [2], where the 2 in the classification head represents the probability of the macro-micro expression interval at each point on the current layer time coordinate, and the regression head represents the distance from each point on the current layer time coordinate to the boundary.

[0100] For step (5), a small number of samples are used to quickly fine-tune the model in the target domain, enabling the model to quickly adapt to the new scene. The specific details are as follows:

[0101] ① Using the weights of the meta-basic model trained in step (4), and a small number of samples from the target domain, the weights are quickly and adaptively fine-tuned, and then tested.

[0102] ② During testing, if the Interchange of Value (IOU) between the predicted result and the true expression is greater than or equal to 0.5, then the prediction is true.

[0103] The following details the changes to the labels during testing and throughout the entire process:

[0104] Category tags: Micro-expressions: 0, Macro-expressions: 1;

[0105] Range labels:

[0106] a. Initial: Frame-level labels [t1, t2]

[0107] b. During feature pre-extraction: The feature time axis labels [T1, T2] are mapped as follows:

[0108]

[0109]

[0110] c. When calculating the loss within the model: feature point T f Distance label to boundary [T] f – T1, T f – T2] Boundary fitting is performed only if the feature point falls within the interval.

[0111] The macro-micro expression interval localization method based on meta-learning, as described above.

[0112] Example 1

[0113] This embodiment uses single-modal RGB video frames (30 frames / s) as the initial input and uses VideoMAEv2 to extract segment-level video spatiotemporal features. The specific steps are as follows:

[0114] ① Using the method described in step (1), perform face registration for video frames, fix the image size to 224*224, and normalize it using the image / 255.0 method;

[0115] ② Set the feature length T extracted by downsampling to 128, the number of window frames w for a single feature to be extracted to 9, the window step size s to 3, and the required video segment length t = 9 + (128-1)*3 = 390;

[0116] ③ Construct the vit_giant_patch14_224 version of VideoMAEv2, whose extracted feature dimension is 1408. Use the pre-trained model weights of vit_g_hybrid_pt_1200e_k710_ft.pth to load the encoder of VideoMAEv2 as the feature extractor;

[0117] ④ The input tensor shape of the extractor is [4, 3, 9, 224, 224] (batch == 4, which can be adjusted according to the device computing power and is not fixed), the output tensor is [4, 1408], and after iterative extraction and concatenation, it becomes [128, 1408]. This tensor is the input of the subsequent expression localization model;

[0118] ⑤ Before inputting into the model, filter out the samples without complete or partial intervals of expressions, and map the feature-level time axis of the frame-level interval and truncate it to the effective range;

[0119] ⑥ Divide the prepared feature samples into n categories according to different faces. Each time, the number of sampled categories n_way < n, the number of tasks n_task for each forward sampling is 3 - 5, the number of support sets n_sup sampled for each task is 2 - 5, the query set n_query is 5 - 10, the number of iterations for a single inner loop is 5, the learning rate of the inner loop model is 0.0001, the number of iterations on the support set for each inner loop is 30. After the iteration, calculate the loss of the query set, and the sum of the losses is used as the loss of the outer loop. The learning rate of the outer loop is the same as that of the inner loop, and the number of iterations of the outer loop model is 20.至此, the meta-learning training stage ends;

[0120] ⑦ Take a small number of samples from the target domain dataset for ⑤ - ⑩ rounds of fast fine-tuning adaptation, and then you can perform testing.

[0121] More specifically, for the structural parameters of the localization model and the data involved in the first embodiment, further explanations are as follows:

[0122] a. The number of mapping layers of the model, the number of non-downsampling Transformer encoder layers, and the configuration of the downsampling encoder layers is [2, 2, 5]. The embedding dimension of the Transformer block is 512, and the dimension of the internal FPN is 512, that is, no dimension increase or decrease is performed, and the classification head and regression head are 1D convolutional sub-networks with shared weights;

[0123] b. Data Flow: Taking a single iteration of the inner loop model on the support set during meta-training as an example, let the simplest batch = n_way*n_task*n_sup, with an input tensor of [batch, 128, 1408]. First, the tensor is expanded by masking the temporal dimension to [batch, 192, 1408]. After passing through a mapping layer, the dimension is increased from 1408 to 512. The dimensions entering and exiting the Transformer encoder remain unchanged, but the temporal feature dimensions output by the downsampling encoder layers are halved sequentially to 192, 96, 48, 24, 12, 6. The input of the classification head and boundary head corresponding to the l-th layer is [batch, 192 / 2]. l - 1 The output is [batch, 512], which is 192 / 2. l - 1 [2], where the 2 in the classification head represents the probability of the macro-micro expression interval at each point on the current layer time coordinate, and the regression head represents the distance from each point on the current layer time coordinate to the boundary.

[0124] c. For the large number of preliminary prediction results, non-maximum suppression (NMS) is used to retain overlapping prediction intervals and remove prediction results with low confidence scores. The confidence score threshold is set to 0.5. The prediction results are feature-level intervals. The feature-level intervals are mapped to the original frame-level intervals and IOU is calculated with the true value interval. If IOU>=0.5, the prediction is considered true.

[0125] Example 2

[0126] This embodiment uses high frame rate video (200 frames / s) as input and the VideoMAEv2 model as the feature extraction model. The specific process is as follows:

[0127] ① The process is basically the same as step one of embodiment one, the only difference being that, due to the high frame rate during alignment, the video can be cut into several segments and aligned separately to reduce the error of long sequence alignment. Alternatively, the frame extraction method in step two below can be adopted before alignment;

[0128] ② For high frame rate videos, in order to obtain a feature temporal representation that is similar to that of a normal 30 frames / second video, the following two methods can be used, including but not limited to:

[0129] a. By adopting a proportional frame extraction method, the high frame rate video is extracted into 30 frames. For every 20 frames, 3 frames can be extracted at intervals of 7, 7, and 6 frames. At the same time, the corresponding frame index mapping is performed on the frame level interval, and the frame level labels are reconstructed.

[0130] b. By using larger w and s, w can be proportionally enlarged to the range of 3 to 50 frames, and s can also be enlarged accordingly. In this way, an approximate feature temporal representation can also be achieved.

[0131] ③ The remaining steps correspond to steps three and thereafter in Example 1.

[0132] Example 3

[0133] This embodiment focuses on the scenario where training does not start from scratch, that is, starting from a pre-trained meta-based model, simulating rapid adaptive fine-tuning under specific scenarios and with limited samples. The specific process is as follows:

[0134] ① The meta-basic model trained in Example 1 is used to load the corresponding model weights;

[0135] ② The sample preprocessing procedures are the same as those described in Example 1;

[0136] ③ For cases with few samples, 1-5 samples can be used to fine-tune the weights in several rounds of the normal training process. This process has almost no time cost.

[0137] ④ The finely tuned model weights can adapt well to the expression localization in the current scene.

[0138] Example 4

[0139] This embodiment uses two modalities of input: RGB frames (30 frames / s) + optical flow. Dual-stream I3D is used for feature extraction. The specific process is as follows:

[0140] ① The processing method for RGB frames is the same as step one in Example 1, the only difference being that the last frame is removed to keep the number of optical flow maps consistent;

[0141] ② For optical flow extraction, the classic TVL1 method is used to extract dense optical flow in adjacent frames. In order to eliminate the influence of frame jitter or sudden changes in facial position, the average optical flow within 6 pixels around the nose tip key point is subtracted from the overall optical flow.

[0142] ③ The extracted feature length and other parameters are the same as in step two of Example 1;

[0143] ④ Using the open-source dual-stream I3D model, along with RGB branch weights and optical flow branch weights, the input shapes are [16, 3, 9, 224, 224] and [16, 2, 9, 224, 224], and the output shape is [16, 1024]. First, they are concatenated by dimension to [16, 2048] (other feature fusion methods can be used here), and then they are stacked in batches to [128, 2048].

[0144] ⑤ The fragment selection method is the same as step five in Example 1;

[0145] ⑥ For the prepared feature samples, they are divided into n classes according to different segment durations. The number of classes sampled each time is n_way < n. The number of tasks sampled in each forward pass is n_task, which is 3~5. The number of support sets sampled for each task is n_sup, which is 2~5. The number of query sets is n_query, which is 5~10. The number of iterations in a single inner loop is 5. The learning rate of the inner loop model is 0.0001. The number of iterations on the support set in each inner loop is 30. After each iteration, the loss of the query set is calculated, and the sum of the losses is used as the loss of the outer loop. The learning rate of the outer loop is the same as that of the inner loop. The number of iterations of the outer loop model is 20. At this point, the meta-learning training phase ends.

[0146] The subsequent process is basically the same as step seven of Example 1. The difference is that the input dimension of the mapping layer needs to be adjusted to 2048 to meet the requirement of changing the input dimension.

[0147] In summary, this invention proposes a meta-learning-based macro-micro-expression interval localization method. Its core idea is to employ meta-learning to simulate the varied task distributions in real-world applications (e.g., tasks are divided according to expression duration, ethnicity, age, or intensity distribution), enabling the model to learn cross-domain generalization capabilities—not just features from a single dataset—during training, but rather how to quickly adapt to new tasks. This method first performs standardized face alignment and preprocessing on the input video. Then, based on the input modality (RGB or RGB+optical flow), a high-performance pre-trained model (such as VideoMAE or dual-stream I3D) is selected to extract robust temporal features. Next, a meta-learning task is constructed based on different attribute dimensions to meta-train the multi-scale Transformer temporal localization model, resulting in a meta-based model with strong generalization capabilities. Finally, in the target scene, only a very small number of samples are needed for rapid fine-tuning, allowing the model to quickly adapt to new environments and achieve high-precision, highly robust macro-micro-expression interval localization.

[0148] This invention effectively solves the problems of insufficient generalization ability and reliance on a large amount of labeled data in existing methods when facing unknown scenarios, and provides reliable technical support for the widespread application of facial expression analysis in practice.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A meta-learning-based macro-micro expression interval positioning method, characterized in that, The method comprises the following steps: performing face alignment and image size uniform processing on an input face expression video frame sequence; performing down-sampling on the processed face expression video frame sequence based on a pre-trained video understanding model to extract time sequence features; based on the time sequence features, dividing and sampling a meta-learning task according to a preset attribute dimension, and simultaneously constructing a time sequence positioning model; performing meta-learning training on the time sequence positioning model according to the meta-learning task to obtain a meta-basic model having a general spatio-temporal feature representation capability; performing adaptive fine-tuning on the meta-basic model, and using the fine-tuned model to position a macro-expression and a micro-expression interval of a target domain face expression video.

2. The method of claim 1, wherein, The process of performing face alignment and image size uniform processing on an input face expression video frame sequence comprises: detecting key points of a template face and a first frame face of an input video by using a face key point detection tool library; calculating a transformation matrix based on the face key points to complete alignment of the template face and the first frame image; performing alignment operation on all subsequent frames of the video sequence by using the affine transformation matrix calculated for the first frame; extending a plurality of pixels outward according to the outermost key points to perform cropping, so that the edge part of the key area is included, and the image is scaled to the same size.

3. The method of claim 1, wherein, The process of extracting time sequence features comprises: calculating the length of a required input video segment; for videos with different frame rates, using frame extraction to obtain features with close time sequence representation ranges; extracting video segments from the original video by using a sliding window method; extracting features from the extracted video segments; the feature extraction is performed according to the input modalities by using the following two methods, but not limited to the following two methods: when the input is an RGB modality, a pre-trained encoder of a single modality video understanding model is used for processing, the input tensor shape is [t, 3, w, H, W], and the output is a feature with a shape of [t, D]; when the input is an RGB and optical flow dual modality, a dual flow video understanding model is used for processing, RGB flow and optical flow features are extracted, dense optical flow is extracted for each adjacent frame, the average optical flow in the pixels around the nose center key point is subtracted, and the last frame of the RGB sequence is removed, the input shape is RGB: [T, 3, w, H, W] and optical flow: [T, 2, w, H, W], the output is a feature with a shape of [T, D], and finally the dual flow features are spliced in the time dimension T to obtain a fusion feature with a shape of [T, 2D].

4. The method of claim 1, wherein, The process of dividing and sampling a meta-learning task according to a preset attribute dimension comprises: sampling a plurality of non-overlapping tasks from the time sequence features based on the preset attribute dimension for meta-learning, each task being a group of sample features, and each group of sample features being divided into a support set and a query set; the preset attribute dimension comprises one or more of the following division methods: division according to expression interval duration: dividing samples into different task sets according to the duration interval of the expression; division according to face race or age: dividing samples into different task sets according to the race or age stage of the face. According to the intensity category distribution of the micro-expression / macro-expression or the mixture of the two, the sample is divided into different task sets.

5. The method of claim 1, wherein, The process of meta-learning training of the time sequence positioning model according to the meta-learning task comprises: Setting the parameters of an initial model, taking the initial model as a target model of meta-learning outer loop, copying n copies of the initial model as n inner loop models of tasks, in the inner loop, for each inner loop model of a task, performing k times of gradient update on the support set of the task: ， wherein, is the parameter of the i-th task model after the j-th gradient update, is the parameter of the i-th task model after the j-th gradient update, is the parameter of the i-th task model after the j-th gradient update, is the inner loop learning rate; Using the query set to test the model, calculating the gradient corresponding to the query set, and updating the target model of the outer loop using the sum of the gradients of the query sets of the n tasks: ， ， wherein, is the sum of the gradients on the corresponding model for all task query sets, is the outer loop target model learning rate, is the parameter of the task model after k gradient descent. th task model after k gradient descent.

6. The method of claim 1, wherein, The time sequence positioning model comprises: A one-dimensional convolution mapping layer for mapping input features to a fixed dimension, the dimension size being consistent with the dimension of the feature embedding layer; A multi-scale Transformer encoder composed of multiple levels, including a non-downsampling layer without downsampling and a downsampling layer with two times of downsampling, for extracting multi-scale time sequence feature representation; A classification head and a boundary regression head, which are one-dimensional convolution subnetworks sharing bottom weights, respectively used for predicting the probability of each time point belonging to an expression interval and the distance of the point to the interval boundary.

7. The method of claim 6, wherein, The non-downsampling layer in the multi-scale Transformer encoder comprises, in sequence, a feature embedding layer, a multi-head self-attention mechanism layer, and a feature pyramid network layer. The downsampling layer increases a down-sampling module on the basis of the structure of the non-downsampling layer.

8. The method of claim 6, wherein, The input of the classification head and the boundary regression head corresponding to the first layer is [batch, T / 2 l - 1 , D e ], wherein D e is the embedding dimension of the encoder; the output is [batch, T / 2 l - 1 , 2], wherein the 2 dimensions of the classification head represent the probability of each macro-micro expression interval at the current layer time coordinate, and the regression head represents the distance from each point at the current layer time coordinate to the boundary.

9. The method of claim 1, wherein, The adaptive fine-tuning refers to using the labeled samples of the target domain to perform a small number of gradient descent updates on the weights of the meta-base model.

10. The method of claim 1, wherein, In the process of adaptive fine-tuning of the meta-base model, the labels are converted according to the following rules: Category label definition: the micro-expression category label is 0, and the macro-expression category label is 1; Interval label conversion: The initial label is a frame-level interval [t1, t2]; After feature extraction, it is converted into an interval [T1, T2] on the feature time axis, and the conversion formula is: ， ， When computing the loss inside the model, convert to feature point T f distance label [T f - T1, T f - T2] and only participate in the computation of the boundary regression loss if the feature point T f is inside the interval [T1, T2].

Images