Efficient multi-modal micro-expression recognition system based on deep learning model

Abstract

This invention relates to the field of multimodal micro-expression recognition technology, specifically to an efficient multimodal micro-expression recognition system based on a deep learning model. The system comprises: a data synchronization module that captures and synchronizes visual images and bioelectrical signals in parallel; a feature distillation module that extracts sensitive features from the aligned data stream; an adaptive fusion module that constructs a dynamic topology graph using feature sensitivity, assigns fusion weights, and generates context-sensitive fusion features; a segmentation recognition module that drives adaptive segmentation and boundary labeling of micro-expression fragments, and performs collaborative inference through a cascaded classifier to generate preliminary recognition labels; and a system optimization module that updates the screening threshold of the feature distillation network based on the inference results, integrates the updated network and classifier, and performs online unsupervised parameter fine-tuning to enable the system to continuously adapt to new input data streams. This invention achieves adaptive fusion of multimodal features and online self-optimization of the system.

Description

Technical Field

[0001] This invention relates to the field of multimodal micro-expression recognition technology, and in particular to an efficient multimodal micro-expression recognition system based on a deep learning model. Background Technology

[0002] Currently, micro-expression recognition systems primarily rely on the analysis of a single visual modality, and their recognition accuracy is significantly affected by environmental factors such as lighting and occlusion. To improve recognition stability, some studies have introduced multimodal physiological signals such as electrocardiograms (ECG) and electromyograms (EMGs) as auxiliary methods. Existing technologies typically employ simple timestamp-based alignment strategies when processing multimodal data. In the feature fusion stage, pre-set fixed weights or simple attention mechanisms are commonly used for weighted superposition. This static fusion method ignores the dynamic differences in the quality and discriminative contribution of each modality signal under different scenarios and individuals, resulting in limited robustness of the system in complex real-world environments.

[0003] Most existing systems employ a static "train-deployment" model. Model parameters remain fixed once offline training is complete, failing to address data drift caused by changes in data distribution over time or across different scenarios. Traditional solutions require manual data relabeling and complete model retraining, which is costly and inefficient. This lack of adaptability and ability to adjust to new data during continuous operation limits the practicality and long-term effectiveness of existing systems. This invention aims to overcome these shortcomings by constructing a micro-expression recognition system capable of dynamic adaptive fusion of multimodal features and online self-optimization. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an efficient multimodal micro-expression recognition system based on a deep learning model.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a high-efficiency multimodal micro-expression recognition system based on a deep learning model, comprising: The data synchronization module initiates a parallel data acquisition process, synchronously acquires visual images and bioelectric signals, forms an initial multimodal data stream, performs intermodal synchronization phase-locked loop processing, and performs temporal and spatial alignment correction on the initial multimodal data stream. The feature distillation module inputs the aligned and corrected data stream into the hierarchical feature distillation network, which peels off and filters out sensitive features related to micro-expressions layer by layer. The adaptive fusion module constructs an adaptive feature topology graph using feature sensitivity, dynamically allocates fusion weights for different modal features, guides the feature fusion process through the weight allocation results, and generates fusion feature representations with contextual relevance. The segmentation recognition module, based on the fusion feature expression-driven adaptive segmentation and boundary labeling of micro-expression fragments, sends the labeled fragments into a cascaded classifier group for collaborative reasoning to generate preliminary recognition labels; The system optimization module updates the screening threshold of the hierarchical feature distillation network based on the collaborative inference results, integrates the updated network with the cascaded classifier group to form a complete micro-expression recognition system, performs online unsupervised parameter fine-tuning of the system, and continuously adapts to new input data streams.

[0006] As a further aspect of the present invention, the initiation of the parallel data capture process includes: Equipped with a high frame rate visible light camera and an infrared thermal imager, video images of the facial area are recorded at the same sampling rate; Simultaneously deploy contact-type bioelectric sensors to collect physiological signals, including electromyography and skin conductivity. Inject millisecond-level timestamps into each frame of visual imagery and each point in time with bioelectrical signals. The video image sequence is initially bound to the physiological signal sequence based on the timestamp; Face detection and pose estimation are performed on the bound video image sequence to extract stable facial regions of interest; The extracted facial regions of interest are subjected to illumination-invariant transformations to generate standardized visual data blocks. Baseline drift removal and power frequency noise filtering are performed on the bound physiological signal sequences to generate clean physiological data sequences; Standardized blocks of visual data are encapsulated together with clean sequences of physiological data to form an initial multimodal data stream.

[0007] As a further aspect of the present invention, the inter-modal synchronization phase-locked loop processing includes: Analyze the inter-frame motion vectors of visual data blocks and the transient change points of physiological data sequences in the initial multimodal data stream; Using significant transient change points in physiological data sequences as temporal reference benchmarks, the relative delay of visual motion vectors is calculated; Establish a time delay compensation function to perform frame-level time offset correction on visual data blocks; Based on temporal alignment, the spatial correspondence between facial regions of interest and physiological signal sources is mapped. Based on spatial correspondence, assign corresponding physiological signal components to each facial sub-region; By fusing time-aligned visual data with spatially mapped physiological signals, a spatiotemporally consistent multimodal data matrix is ​​generated; To verify the synchronicity of signals from different channels in a multimodal data matrix at key event points; If the synchronization does not reach the preset threshold, the delay compensation function and spatial mapping relationship are iteratively adjusted until the data matrix completes alignment correction.

[0008] As a further aspect of the present invention, the step of inputting the aligned and corrected data stream into the hierarchical feature distillation network includes: Construct a hierarchical feature distillation network consisting of multiple stacked feature filtering layers, each layer containing a feature extractor and a sensitivity estimator; The aligned and corrected multimodal data matrix is ​​input into the first layer feature extractor of the network to extract preliminary edge and texture features; The sensitivity evaluator of the first layer calculates the mutual information between each preliminary feature and the known micro-expression sample library to obtain the feature sensitivity score; Based on the feature sensitivity score, redundant features with scores below the threshold are eliminated, and the filtered features are passed to the next layer. In each subsequent layer of the network, the incoming features are subjected to higher-order abstraction and combination; Each layer's sensitivity evaluator recalculates the correlation strength between the current layer and micro-expressions based on the abstract features of that layer. After all levels of filtering, a refined set of sensitive features highly correlated with micro-expressions is finally output. Record the features eliminated at each layer and their sensitivity scores to form a feature distillation path log.

[0009] As a further aspect of the present invention, the method of constructing an adaptive feature topology graph using feature sensitivity includes: Read the set of sensitive features output by the hierarchical feature distillation network and its corresponding feature sensitivity scores; Each sensitive feature is treated as a node in a topological graph, and the weight of the node is its feature sensitivity score. Calculate the correlation coefficient between any two sensitive feature nodes in the multimodal data matrix; Using the correlation coefficient as the weight of the edge, two feature nodes are connected to initially construct a fully connected feature topology graph; Based on the weights of nodes and edges, a graph pruning algorithm is applied to remove nodes and edges with too low weights and sparse connections to the main subgraph. Community detection was performed on the pruned feature topology to identify feature communities with tight internal connections. Assign a community identifier to each feature community and calculate the overall sensitivity weight of the feature community; Based on the community identifier and the overall sensitivity weight, the final adaptive feature topology graph structure description is generated.

[0010] As a further aspect of the present invention, the step of guiding the feature fusion process through weight allocation results includes: Analyze the adaptive feature topology graph structure description to obtain the overall sensitivity weights of different feature clusters; A normalization method is used to transform the overall sensitivity weight into the initial contribution coefficient of each feature cluster in the fusion process; For feature nodes within the same cluster, the internal fusion ratio is allocated according to their node weights; Design a feature fusion function that uses feature clusters as units and weights and aggregates the original sensitive features according to the initial contribution coefficient and the internal fusion ratio. In the weighted aggregation process, a cross-community feature interaction term is introduced, which is determined by the weight of the edges connecting different communities. Execute the feature fusion function to generate a unified fused feature vector. The fused feature vector is input into a lightweight feedforward network, where it undergoes nonlinear transformation and dimensionality adjustment to output a fused feature representation with contextual relevance. The fused feature representation simultaneously encodes the importance of features, community structure, and cross-modal association information.

[0011] As a further aspect of the present invention, the adaptive segmentation and boundary labeling of micro-expression fragments driven by fusion feature representation includes: Construct a sliding time window that moves continuously over the time series of fused feature representations with contextual relevance; At each window location, calculate the rate of change of statistical properties of the fused feature representation within the window; When the rate of change exceeds the dynamically set segmentation threshold, the time point is marked as the potential start or end point of a micro-expression event. Starting from the marked potential points, the analysis is expanded in both forward and backward directions to find local extreme points of characteristic changes in order to correct the boundary. Based on the corrected boundary points, the corresponding data segments are extracted from the original multimodal data matrix to form candidate micro-expression fragments; For each candidate micro-expression fragment, extract its short-term trajectory pattern expressed by fused features; The internal consistency score of candidate micro-expression segments is calculated based on trajectory patterns to filter out false segments with low internal consistency. The selected candidate micro-expression fragments and their start and end timestamps are output as a calibrated fragment set.

[0012] As a further aspect of the present invention, the step of sending the calibrated fragments into a cascaded classifier group for collaborative reasoning includes: Configure a cascaded classifier group consisting of multiple heterogeneous classifiers, each classifier focusing on recognizing feature patterns in different dimensions; Multiple complementary feature views are extracted from the calibrated fragment set, including a global statistical view, a local dynamic view, and a temporal evolution view; The global statistical view is input into the support vector machine classifier in the cascaded classifier group to obtain a preliminary probability distribution vector; The local dynamic view is input into the random forest classifier in the cascaded classifier group to obtain another probability distribution vector; The temporal evolution view is input into the temporal convolutional network classifier in the cascaded classifier group to obtain the third probability distribution vector; Design a meta-decision fusion module that receives three probability distribution vectors; The meta-decision fusion module learns the reliability weights of each classifier under different feature conditions; The integrated probability distribution is obtained by weighting the three probability distribution vectors according to the reliability weights. The category with the highest probability is selected from the ensemble probability distribution as the initial identification label for micro-expression segments; The initial identification labels and their corresponding integrated probability distributions are output together.

[0013] As a further aspect of the present invention, the filtering threshold for updating the hierarchical feature distillation network based on the collaborative reasoning results includes: Collect the preliminary identification labels and their ensemble probability distributions output by the cascaded classifier group over a period of time; Calculate the discriminant index of each sensitive feature retained by the hierarchical feature distillation network under different identification labels; A new feature utility score is generated by combining the feature's discriminative index with its original feature sensitivity score. Analyze the feature distillation path log to identify features that were eliminated but associated with samples of high ensemble probability distribution; Establish a feedback loop to transmit the feature utility score and the information of the eliminated features collected back to the sensitivity evaluator of the hierarchical feature distillation network. The sensitivity evaluator dynamically adjusts its internal parameters for calculating mutual information and the threshold for feature elimination based on feedback information. The updated sensitivity evaluator will use a new threshold to filter features in subsequent data processing; This process enables the hierarchical feature distillation network to adaptively optimize its feature selection strategy, focusing on features that are more effective for the current recognition task.

[0014] As a further aspect of the present invention, the online unsupervised parameter fine-tuning of the execution system includes: After the complete micro-expression recognition system is put into operation, it continuously captures and caches new input multimodal data streams; For new cached data streams, use the current system to generate recognition results, but do not use them as truth labels; By applying temporal consistency constraints and intermodal consistency constraints, the credibility of the system's recognition results on new data is evaluated. High-confidence identification results that meet the consistency constraints are selected and treated as pseudo-label data; Construct an online fine-tuning dataset using pseudo-labeled data; Employ unsupervised or self-supervised learning objectives, such as contrastive learning or reconstruction learning, and define an online fine-tuning loss function; Without interrupting system services, the parameters of the hierarchical feature distillation network and cascade classifier group in the system are fine-tuned using mini-batch gradient descent with online fine-tuning dataset and loss function. Monitor the performance differences of the system on a fixed validation set before and after fine-tuning to prevent performance drift; By iteratively executing capture, evaluation, filtering, and fine-tuning steps, the system is able to continuously adapt to changes in the distribution of the input data stream.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: By utilizing feature sensitivity metrics extracted from hierarchical feature distillation networks, an adaptive feature topology graph is dynamically constructed. This graph analyzes the discriminative contribution of each modality feature in real time based on the input data and dynamically allocates fusion weights accordingly, changing the fusion mode of fixed weights or simple attention mechanisms. This mechanism enables the system to adaptively adjust the fusion strategy according to specific samples. When the signal of one modality is disturbed or of poor quality, it automatically enhances the decision weight of another reliable modality, thereby achieving more robust and accurate multimodal feature representation in complex application scenarios.

[0016] A closed-loop system optimization process was designed, utilizing the collaborative inference results of cascaded classifier groups to update the feature selection threshold in the feature distillation network. This process requires no manual intervention or additional annotation, achieving unsupervised online parameter fine-tuning. While completing the recognition task, the system can integrate the updated network components and continuously adapt to the feature distribution of new input data streams. This frees the system from dependence on periodic offline retraining, enabling it to self-optimize and evolve over long-term operation, improving its adaptability to new environments and objects, and enhancing its performance stability throughout its lifecycle. Attached Figure Description

[0017] Figure 1 This is a timing diagram of the efficient multimodal micro-expression recognition system based on a deep learning model as described in this invention. Figure 2 A flowchart for parallel data capture; Figure 3 A flowchart illustrating how a hierarchical feature distillation network works; Figure 4 A bar and line graph comparing the probability output of the cascaded classifier with the ensemble probability; Figure 5 This is a graph showing the trend of validation set accuracy and contrast loss as a function of iteration rounds during the online parameter fine-tuning stage. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] See Figure 1 The data synchronization module initiates a parallel data acquisition process, simultaneously acquiring visual images and bioelectrical signals to form an initial multimodal data stream. It then performs intermodal synchronization phase-locked loop processing to align and correct this data stream in both the temporal and spatial domains. The corrected data stream is then fed to the feature distillation module, which uses a hierarchical feature distillation network to peel away and filter out sensitive features related to micro-expressions layer by layer. The adaptive fusion module constructs an adaptive feature topology graph using feature sensitivity, dynamically assigns fusion weights to different modal features, and guides the feature fusion process through the weight allocation results, generating a context-sensitive fusion feature representation. The segmentation recognition module drives adaptive segmentation and boundary labeling of micro-expression fragments based on this fusion feature representation, and sends the labeled fragments to a cascaded classifier group for collaborative inference to generate preliminary recognition labels. The system optimization module updates the filtering threshold of the hierarchical feature distillation network based on the collaborative inference results, integrates the updated network with the cascaded classifier group to form a complete micro-expression recognition system, and performs online unsupervised parameter fine-tuning to continuously adapt to new input data streams.

[0021] See Figure 2In one embodiment of the present invention, the specific operation of the data synchronization module includes initiating a parallel data acquisition process and performing inter-modal synchronization phase-locked loop processing. Initiating the parallel data acquisition process includes configuring a high frame rate visible light camera with a sampling rate of 200 frames per second and an infrared thermal imager with the same sampling rate. Both devices are aimed at the same facial area to record video images. The synchronously deployed contact bioelectric sensors include a surface electromyography (EMG) sensor and a skin conductivity sensor. The EMG sensor is attached to specific facial muscle groups, and the skin conductivity sensor is fixed to the fingertip. The bioelectric sensors acquire physiological signals at a sampling rate of 1000 Hz. In specific implementation, a timestamp accurate to milliseconds is injected into each frame of the visual image recorded by the visible light camera and the infrared thermal imager. Simultaneously, a timestamp of the same precision is injected into each physiological signal data point acquired by the EMG sensor and the skin conductivity sensor. Based on the timestamps, the visible light video image sequence, the infrared thermal imaging video image sequence, the EMG signal sequence, and the skin conductivity signal sequence are initially bound together to form a preliminary bound dataset indexed by time. Face detection and head pose estimation are performed on visible light and infrared video image sequences in the initial binding dataset, extracting pose-normalized facial regions of interest (ROIs). Illumination-invariant transformations are applied to the extracted ROIs, specifically using histogram specification to map the images to standard illumination conditions, generating standardized visual data blocks. Baseline drift removal and power frequency noise filtering are performed on the electromyography (EMG) and electrical conductivity (EDS) signal sequences in the initial binding dataset. Baseline drift removal uses high-pass digital filtering, and power frequency noise filtering uses notch filtering, generating clean physiological data sequences. Finally, the standardized visual data blocks and the clean physiological data sequences are aligned and encapsulated according to timestamps to form the initial multimodal data stream.

[0022] In some embodiments, performing intermodal synchronization phase-locked loop (PLL) processing requires in-depth analysis of the initial multimodal data stream. This involves analyzing the inter-frame motion vectors of standardized visual data blocks in the initial multimodal data stream, calculated using optical flow. Transient change points in the electromyography (EMG) and skin conductivity (SCT) signal sequences are analyzed, determined by the position where the first derivative of the detected signal exceeds a preset threshold. Using the most significant transient change point in the physiological data sequence as a temporal reference, the relative delay time of the peak value of the inter-frame motion vector of the visual data block relative to the physiological transient change point is calculated. A time delay compensation function is established to correct the frame-level time offset of the visual data block; this function is defined as a linear translation of the visual data block's timestamp based on the calculated relative delay time. Based on time alignment, a spatial correspondence mapping is established between the two-dimensional coordinates of the facial region of interest and the sensor positions of the physiological signal source. According to this spatial correspondence mapping, a set of corresponding EMG and SCT signal components is assigned to each facial sub-region divided by facial key points. By fusing visual data that has been corrected for time offset with physiological signal components that have been spatially mapped, a spatiotemporally consistent multimodal data matrix is ​​generated. The rows of the multimodal data matrix correspond to time points, and the columns correspond to feature channels of different modalities.

[0023] In practice, the synchronicity of different channel signals in the multimodal data matrix at key event points is examined. Synchronicity is evaluated by calculating the cross-correlation coefficient between the visual motion feature channel and the physiological signal feature channel within a time window near the key event point. If the maximum calculated cross-correlation coefficient is lower than a preset synchronicity threshold, the weight allocation in the translation parameter and spatial correspondence mapping of the time delay compensation function is iteratively adjusted. In each iteration, the relative delay time is recalculated and the matching weights of the facial sub-region and the physiological signal components are optimized. The fusion and synchronicity verification steps are repeated until the cross-correlation coefficient of the generated multimodal data matrix at key event points reaches or exceeds the preset threshold. At this point, the data matrix is ​​considered to have completed alignment correction.

[0024] See Figure 3In one embodiment of the present invention, the feature distillation module includes inputting the alignment-corrected data stream into a hierarchical feature distillation network and constructing an adaptive feature topology map using feature sensitivity. Inputting the alignment-corrected data stream into the hierarchical feature distillation network includes constructing a hierarchical feature distillation network consisting of five stacked feature filtering layers. Each feature filtering layer includes a convolutional neural network feature extractor and a mutual information-based sensitivity estimator. The alignment-corrected multimodal data matrix is ​​input into the first-layer feature extractor of the hierarchical feature distillation network. The first-layer feature extractor uses a 3x3 convolutional kernel to extract preliminary edge and texture features and generate 256 feature maps. The first-layer sensitivity estimator calculates the mutual information between each preliminary feature map and labeled samples in a known micro-expression sample library to obtain a feature sensitivity score for each feature map. The feature sensitivity score is calculated using the following formula:

[0025] Where: symbol S(f) i ) represents the i-th feature map f i Feature sensitivity score, symbol X i Representing the feature map f i The set of activation values ​​is denoted by Y, which represents the set of micro-expression categories. P(x,y) is the joint probability distribution of feature activation value x and category y, and p(x) and P(y) are the marginal probability distributions, respectively. Redundant feature maps with scores below a preset threshold of 0.05 are eliminated based on their feature sensitivity scores. The 128 filtered feature maps are then passed to the next layer of the hierarchical feature distillation network. In each subsequent layer of the network, the input feature maps undergo higher-order abstraction and combination; for example, the second-layer feature extractor uses a 5x5 convolutional kernel for feature combination. The sensitivity evaluator at each layer recalculates the mutual information between the abstract feature map of the current layer and the micro-expression category to assess the correlation strength. After all five layers of hierarchical feature filtering, a set of 64 refined sensitive features highly correlated with micro-expressions is finally output. The eliminated feature maps and their sensitivity scores at each layer are recorded to form a feature distillation path log. This log stores the identifiers and sensitivity scores of the eliminated features in matrix form.

[0026] In some embodiments, constructing an adaptive feature topology graph using feature sensitivity includes reading the set of sensitive features output by the hierarchical feature distillation network and their corresponding feature sensitivity scores. Each sensitive feature is considered as a node in the adaptive feature topology graph, and the weight of the node is its feature sensitivity score, S(f). iThe algorithm calculates the Pearson correlation coefficient between any two sensitive feature nodes in the multimodal data matrix and connects them using the Pearson correlation coefficient as the edge weight, thus initially constructing a fully connected feature topology graph. Based on the node and edge weights, a graph pruning algorithm is applied to remove nodes and edges with excessively low weights and sparse connections to the main subgraph. The graph pruning algorithm sets node weight thresholds of 0.01 and edge weights of 0.3. Community detection is then performed on the pruned feature topology graph, using the Louvain algorithm to identify tightly connected feature communities. A unique community identifier is assigned to each feature community, and the overall sensitivity weight of the feature community is calculated. The overall sensitivity weight is defined as the arithmetic mean of the feature sensitivity scores of all nodes within the community. Based on the community identifier and the overall sensitivity weight, a final adaptive feature topology graph structure description is generated, stored in the form of an adjacency list and a node attribute table. It can be understood that the hierarchical feature distillation network ensures the discriminative power of features through layer-by-layer filtering.

[0027] In one embodiment of the present invention, the adaptive fusion module and the segmentation recognition module include guiding the feature fusion process through weight allocation results and adaptive segmentation and boundary labeling of micro-expression segments driven by fused feature representation. Guiding the feature fusion process through weight allocation results includes parsing the adaptive feature topology graph structure description to obtain the overall sensitivity weights of different feature clusters, with overall sensitivity weights of 0.85, 0.72, and 0.61, respectively. A normalization method is used to convert the overall sensitivity weights into initial contribution coefficients for each feature cluster in the fusion process. The normalization method uses the softmax function, and the calculated initial contribution coefficients are 0.39, 0.33, and 0.28, respectively. For feature nodes within the same cluster, an internal fusion ratio is allocated according to their node weights. The internal fusion ratio is calculated based on the proportion of the node weight within the cluster. For example, if the weights of three nodes in a cluster are 0.4, 0.3, and 0.3, then the corresponding internal fusion ratios are also 0.4, 0.3, and 0.3. The feature fusion function is designed to weight and aggregate the original sensitive features using feature clusters as units, according to the initial contribution coefficient and the internal fusion ratio. A cross-cluster feature interaction term is introduced during the weighted aggregation process, determined by the weights of the edges connecting different clusters. Executing the feature fusion function generates a unified fused feature vector. The mathematical expression of the feature fusion function is as follows:

[0028] Where: symbol F v This represents the output fused feature vector, where K represents the number of feature clusters, and W represents the number of feature clusters. k C represents the initial contribution coefficient of the k-th characteristic community. k Let α represent the set of nodes belonging to the k-th feature community.i f represents the internal fusion ratio of node i within community k. i β represents the original sensitive feature corresponding to node i. ki The hyperparameter denoted by r represents the control of the intensity of cross-community interactions. kj The average weight of the edge connecting community k and the community to which node j belongs is represented by the symbol f. j This represents the original sensitive feature corresponding to node j outside community k. The fused feature vector F... v Input a feedforward network with two fully connected layers, perform nonlinear transformation and dimension adjustment, and output a context-sensitive fusion feature representation, which is a vector sequence with a dimension of 128.

[0029] In some embodiments, adaptive segmentation and boundary labeling of micro-expression fragments driven by fused feature representations include constructing a 15-frame sliding time window that moves continuously in 1-frame steps over the time series of fused feature representations with contextual relevance. At each window position, the statistical characteristic change rate of the fused feature representation within the window is calculated, obtained by calculating the variance change value of the 128-dimensional feature vector within the window. When the calculated variance change rate exceeds a dynamically set segmentation threshold, the time point is marked as a potential micro-expression event start or end point. The dynamically set segmentation threshold is adaptively adjusted based on the median of historical window change rates. Starting from the marked potential point, the analysis extends forward and backward by 10 frames each to find local extrema of feature changes to correct the boundary. Local extrema are determined by finding points where the first derivative of the variance change rate is zero. Based on the corrected boundary points, corresponding data segments are extracted from the original multimodal data matrix to form candidate micro-expression fragments. For each candidate micro-expression fragment, a short-time trajectory pattern of its fused feature representation is extracted. The short-time trajectory pattern is formed by concatenating the 128-dimensional vectors of each frame within the fragment after dimensionality reduction using principal component analysis. The internal consistency score of candidate micro-expression segments is calculated based on trajectory patterns. The internal consistency score is obtained by calculating the mean cosine similarity between the dimensionality-reduced features of all frames within a segment. False segments with an internal consistency score below 0.6 are filtered out. The filtered candidate micro-expression segments and their start and end timestamps are output as a calibrated segment set.

[0030] In one embodiment of the present invention, the segment recognition module and the system optimization module feed the calibrated segments into a cascaded classifier group for collaborative inference and update the screening threshold of the hierarchical feature distillation network based on the collaborative inference results. Feeding the calibrated segments into the cascaded classifier group for collaborative inference includes configuring a cascaded classifier group consisting of three heterogeneous classifiers: a support vector machine classifier, a random forest classifier, and a temporal convolutional network classifier. Each classifier focuses on recognizing feature patterns in different dimensions. Multiple complementary feature views are extracted from the calibrated segment set. These feature views include a global statistical view, a local dynamic view, and a temporal evolution view. The global statistical view is obtained by calculating the mean and variance of the fused feature expressions of all frames within the segment. The local dynamic view is obtained by calculating the difference in the fused feature expressions between adjacent frames within the segment. The temporal evolution view is obtained by inputting the sequence of fused feature expressions of the segment into a long short-term memory network encoder. The global statistical view is input into the support vector machine classifier in the cascaded classifier group. The support vector machine classifier uses radial basis function kernels and outputs a preliminary probability distribution vector covering seven micro-expression categories. The local dynamic view is input into the random forest classifier in the cascaded classifier group. The random forest classifier contains one hundred decision trees and outputs another probability distribution vector covering seven micro-expression categories. The temporal evolution view is input into the temporal convolutional network classifier in the cascaded classifier group. The temporal convolutional network classifier contains three convolutional layers and outputs a third probability distribution vector covering seven micro-expression categories.

[0031] In some embodiments, a meta-decision fusion module is designed to receive three probability distribution vectors from a support vector machine classifier, a random forest classifier, and a temporal convolutional network classifier. The meta-decision fusion module learns the reliability weights of each classifier under different feature conditions. These reliability weights are learned through a shallow neural network whose input is a concatenated vector of the three probability distribution vectors, and whose output is the weight values ​​of the three classifiers. A weighted average of the three probability distribution vectors based on the reliability weights is used to obtain an ensemble probability distribution. The ensemble probability distribution is calculated by summing the probability of each category as a weighted sum of the probabilities of the corresponding categories from the three classifiers. The category with the highest probability from the ensemble probability distribution is selected as the initial identification label for the micro-expression fragment, and the initial identification label and the corresponding ensemble probability distribution are output together. To illustrate the data flow during collaborative inference, an exemplary probability distribution output fragment is provided in Table 1.

[0032] Table 1: Probability Output and Fusion Results of Cascaded Classifier Groups for Example Segments

[0033] In Table 1, the "Learned Reliability Weights" column represents the weights assigned to each classifier by the meta-decision fusion module, the "Weighted Probability" column is the product of each classifier's probability and its corresponding weight, and the "Equilibrium Probability" column is the sum of all weighted probabilities for each category. Updating the screening threshold for the hierarchical feature distillation network based on collaborative inference results involves collecting preliminary recognition labels and their ensemble probability distributions from the outputs of the cascaded classifier groups over a period of time, such as collecting the recognition results of the most recent thousand micro-expression fragments. The discriminancy index of each sensitive feature retained by the hierarchical feature distillation network under different recognition labels is calculated using the Fisher discriminant ratio. A new feature utility score is generated by combining the feature's discriminancy index with its original feature sensitivity score. The formula for calculating the feature utility score is as follows:

[0034] Where: symbol U(g) m D(g) represents the feature utility score of the m-th sensitive feature gm, denoted by D(g). m ) represents the sensitive feature g m The discrimination index, denoted by S(g) m ) represents the sensitive feature g m The original feature sensitivity score, with the symbol λ being a harmonic coefficient between 0 and 1 used to balance the contributions of the two terms.

[0035] In practice, the feature distillation path log is analyzed to identify features that have been eliminated but are associated with samples of high ensemble probability distribution. Samples of high ensemble probability distribution are those with a probability value exceeding 0.8 in the ensemble probability distribution. A feedback loop is established to transmit feature utility scores and the information of eliminated features to the sensitivity evaluator of the hierarchical feature distillation network. This feedback loop is triggered every 500 new samples processed. The sensitivity evaluator dynamically adjusts its internal parameters for calculating mutual information and the feature elimination threshold based on the feedback information. The adjustment method involves eliminating features with utility scores below the new threshold in subsequent screenings. The new threshold is dynamically set based on the historical distribution of feature utility scores. The updated sensitivity evaluator will use the new threshold to screen features in subsequent data processing. This process allows the hierarchical feature distillation network to adaptively optimize its feature selection strategy and focus on features with higher utility for the current recognition task.

[0036] See Figure 4The results show a comparison of the probability outputs and ensemble probabilities of different classifiers. Specifically, the horizontal axis represents seven micro-expression categories (happiness, sadness, surprise, fear, disgust, anger, and neutral), and the vertical axis represents the probability values. The Support Vector Machine (SVM) classifier (blue bar) outputs the highest probability (0.50) in the "surprise" category, reflecting its ability to capture global statistical features. The Random Forest (purple bar) classifier also performs well in the "surprise" category (0.45), demonstrating its advantage in recognizing local dynamic features. The Temporal Convolutional Network (yellow bar) classifier has the highest probability value (0.55) in the "surprise" category, highlighting its ability to model temporal evolution features. The ensemble probability (red line) reaches its peak (approximately 0.51) in the "surprise" category. This is the result of the meta-decision fusion module weighting the probability outputs of the three classifiers using reliability weights, verifying the effectiveness of multi-classifier collaborative reasoning. Furthermore, in low-probability categories such as "neutral," both the outputs of each classifier and the ensemble probability remain at low levels, further indicating the system's consistency in recognizing different micro-expression categories.

[0037] In one embodiment of the present invention, the specific operation of the system optimization module refers to the online unsupervised parameter fine-tuning of the system. After the complete micro-expression recognition system is put into operation, the system continuously captures and caches new input multimodal data streams, with the cache queue size set to 200 sample batches. For the cached new input multimodal data streams, the current system generates recognition results, where the current system refers to the complete micro-expression recognition system integrating the updated hierarchical feature distillation network and cascaded classifier group. The generated recognition results include preliminary recognition labels and ensemble probability distributions, but these recognition results are not used as ground truth labels. Temporal consistency constraints and intermodal consistency constraints are applied to evaluate the credibility of the system's recognition results on the new input multimodal data streams. Temporal consistency constraints check the coherence between the preliminary recognition labels of the current micro-expression segment and the preliminary recognition labels of its preceding and following segments. Intermodal consistency constraints compare the consistency between the recognition results generated based on visual data block substreams and the recognition results generated based on physiological data sequence substreams.

[0038] In some embodiments, high-confidence recognition results that meet consistency constraints are selected and considered as pseudo-label data. The consistency constraint is met when both the temporal consistency score and the inter-modal consistency score exceed 0.7. Pseudo-label data includes high-confidence preliminary recognition labels and their corresponding new input multimodal data stream fragments. An online fine-tuning dataset is constructed using the pseudo-label data. This dataset is managed using a circular buffer, where the most recently added pseudo-label data replaces the oldest data. An online fine-tuning loss function is defined using an unsupervised or self-supervised learning objective. Contrastive learning is chosen as the unsupervised or self-supervised learning objective, and the online fine-tuning loss function is defined as a contrastive loss based on the fused feature representation of different samples in the pseudo-label data. The mathematical expression of the contrastive loss function is:

[0039] Where: symbol L c This represents the online fine-tuning loss function, also known as contrastive loss, denoted by z. i The fused feature representation vector of the anchor sample is denoted by z. p The fused feature representation vector, denoted by z, represents the positive sample belonging to the same pseudo-label category as the anchor sample. k This represents the fused feature representation vector of other samples in the batch, with the symbol sim representing the cosine similarity function. Temperature hyperparameters are used to control the sensitivity of the loss function. The symbol N represents the total number of samples in a training batch. It is an indicator function that takes the value 1 when k ≠ i and otherwise takes the value 0.

[0040] In practice, without interrupting the micro-expression recognition system service, mini-batch gradient descent is used to fine-tune some parameters of the hierarchical feature distillation network and cascaded classifier group in the system using an online fine-tuning dataset and an online fine-tuning loss function. The batch size of mini-batch gradient descent is set to 32, and the learning rate is set to 0.0001. Each fine-tuning iteration only updates the convolutional weights of the last two feature selection layers in the hierarchical feature distillation network and the fully connected layer weights of the temporal convolutional network classifier in the cascaded classifier group. The performance difference of the system before and after fine-tuning on a fixed validation set is monitored to prevent performance drift. The fixed validation set is a labeled dataset that is independent of the training data and does not participate in online fine-tuning. The performance difference is measured by the change in classification accuracy on the fixed validation set before and after fine-tuning. If the accuracy drops by more than 1%, all parameter changes in this fine-tuning are rolled back. By iteratively executing the steps of capturing new data, evaluating confidence, filtering pseudo-labels, and fine-tuning parameters, the micro-expression recognition system can continuously adapt to changes in the distribution of the input data stream.

[0041] See Figure 5This study presents the dynamic relationship between system performance and optimization goals. The changes in validation set classification accuracy (blue curve) and contrastive loss (yellow curve) with each fine-tuning iteration reflect the system's adaptation process: In the initial stage, the contrastive loss rapidly decreases from a high of approximately 2.6, corresponding to a significant drop in validation set accuracy after a slight fluctuation. At the 12th iteration, the accuracy reaches the preset lower limit of 91.5%, triggering the parameter rollback mechanism (marked with a red cross). After the rollback, the system continuously optimizes through pseudo-label-driven contrastive learning, gradually converging the contrastive loss to near zero. The validation set accuracy subsequently rebounds, peaking between the 25th and 35th iterations (approximately 93.8%), and then slowly declines amidst fluctuations, but overall remains above the safe threshold of 91.5%. This trend indicates a strong negative correlation between the effective reduction of contrastive loss and the improvement in system classification accuracy, validating the effectiveness of the "contrastive learning goal-driven + accuracy monitoring rollback" mechanism in the online fine-tuning strategy. The setting of the lower bound of accuracy (red dashed line) ensures that the system does not experience performance drift when adapting to new data distributions, while the convergence of the contrastive loss reflects the continuous enhancement of the discriminative ability of the fused feature representation under pseudo-label supervision.

[0042] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A high-efficiency multimodal micro-expression recognition system based on a deep learning model, characterized in that, The implementation steps include: The data synchronization module initiates a parallel data acquisition process, synchronously acquires visual images and bioelectric signals, forms an initial multimodal data stream, performs intermodal synchronization phase-locked loop processing, and performs temporal and spatial alignment correction on the initial multimodal data stream. The feature distillation module inputs the aligned and corrected data stream into the hierarchical feature distillation network, which peels off and filters out sensitive features related to micro-expressions layer by layer. The adaptive fusion module constructs an adaptive feature topology graph using feature sensitivity, dynamically allocates fusion weights for different modal features, guides the feature fusion process through the weight allocation results, and generates fusion feature representations with contextual relevance. The segmentation recognition module, based on the fusion feature expression-driven adaptive segmentation and boundary labeling of micro-expression fragments, sends the labeled fragments into a cascaded classifier group for collaborative reasoning to generate preliminary recognition labels; The system optimization module updates the screening threshold of the hierarchical feature distillation network based on the collaborative inference results, integrates the updated network with the cascaded classifier group to form a complete micro-expression recognition system, performs online unsupervised parameter fine-tuning of the system, and continuously adapts to new input data streams.

2. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 1, characterized in that, The process of initiating parallel data capture includes: Equipped with a high frame rate visible light camera and an infrared thermal imager, video images of the facial area are recorded at the same sampling rate; Simultaneously deploy contact-type bioelectric sensors to collect physiological signals, including electromyography and skin conductivity. Inject millisecond-level timestamps into each frame of visual imagery and each point in time with bioelectrical signals. The video image sequence is initially bound to the physiological signal sequence based on the timestamp; Face detection and pose estimation are performed on the bound video image sequence to extract stable facial regions of interest; The extracted facial regions of interest are subjected to illumination-invariant transformations to generate standardized visual data blocks. Baseline drift removal and power frequency noise filtering are performed on the bound physiological signal sequences to generate clean physiological data sequences; Standardized blocks of visual data are encapsulated together with clean sequences of physiological data to form an initial multimodal data stream.

3. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 2, characterized in that, The inter-modal synchronization phase-locked loop processing includes: Analyze the inter-frame motion vectors of visual data blocks and the transient change points of physiological data sequences in the initial multimodal data stream; Using significant transient change points in physiological data sequences as temporal reference benchmarks, the relative delay of visual motion vectors is calculated; Establish a time delay compensation function to perform frame-level time offset correction on visual data blocks; Based on temporal alignment, the spatial correspondence between facial regions of interest and physiological signal sources is mapped. Based on spatial correspondence, assign corresponding physiological signal components to each facial sub-region; By fusing time-aligned visual data with spatially mapped physiological signals, a spatiotemporally consistent multimodal data matrix is ​​generated; To verify the synchronicity of signals from different channels in a multimodal data matrix at key event points; If the synchronization does not reach the preset threshold, the delay compensation function and spatial mapping relationship are iteratively adjusted until the data matrix completes alignment correction.

4. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 3, characterized in that, The step of inputting the aligned and corrected data stream into the hierarchical feature distillation network includes: Construct a hierarchical feature distillation network consisting of multiple stacked feature filtering layers, each layer containing a feature extractor and a sensitivity estimator; The aligned and corrected multimodal data matrix is ​​input into the first layer feature extractor of the network to extract preliminary edge and texture features; The sensitivity evaluator of the first layer calculates the mutual information between each preliminary feature and the known micro-expression sample library to obtain the feature sensitivity score; Based on the feature sensitivity score, redundant features with scores below the threshold are eliminated, and the filtered features are passed to the next layer. In each subsequent layer of the network, the incoming features are subjected to higher-order abstraction and combination; Each layer's sensitivity evaluator recalculates the strength of its association with micro-expressions based on the abstract features of the current layer; After all levels of filtering, a refined set of sensitive features highly correlated with micro-expressions is finally output. Record the features eliminated at each layer and their sensitivity scores to form a feature distillation path log.

5. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 4, characterized in that, The method of constructing an adaptive feature topology graph using feature sensitivity includes: Read the set of sensitive features output by the hierarchical feature distillation network and its corresponding feature sensitivity scores; Each sensitive feature is treated as a node in a topological graph, and the weight of the node is its feature sensitivity score. Calculate the correlation coefficient between any two sensitive feature nodes in the multimodal data matrix; Using the correlation coefficient as the weight of the edge, two feature nodes are connected to initially construct a fully connected feature topology graph; Based on the weights of nodes and edges, a graph pruning algorithm is applied to remove nodes and edges with too low weights and sparse connections to the main subgraph. Community detection was performed on the pruned feature topology to identify feature communities with tight internal connections. Assign a community identifier to each feature community and calculate the overall sensitivity weight of the feature community; Based on the community identifier and the overall sensitivity weight, the final adaptive feature topology graph structure description is generated.

6. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 5, characterized in that, The process of guiding feature fusion based on weight allocation results includes: Analyze the adaptive feature topology graph structure description to obtain the overall sensitivity weights of different feature clusters; A normalization method is used to transform the overall sensitivity weight into the initial contribution coefficient of each feature cluster in the fusion process; For feature nodes within the same cluster, the internal fusion ratio is allocated according to their node weights. Design a feature fusion function that uses feature clusters as units and weights and aggregates the original sensitive features according to the initial contribution coefficient and the internal fusion ratio. In the weighted aggregation process, a cross-community feature interaction term is introduced, which is determined by the weight of the edges connecting different communities. Execute the feature fusion function to generate a unified fused feature vector. The fused feature vector is input into a lightweight feedforward network, where it undergoes nonlinear transformation and dimensionality adjustment to output a fused feature representation with contextual relevance. The fused feature representation simultaneously encodes feature importance, community structure, and cross-modal association information.

7. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 6, characterized in that, The adaptive segmentation and boundary labeling of micro-expression fragments driven by fusion feature representation includes: Construct a sliding time window that moves continuously over the time series of fused feature representations with contextual relevance; At each window location, calculate the rate of change of statistical properties of the fused feature representation within the window; When the rate of change exceeds the dynamically set segmentation threshold, the time point is marked as the potential start or end point of a micro-expression event. Starting from the marked potential points, the analysis is expanded in both forward and backward directions to find local extreme points of characteristic changes in order to correct the boundary. Based on the corrected boundary points, the corresponding data segments are extracted from the original multimodal data matrix to form candidate micro-expression fragments; For each candidate micro-expression fragment, extract its short-term trajectory pattern expressed by fused features; The internal consistency score of candidate micro-expression segments is calculated based on trajectory patterns to filter out false segments with low internal consistency. The selected candidate micro-expression fragments and their start and end timestamps are output as a calibrated fragment set.

8. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 7, characterized in that, The step of feeding the calibrated segments into a cascaded classifier group for collaborative reasoning includes: Configure a cascaded classifier group consisting of multiple heterogeneous classifiers, each classifier focusing on recognizing feature patterns in different dimensions; Multiple complementary feature views are extracted from the calibrated fragment set, including a global statistical view, a local dynamic view, and a temporal evolution view; The global statistical view is input into the support vector machine classifier in the cascaded classifier group to obtain a preliminary probability distribution vector; The local dynamic view is input into the random forest classifier in the cascaded classifier group to obtain another probability distribution vector; The temporal evolution view is input into the temporal convolutional network classifier in the cascaded classifier group to obtain the third probability distribution vector; Design a meta-decision fusion module that receives three probability distribution vectors; The meta-decision fusion module learns the reliability weights of each classifier under different feature conditions; The integrated probability distribution is obtained by weighting the three probability distribution vectors according to the reliability weights. The category with the highest probability is selected from the ensemble probability distribution as the initial identification label for micro-expression segments; The initial identification labels and their corresponding integrated probability distributions are output together.

9. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 8, characterized in that, The screening thresholds for updating the hierarchical feature distillation network based on collaborative reasoning results include: Collect the preliminary identification labels and their ensemble probability distributions output by the cascaded classifier group over a period of time; Calculate the discriminant index of each sensitive feature retained by the hierarchical feature distillation network under different identification labels; A new feature utility score is generated by combining the feature's discriminative index with its original feature sensitivity score. Analyze the feature distillation path log to identify features that were eliminated but associated with samples of high ensemble probability distribution; Establish a feedback loop to transmit the feature utility score and the information of the eliminated features collected back to the sensitivity evaluator of the hierarchical feature distillation network. The sensitivity evaluator dynamically adjusts its internal parameters for calculating mutual information and the threshold for feature elimination based on feedback information. The updated sensitivity evaluator will use a new threshold to filter features in subsequent data processing; This process enables the hierarchical feature distillation network to adaptively optimize its feature selection strategy, focusing on features that are more effective for the current recognition task.

10. The efficient multimodal micro-expression recognition system based on a deep learning model according to claim 9, characterized in that, The online unsupervised parameter fine-tuning of the execution system includes: After the complete micro-expression recognition system is put into operation, it continuously captures and caches new input multimodal data streams; For new cached data streams, use the current system to generate recognition results, but do not use them as truth labels; By applying temporal consistency constraints and intermodal consistency constraints, the credibility of the system's recognition results on new data is evaluated. High-confidence identification results that meet the consistency constraints are selected and treated as pseudo-label data; Construct an online fine-tuning dataset using pseudo-labeled data; Employ unsupervised or self-supervised learning objectives, such as contrastive learning or reconstruction learning, and define an online fine-tuning loss function; Without interrupting system services, the parameters of some parameters of the hierarchical feature distillation network and cascade classifier group in the system are fine-tuned using mini-batch gradient descent with online fine-tuning dataset and loss function. Monitor the performance differences of the system on a fixed validation set before and after fine-tuning to prevent performance drift; By iteratively executing capture, evaluation, filtering, and fine-tuning steps, the system is able to continuously adapt to changes in the distribution of the input data stream.

Images