Multi-modal feature analysis method and system for micro-action recognition

Abstract

The invention discloses a multi-modal feature analysis method and system for micro-action recognition, and relates to the technical field of computer vision and artificial intelligence. Comprising the steps of 1, creating a multi-modal feature analysis system for micro-action recognition, 2, carrying out multi-modal data preprocessing and feature extraction through a preprocessing and feature extraction module, 3, carrying out time synchronization and space alignment on each modal feature through a synchronization and alignment module, 4, carrying out cross-modal association and fusion through a fusion module, and 5, carrying out multi-modal feature analysis. 5, space-time verification and optimization after multi-modal feature fusion are carried out through a verification module, and 6, the multi-modal features are integrated through an output module according to the adjusted weights, micro-action categories are recognized through a classifier according to the integrated multi-modal features, and a structured result is output; according to the method, the problems of asynchronous multi-modal feature acquisition, non-uniform space coordinates, weak semantic association and poor time sequence consistency are solved.

Description

Technical Field

[0001] This invention discloses a multimodal feature analysis method and system for micro-motion recognition, which relates to the fields of computer vision and artificial intelligence. Background Technology

[0002] Existing action recognition technologies have some limitations, such as: single-modal recognition methods are not reliable enough in complex environments and are easily affected by factors such as occlusion and changes in lighting; multimodal feature fusion technologies often use simple feature splicing or post-decision fusion, lacking in-depth consideration of spatial alignment and spatiotemporal consistency of heterogeneous features.

[0003] Existing methods struggle to effectively handle the subtle changes and cross-modal collaborative relationships unique to micro-motion recognition. Especially in dynamic environments, detection results from different modalities may exhibit spatiotemporal inconsistencies and confidence level conflicts, leading to a decrease in recognition accuracy. Summary of the Invention

[0004] This invention addresses the problems of existing technologies by providing a multimodal feature analysis method and system for micro-motion recognition. It solves the problems of asynchronous multimodal feature acquisition, inconsistent spatial coordinates, weak semantic association, and poor temporal consistency. It is suitable for application scenarios such as human-computer interaction, medical rehabilitation, and security monitoring that require accurate recognition of subtle human movements.

[0005] The specific solution proposed in this invention is as follows:

[0006] This invention also provides a multimodal feature analysis method for micro-motion recognition, comprising:

[0007] Step 1: Create a multimodal feature analysis system for micro-motion recognition. The system includes a preprocessing and feature extraction module, a synchronization and alignment module, a fusion module, a verification module, and an output module.

[0008] Step 2: Multimodal data preprocessing and feature extraction are performed using the preprocessing and feature extraction module: Image frames are generated by copying the input video stream, and these image frames are input into each model for parallel processing to extract facial, hand, and pose features respectively.

[0009] Step 3: Perform temporal synchronization and spatial alignment of each modal feature using the synchronization and alignment module: First, determine the synchronization status of each modal feature, and then perform temporal alignment by combining the corresponding features in chronological order according to the synchronization status; transform the normalized coordinates of each model output to a unified pixel coordinate system, calculate the spatial offset of each modal feature, and correct the differences in model output based on the spatial offset to achieve spatial alignment.

[0010] Step 4: Perform cross-modal association and fusion through the fusion module: Select facial features as anchor points as reference benchmarks, dynamically adjust the spatial positions of other modalities to adapt to coordinate changes caused by head movements; establish semantic mapping between modalities, use a sliding window to dynamically adjust the fusion weights of each modality for multimodal feature fusion, and output the multimodal feature fusion confidence score based on a weighted geometric algorithm.

[0011] Step 5: Spatiotemporal verification and optimization of multimodal feature fusion using the verification module: Calculate and score the spatiotemporal consistency between the current frame and historical image frames, and adjust the multimodal fusion weights based on the score.

[0012] Step 6: The output module integrates multimodal features based on the adjusted weights, and the classifier identifies the micro-motion categories based on the integrated multimodal features, outputting structured results.

[0013] Furthermore, in step 2 of the multimodal feature analysis method for micro-motion recognition, the preprocessing and feature extraction module includes a parallel processing flow construction module. This module uses a three-channel memory mapping method to generate image frames from the input video stream. These image frames are then input into various models for parallel processing, extracting facial, hand, and pose features respectively. Specifically, the parallel processing flow construction module creates three physically contiguous but logically independent memory mapping regions for each frame of the input video stream, storing a copy of the facial image I. face Hand image copy I hand and pose image copy I pose Each image copy retains the color space and resolution attributes of the original image. The three image copies are loaded into the corresponding models respectively, and facial, hand and pose features are extracted in parallel.

[0014] Furthermore, in step 3 of the multimodal feature analysis method for micro-motion recognition, the synchronization and alignment module includes a timestamp synchronization module, a data alignment verification module, a coordinate system transformation module, a spatial alignment module, and a deviation correction module.

[0015] The synchronization status of each modality feature is first determined using the timestamp synchronization module: a timestamp difference matrix is ​​constructed. t face t hand t pose These are the facial feature timestamps, hand feature timestamps, and pose feature timestamps for the same frame; a synchronization threshold δ is set. t According to the formula: δ t =1.2×(1 / fps) Real-time adjustment δ t FPS is the video stream frame rate, which dynamically changes between 30 and 60 frames. If the maximum value of all elements in the timestamp difference matrix D of three consecutive frames satisfies max(D) < δ...t Then the corresponding feature group will be determined to be in a synchronization state.

[0016] The data alignment verification module performs temporal alignment by combining corresponding features in chronological order based on the synchronization status: first, it checks whether the timestamp difference is within the synchronization threshold range; then, it employs a dual-pointer circular queue design and sets the facial feature queue Q. face Hand feature queue Q hand and pose feature queue Q pose Each queue has a depth of 8 frames. When a synchronization failure is detected, the queue is cleaned up until all features reach a synchronized state. The synchronized features are then combined in chronological order into a feature tensor T∈R. (N ×3×K) Where N is the total number of feature points, 3 represents the three-dimensional coordinate dimension, K is the time series length, and the output is a multimodal feature data packet with timestamps;

[0017] The coordinate system transformation module transforms the normalized coordinates output by each model to a unified pixel coordinate system: the image pixel coordinate system is established as the reference system, and the normalized coordinates output by each model are transformed into pixel coordinates using homogeneous coordinate transformation.

[0018] The spatial alignment module calculates the spatial offset of each modality feature: cluster analysis of feature points is performed to group and classify the facial, hand, and body pose feature points output by different models; the spatial offset between different feature point sets is calculated based on the similarity metric function of Euclidean distance; and the positional deviation between feature point sets is gradually reduced through multiple rotation and translation transformations using the iterative nearest point algorithm.

[0019] Spatial alignment is achieved by using the deviation correction module to correct the differences in the output of the spatial offset correction model: the smaller value between the image width and height is used as the depth scaling reference parameter, and the differences in the output of the spatial offset correction model are corrected accordingly.

[0020] Furthermore, in step 4 of the multimodal feature analysis method for micro-motion recognition, the fusion module includes a facial reference system construction module, a spatial transformation calculation module, a semantic association modeling module, and a confidence fusion calculation module.

[0021] The facial reference system construction module uses selected facial features as anchor points as reference benchmarks: facial key points are detected through real-time video stream, and the tip of the nose point P is selected. nose The outer corner of the left eye outer corner of right eye Three feature points;

[0022] The spatial transformation calculation module dynamically adjusts the spatial position of other modalities to adapt to coordinate changes caused by head movements.

[0023] Establish intermodal semantic mapping through the semantic association modeling module: establish the coordinates P of finger feature points. finge r and the coordinates of facial feature points P face The mapping relationship between the two; calculate the Euclidean distance d = ||P finger -P face When d < 15 pixels is detected, contact relationship determination is triggered, and spatial association markers are generated. A Gaussian kernel function is used to construct a semantic association function.

[0024] f semantic =exp(-d 2 / 144.5), where the scale parameter σ = 8.5 pixels, determined through calibration experiments, and the function output range is [0,1], used to quantify the spatial correlation strength;

[0025] The confidence fusion calculation module uses a sliding window to dynamically adjust the fusion weights of each modality for multimodal feature fusion, and outputs the multimodal feature fusion confidence based on a weighted geometric algorithm: through the weight update formula w. i (t)=0.85·w i (t-1)+0.15·(acc i / ∑acc j To achieve dynamic adjustment of fusion weights, ensure ∑w i The constraint condition is 1, acc i Let ∑acc be the historical accuracy of the i-th mode within the sliding window. j This represents the sum of historical accuracy for all modalities within the sliding window; using the formula: C fused c represents the overall confidence level after fusion. i Let C be the original confidence level of the i-th mode. fused When the value is greater than 0.6, a valid micro-motion determination signal is generated, and the fusion confidence value is output at the same time.

[0026] Furthermore, in step 5 of the multimodal feature analysis method for micro-action recognition, the verification module includes a conflict arbitration execution module, a historical state maintenance module, a spatiotemporal consistency measurement module, and a conflict resolution execution module.

[0027] The conflict degree of each modal feature is measured through the conflict arbitration execution module: the conflict degree measurement formula is: Conflict=1-∏(1-|c i -c j |) Quantization inconsistency, (i,j)∈all modal pairs, c i and c j Arbitration is triggered when Conflict > 0.7 is detected, based on the confidence levels for different modalities; and the spatiotemporal consistency scoring formula S is used. consistency=0.6·spatial sim +0.4·temporal sim Modality optimization is performed in conflict scenarios, where spatial similarity (spatial_sim) measures spatial information between modalities by the consistency of feature point distribution, and temporal similarity (temporal) is used. sim The synchronization of time information between modalities is measured by action timing alignment degree calculation, and S is ultimately selected. consistency The highest mode was used as the arbitration result.

[0028] The historical image frame data buffer is managed through a historical state maintenance module. This buffer stores the fused feature state sequence and always retains valid state data from the most recent L frames. The default buffer length L is set to 10 frames. A timestamp is added to the feature state of each frame, and a temporal index relationship is established. Feature state retrieval is achieved by constructing a mapping table between the time dimension and the feature state.

[0029] The spatiotemporal consistency measurement module calculates the degree of spatiotemporal consistency between the current frame and historical image frames: Spatial dispersion is calculated based on the feature vectors of each modality in the current frame, and the standardized spatial variance index is obtained by dividing the sum of squared Euclidean distances between features by the number of modalities; the cumulative temporal difference between the features of the current frame and the features of the previous L-1 frames in the historical buffer is calculated using a sliding window approach, and the mean square error of feature changes is used as a temporal consistency measure; the spatial and temporal variance indices are fused, and a comprehensive consistency score is generated through a linear weighting method, where the balance coefficients α = 0.4 and β = 0.3. When the score is below 0.6, a conflict resolution process is triggered.

[0030] The multimodal fusion weights are adjusted through the conflict resolution execution module: when the conflict resolution process is triggered, an exponential decay strategy is used to reduce the weights of inconsistent modes, and the product of the original weights and the consistency scores is used as the new weight values; when it is detected that the consistency scores of all modes are lower than the threshold, a status flag is generated and the decision output of the current frame is paused; the weight values ​​released by the reduction are distributed proportionally to the highly consistent modes through normalization processing to ensure that the total weight remains constant, while setting an upper limit for the weight adjustment range to prevent drastic weight fluctuations.

[0031] This invention also provides a multimodal feature analysis system for micro-motion recognition, including a preprocessing and feature extraction module, a synchronization and alignment module, a fusion module, a verification module, and an output module.

[0032] The preprocessing and feature extraction module performs multimodal data preprocessing and feature extraction: it generates image frames by copying the input video stream, inputs these image frames into various models for parallel processing, and extracts facial, hand, and pose features respectively.

[0033] The synchronization and alignment module performs temporal synchronization and spatial alignment of each modal feature: first, it determines the synchronization state of each modal feature, and then combines the corresponding features in chronological order for temporal alignment based on the synchronization state; second, it transforms the normalized coordinates of each model output to a unified pixel coordinate system, calculates the spatial offset of each modal feature, and corrects the differences in model output based on the spatial offset to achieve spatial alignment.

[0034] The fusion module performs cross-modal association and fusion: facial features are selected as anchor points as reference benchmarks, and the spatial positions of other modalities are dynamically adjusted to adapt to coordinate changes caused by head movements; a semantic mapping between modalities is established, and a sliding window is used to dynamically adjust the fusion weights of each modality for multimodal feature fusion; and a multimodal feature fusion confidence score is output based on a weighted geometric algorithm.

[0035] The verification module performs spatiotemporal verification and optimization after multimodal feature fusion: it calculates and scores the spatiotemporal consistency between the current frame and historical image frames, and adjusts the multimodal fusion weights based on the scores.

[0036] The output module integrates multimodal features based on the adjusted weights, and the classifier identifies the micro-motion categories based on the integrated multimodal features, outputting structured results.

[0037] Furthermore, the preprocessing and feature extraction module of the multimodal feature analysis system for micro-motion recognition includes a parallel processing flow construction module. This module uses a three-channel memory mapping method to generate image frames from the input video stream. These image frames are then input into various models for parallel processing, extracting facial, hand, and pose features respectively. Specifically, the parallel processing flow construction module creates three physically contiguous but logically independent memory mapping regions for each frame of the input video stream, storing a copy of the facial image I. face Hand image copy I hand and pose image copy I pose Each image copy retains the color space and resolution attributes of the original image. The three image copies are loaded into the corresponding models respectively, and facial, hand and pose features are extracted in parallel.

[0038] Furthermore, the synchronization and alignment module of the multimodal feature analysis system for micro-motion recognition includes a timestamp synchronization module, a data alignment verification module, a coordinate system transformation module, a spatial alignment module, and a deviation correction module.

[0039] The synchronization status of each modality feature is first determined using the timestamp synchronization module: a timestamp difference matrix is ​​constructed. t face t hand t pose These are the facial feature timestamps, hand feature timestamps, and pose feature timestamps for the same frame; a synchronization threshold δ is set.t According to the formula: δ t =1.2×(1 / fps) Real-time adjustment δ t FPS is the video stream frame rate, which dynamically changes between 30 and 60 frames. If the maximum value of all elements in the timestamp difference matrix D of three consecutive frames satisfies max(D) < δ... t Then the corresponding feature group will be determined to be in a synchronization state.

[0040] The data alignment verification module performs temporal alignment by combining corresponding features in chronological order based on the synchronization status: first, it checks whether the timestamp difference is within the synchronization threshold range; then, it employs a dual-pointer circular queue design and sets the facial feature queue Q. face Hand feature queue Q hand and pose feature queue Q pose Each queue has a depth of 8 frames. When a synchronization failure is detected, the queue is cleaned up until all features reach a synchronized state. The synchronized features are then combined in chronological order into a feature tensor T∈R. (N ×3×K) Where N is the total number of feature points, 3 represents the three-dimensional coordinate dimension, K is the time series length, and the output is a multimodal feature data packet with timestamps;

[0041] The coordinate system transformation module transforms the normalized coordinates output by each model to a unified pixel coordinate system: the image pixel coordinate system is established as the reference system, and the normalized coordinates output by each model are transformed into pixel coordinates using homogeneous coordinate transformation.

[0042] The spatial alignment module calculates the spatial offset of each modality feature: cluster analysis of feature points is performed to group and classify the facial, hand, and body pose feature points output by different models; the spatial offset between different feature point sets is calculated based on the similarity metric function of Euclidean distance; and the positional deviation between feature point sets is gradually reduced through multiple rotation and translation transformations using the iterative nearest point algorithm.

[0043] Spatial alignment is achieved by using the deviation correction module to correct the differences in the output of the spatial offset correction model: the smaller value between the image width and height is used as the depth scaling reference parameter, and the differences in the output of the spatial offset correction model are corrected accordingly.

[0044] Furthermore, the fusion module of the multimodal feature analysis system for micro-motion recognition includes a facial reference system construction module, a spatial transformation calculation module, a semantic association modeling module, and a confidence fusion calculation module.

[0045] The facial reference system construction module uses selected facial features as anchor points as reference benchmarks: facial key points are detected through real-time video stream, and the tip of the nose point P is selected. nose The outer corner of the left eye outer corner of right eye Three feature points;

[0046] The spatial transformation calculation module dynamically adjusts the spatial position of other modalities to adapt to coordinate changes caused by head movements.

[0047] Establish intermodal semantic mapping through the semantic association modeling module: establish the coordinates P of finger feature points. finge r and the coordinates of facial feature points P face The mapping relationship between the two; calculate the Euclidean distance d = ||P finger -P face When d < 15 pixels is detected, contact relationship determination is triggered, and spatial association markers are generated. A Gaussian kernel function is used to construct a semantic association function.

[0048] f semantic =exp(-d 2 / 144.5), where the scale parameter σ = 8.5 pixels, determined through calibration experiments, and the function output range is [0,1], used to quantify the spatial correlation strength;

[0049] The confidence fusion calculation module uses a sliding window to dynamically adjust the fusion weights of each modality for multimodal feature fusion, and outputs the multimodal feature fusion confidence based on a weighted geometric algorithm: through the weight update formula w. i (t)=0.85·w i (t-1)+0.15·(acc i / ∑acc j To achieve dynamic adjustment of fusion weights, ensure ∑w i The constraint condition is 1, acc i Let ∑acc be the historical accuracy of the i-th mode within the sliding window. j This represents the sum of historical accuracy for all modalities within the sliding window; using the formula: C fused c represents the overall confidence level after fusion. i Let C be the original confidence level of the i-th mode. fused When the value is greater than 0.6, a valid micro-motion determination signal is generated, and the fusion confidence value is output at the same time.

[0050] Furthermore, the verification module of the multimodal feature analysis system for micro-action recognition includes a conflict arbitration execution module, a historical state maintenance module, a spatiotemporal consistency measurement module, and a conflict resolution execution module.

[0051] The conflict degree of each modal feature is measured through the conflict arbitration execution module: the conflict degree measurement formula is: Conflict=1-∏(1-|c i -c j|) Quantization inconsistency, (i,j)∈all modal pairs, c i and c j Arbitration is triggered when Conflict > 0.7 is detected, based on the confidence levels for different modalities; and the spatiotemporal consistency scoring formula S is used. consistency =0.6·spatial sim +0.4·temporal sim Modality optimization is performed in conflict scenarios, where spatial similarity (spatial_sim) measures spatial information between modalities by the consistency of feature point distribution, and temporal similarity (temporal) is used. sim The synchronization of time information between modalities is measured by action timing alignment degree calculation, and S is ultimately selected. consistency The highest mode was used as the arbitration result.

[0052] The historical image frame data buffer is managed through a historical state maintenance module. This buffer stores the fused feature state sequence and always retains valid state data from the most recent L frames. The default buffer length L is set to 10 frames. A timestamp is added to the feature state of each frame, and a temporal index relationship is established. Feature state retrieval is achieved by constructing a mapping table between the time dimension and the feature state.

[0053] The spatiotemporal consistency measurement module calculates the degree of spatiotemporal consistency between the current frame and historical image frames: Spatial dispersion is calculated based on the feature vectors of each modality in the current frame, and the standardized spatial variance index is obtained by dividing the sum of squared Euclidean distances between features by the number of modalities; the cumulative temporal difference between the features of the current frame and the features of the previous L-1 frames in the historical buffer is calculated using a sliding window approach, and the mean square error of feature changes is used as a temporal consistency measure; the spatial and temporal variance indices are fused, and a comprehensive consistency score is generated through a linear weighting method, where the balance coefficients α = 0.4 and β = 0.3. When the score is below 0.6, a conflict resolution process is triggered.

[0054] The multimodal fusion weights are adjusted through the conflict resolution execution module: when the conflict resolution process is triggered, an exponential decay strategy is used to reduce the weights of inconsistent modes, and the product of the original weights and the consistency scores is used as the new weight values; when it is detected that the consistency scores of all modes are lower than the threshold, a status flag is generated and the decision output of the current frame is paused; the weight values ​​released by the reduction are distributed proportionally to the highly consistent modes through normalization processing to ensure that the total weight remains constant, while setting an upper limit for the weight adjustment range to prevent drastic weight fluctuations.

[0055] The advantages of this invention are:

[0056] This invention utilizes a multimodal feature analysis system to form a complete technical closed loop of multimodal feature acquisition, spatiotemporal alignment, semantic association, confidence fusion, and conflict resolution, achieving accurate recognition and collaborative analysis of micro-movements. It solves the problems of asynchronous multimodal feature acquisition, inconsistent spatial coordinates, weak semantic association, and poor temporal consistency, and is suitable for application scenarios such as human-computer interaction, medical rehabilitation, and security monitoring that require accurate recognition of subtle human movements. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0059] Example 1

[0060] This invention also provides a multimodal feature analysis method for micro-motion recognition, comprising:

[0061] Step 1: Create a multimodal feature analysis system for micro-motion recognition. The system includes a preprocessing and feature extraction module, a synchronization and alignment module, a fusion module, a verification module, and an output module.

[0062] Step 2: Perform multimodal data preprocessing and feature extraction through the preprocessing and feature extraction module: Generate image frames by copying the input video stream, input the image frames into each model for parallel processing, and extract facial, hand and pose features respectively.

[0063] The preprocessing and feature extraction module includes a parallel processing flow construction module. This module uses a three-channel memory mapping method to generate image frames from the input video stream. These image frames are then input into various models for parallel processing, extracting facial, hand, and pose features. Specifically, the parallel processing flow construction module creates three physically contiguous but logically independent memory mapping regions for each frame of the input video stream, storing a copy of the facial image I. face Hand image copy I hand and pose image copy I poseEach image copy retains the color space and resolution attributes of the original image. The three image copies are loaded into their respective models, and facial, hand, and pose features are extracted in parallel. For example, the facial mesh model outputs 468 3D feature points, the hand detection model outputs the 3D coordinates of 21 key points, and the pose estimation model generates the spatial location information of 33 human key points. GPU resources can also be dynamically allocated based on computational complexity: 40% of CUDA cores are allocated for facial feature extraction, 35% for hand detection, and the remaining 25% for pose estimation. For example, a work-stealing algorithm can be used to achieve load balancing among CPU threads.

[0064] Step 3: Perform temporal synchronization and spatial alignment of each modal feature through the synchronization and alignment module: First, determine the synchronization status of each modal feature, and then combine the corresponding features in time order according to the synchronization status to perform temporal alignment; transform the normalized coordinates output by each model to a unified pixel coordinate system, calculate the spatial offset of each modal feature, and correct the model output differences according to the spatial offset to achieve spatial alignment.

[0065] The synchronization and alignment module includes a timestamp synchronization module, a data alignment verification module, a coordinate system transformation module, a spatial alignment module, and a deviation correction module.

[0066] The timestamp synchronization module, designed for domestically produced server environments, employs a high-precision event timer or a similar timing mechanism based on the ARM architecture. In the callback function after model inference completion, it reads the CPU cycle count from the system's high-precision timer, such as the CNTPCT register of ARMv8, and converts it into a microsecond-level timestamp based on the crystal oscillator reference frequency. The timestamp synchronization module first determines the synchronization status of each modal feature by constructing a timestamp difference matrix. t face t hand t pose These are the facial feature timestamps, hand feature timestamps, and pose feature timestamps from the same frame. A sliding window filtering algorithm, such as a moving average with a window size of 5, can be applied to each element in the matrix to smooth out instantaneous jitter and reduce deviations caused by noise. A synchronization threshold δ is set. t According to the formula: δ t =1.2×(1 / fps) Real-time adjustment δ t FPS is the video stream frame rate, which dynamically changes between 30 and 60 frames. If the maximum value of all elements in the timestamp difference matrix D of three consecutive frames satisfies max(D) < δ... t If the corresponding feature group is determined to be in a synchronized state, the system's robustness to frame rate fluctuations will be enhanced.

[0067] The data alignment verification module performs temporal alignment by combining corresponding features in chronological order based on the synchronization status: first, it checks whether the timestamp difference is within the synchronization threshold range; then, it employs a dual-pointer circular queue design and sets the facial feature queue Q. face Hand feature queue Q hand and pose feature queue Q pose Each queue has a depth of 8 frames. When a synchronization failure is detected, the queue is cleaned up until all features reach a synchronized state. The synchronized features are then combined in chronological order into a feature tensor T∈R. (N ×3×K) Where N is the total number of feature points, 3 represents the three-dimensional coordinate dimension, and K is the time series length, the output is a multimodal feature data packet with timestamps.

[0068] The coordinate system transformation module transforms the normalized coordinates output by each model to a unified pixel coordinate system: an image pixel coordinate system is established as the reference system, and the normalized coordinates output by each model are converted into pixel coordinates using homogeneous coordinate transformation; a learnable scaling parameter matrix can be introduced to dynamically adjust the coordinate mapping relationship of feature points of different modalities; the depth information scaling factor is preserved during the transformation process, and the spatial structure is kept realistic through 3D to 2D projection transformation.

[0069] The spatial alignment module calculates the spatial offset of each modality feature: cluster analysis of feature points is performed to group and classify the facial, hand, and body pose feature points output by different models; the spatial offset between different feature point sets is calculated based on the similarity metric function of Euclidean distance; and the positional deviation between feature point sets is gradually reduced through multiple rotation and translation transformations using the iterative nearest point algorithm.

[0070] Spatial alignment is achieved by using a deviation correction module to correct differences in the output of a spatial offset correction model. The smaller of the image width and height is used as the depth scaling reference parameter, and the spatial offset correction model outputs the differences. This can be achieved by establishing a deviation propagation model and calculating the correction amount for each feature point set using a backpropagation algorithm; additionally, a spatial consistency constraint function can be designed to maintain the relative distance ratio between feature points during scaling.

[0071] Among them, the coordinate transformation formula can adopt the perspective transformation model, and add a depth preservation term to the transformation matrix to ensure the proportional relationship of three-dimensional spatial information in the two-dimensional mapping process;

[0072] Pixel-level coordinate mapping can be achieved using bilinear interpolation, which maintains the spatial continuity of feature points during coordinate transformation.

[0073] The minimum dimension baseline can be achieved through dynamic threshold adjustment, automatically selecting the optimal scaling baseline parameters based on the actual size of the input image.

[0074] Step 4: Perform cross-modal association and fusion through the fusion module: Select facial features as anchor points as reference benchmarks, dynamically adjust the spatial positions of other modalities to adapt to coordinate changes caused by head movements; establish semantic mapping between modalities, use a sliding window to dynamically adjust the fusion weights of each modality for multimodal feature fusion, and output the confidence score of multimodal feature fusion based on a weighted geometric algorithm.

[0075] The fusion module includes a facial reference system construction module, a spatial transformation calculation module, a semantic association modeling module, and a confidence fusion calculation module.

[0076] The facial reference system construction module uses selected facial features as anchor points as reference benchmarks: facial key points are detected through real-time video streams, and a stability scoring algorithm is used to select the tip of the nose point P. nose The outer corner of the left eye outer corner of right eye Three feature points; its stability threshold is set as the displacement variance σ. 2 <0.01; The coordinate system is established at the centroid C. face Then, the x-axis vector v is processed by Schmidt orthogonalization. x And use the cross product operation to generate an orthogonal y-axis vector v. y =(P nose -C face )×v x Finally, the z-axis vector v is determined using the right-hand rule. z =v x ×v y This forms an orthonormal basis matrix.

[0077] The spatial transformation calculation module dynamically adjusts the spatial positions of other modalities to adapt to coordinate changes caused by head movements. The rotation matrix R can be calculated using quaternion interpolation, and its expression is as follows: The translation vector T is obtained by the difference in the centroids of the feature points; the objective function can be solved iteratively using the Levenberg-Marquardt optimization algorithm, with the convergence condition set as the change in error between adjacent iterations Δε < 10. -6 The orthogonality of the matrices was verified by SVD decomposition.

[0078] The fusion module may also include a coordinate registration and optimization module, which calculates the loss L using a combination of reprojection error and Huber loss function. δ (e), whose mathematical expression is: Recalibration is triggered when the error e > δ = 0.05 for three consecutive frames; the integrated Kalman filter predicts the facial motion trajectory, and a motion model is established through the state vector [position; velocity; acceleration], and the 6-DOF parameters of the transformation matrix are updated in real time to ensure the stability of the coordinate system within the range of ±45° head deflection.

[0079] Establish intermodal semantic mapping through the semantic association modeling module: establish the coordinates P of finger feature points. finge r and the coordinates of facial feature points P face The mapping relationship between the two; calculate the Euclidean distance d = ||P finger -P face When d < 15 pixels is detected, contact relationship determination is triggered, and spatial association markers are generated. A Gaussian kernel function is used to construct a semantic association function.

[0080] f semantic =exp(-d 2 / 144.5), where the scale parameter σ = 8.5 pixels, determined through calibration experiments, and the function output range is [0,1], used to quantify the spatial correlation strength;

[0081] The confidence fusion calculation module uses a sliding window to dynamically adjust the fusion weights of each modality for multimodal feature fusion, and outputs the multimodal feature fusion confidence based on a weighted geometric algorithm: through the weight update formula w. i (t)=0.85·w i (t-1)+0.15·(acc i / ∑acc j To achieve dynamic adjustment of fusion weights, ensure ∑w i The constraint condition is 1, acc i Let ∑acc be the historical accuracy of the i-th mode within the sliding window. j This represents the sum of historical accuracy for all modalities within the sliding window; using the formula: C fused c represents the overall confidence level after fusion. i Let C be the original confidence level of the i-th mode. fused When the value is greater than 0.6, a valid micro-motion determination signal is generated, and the fusion confidence value is output at the same time.

[0082] Step 5: Perform spatiotemporal verification and optimization of multimodal feature fusion through the verification module: calculate and score the spatiotemporal consistency between the current frame and historical image frame states, and adjust the multimodal fusion weights based on the scores.

[0083] The verification module includes a conflict arbitration execution module, a historical state maintenance module, a spatiotemporal consistency measurement module, and a conflict resolution execution module.

[0084] The conflict degree of each modal feature is measured through the conflict arbitration execution module: the conflict degree measurement formula is: Conflict=1-∏(1-|c i -c j |) Quantization inconsistency, (i,j)∈all modal pairs, c i and c j Arbitration is triggered when Conflict > 0.7 is detected, based on the confidence levels for different modalities; and the spatiotemporal consistency scoring formula S is used. consistency =0.6·spatial sim +0.4·temporal sim Modality optimization is performed in conflict scenarios, where spatial similarity (spatial_sim) measures spatial information between modalities by the consistency of feature point distribution, and temporal similarity (temporal) is used. sim The synchronization of time information between modalities is measured by action timing alignment degree calculation, and S is ultimately selected. consistency The highest mode was used as the arbitration result.

[0085] The historical image frame data buffer is managed through a historical state maintenance module. This buffer stores the fused feature state sequence and always retains valid state data from the most recent L frames. The default buffer length L is set to 10 frames. A timestamp is added to the feature state of each frame, and a temporal index relationship is established. Feature state retrieval is achieved by constructing a mapping table between the time dimension and the feature state.

[0086] The verification module may also include a feature trajectory optimization module, which uses the Kalman filter algorithm to smooth the feature state sequence. By establishing a motion model and an observation model of the feature state, the feature state in the historical buffer is used as the input filter of the observation sequence. The filter parameters are optimized by the system to set the process noise covariance Q = 0.01 and the observation noise covariance R = 0.1. The noise suppression and outlier correction of the feature trajectory are achieved through prediction-update loop iteration, and finally the smoothed feature state sequence is output.

[0087] The spatiotemporal consistency measurement module calculates the degree of spatiotemporal consistency between the current frame and historical image frames: Spatial dispersion is calculated based on the feature vectors of each modality in the current frame, and the standardized spatial variance index is obtained by dividing the sum of squared Euclidean distances between features by the number of modalities; the cumulative temporal difference between the features of the current frame and the features of the previous L-1 frames in the historical buffer is calculated using a sliding window approach, and the mean square error of feature changes is used as a temporal consistency measure; the spatial and temporal variance indices are fused, and a comprehensive consistency score is generated through a linear weighting method, where the balance coefficients α = 0.4 and β = 0.3. When the score is below 0.6, a conflict resolution process is triggered.

[0088] The multimodal fusion weights are adjusted through the conflict resolution execution module: when the conflict resolution process is triggered, an exponential decay strategy is used to reduce the weights of inconsistent modes, and the product of the original weights and the consistency scores is used as the new weight values; when it is detected that the consistency scores of all modes are lower than the threshold, a status flag is generated and the decision output of the current frame is paused; the weight values ​​released by the reduction are distributed proportionally to the highly consistent modes through normalization processing to ensure that the total weight remains constant, while setting an upper limit for the weight adjustment range to prevent drastic weight fluctuations.

[0089] Step 6: The output module integrates multimodal features based on the adjusted weights, and the classifier identifies the micro-motion categories based on the integrated multimodal features, outputting structured results.

[0090] Example 2

[0091] This invention also provides a multimodal feature analysis system for micro-motion recognition, including a preprocessing and feature extraction module, a synchronization and alignment module, a fusion module, a verification module, and an output module.

[0092] The preprocessing and feature extraction module performs multimodal data preprocessing and feature extraction: it generates image frames by copying the input video stream, inputs these image frames into various models for parallel processing, and extracts facial, hand, and pose features respectively.

[0093] The synchronization and alignment module performs temporal synchronization and spatial alignment of each modal feature: first, it determines the synchronization state of each modal feature, and then combines the corresponding features in chronological order for temporal alignment based on the synchronization state; second, it transforms the normalized coordinates of each model output to a unified pixel coordinate system, calculates the spatial offset of each modal feature, and corrects the differences in model output based on the spatial offset to achieve spatial alignment.

[0094] The fusion module performs cross-modal association and fusion: facial features are selected as anchor points as reference benchmarks, and the spatial positions of other modalities are dynamically adjusted to adapt to coordinate changes caused by head movements; a semantic mapping between modalities is established, and a sliding window is used to dynamically adjust the fusion weights of each modality for multimodal feature fusion; and a multimodal feature fusion confidence score is output based on a weighted geometric algorithm.

[0095] The verification module performs spatiotemporal verification and optimization after multimodal feature fusion: it calculates and scores the spatiotemporal consistency between the current frame and historical image frames, and adjusts the multimodal fusion weights based on the scores.

[0096] The output module integrates multimodal features based on the adjusted weights, and the classifier identifies the micro-motion categories based on the integrated multimodal features, outputting structured results.

[0097] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0098] Similarly, the system of the present invention forms a complete technical closed loop of multimodal feature acquisition, spatiotemporal alignment, semantic association, confidence fusion and conflict resolution, realizing accurate recognition and collaborative analysis of micro-movements, solving the problems of asynchronous multimodal feature acquisition, inconsistent spatial coordinates, weak semantic association and poor temporal consistency, and is suitable for application scenarios such as human-computer interaction, medical rehabilitation and security monitoring that require accurate recognition of subtle human movements.

[0099] It should be noted that not all steps and modules in the above processes and system structures are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments can be physical or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0100] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A multi-modal feature analysis method for micro-motion recognition, characterized in that The application relates to a micro-motion recognition method based on multi-modal feature analysis. Step 1: creating a multi-modal feature analysis system for micro-motion recognition, the system comprising a preprocessing and feature extraction module, a synchronization and alignment module, a fusion module, a verification module and an output module, Step 2: multi-modal data preprocessing and feature extraction through the preprocessing and feature extraction module: generating image frames according to the input video stream, inputting the image frames into each model for parallel processing, and extracting facial, hand and posture features respectively, Step 3: time synchronization and space alignment of each modal feature through the synchronization and alignment module: first determining the synchronization state of each modal feature, combining the corresponding features in time sequence according to the synchronization state for time sequence alignment; converting the normalized coordinates output by each model into a unified pixel coordinate system, calculating the spatial offset of each modal feature, correcting the model output difference according to the spatial offset, and realizing space alignment, Step 4: cross-modal correlation and fusion through the fusion module: screening the facial features as anchor points as the reference benchmark, dynamically adjusting the spatial positions of other modalities to adapt to the coordinate changes caused by head movement; establishing inter-modal semantic mapping, dynamically adjusting the fusion weights of each modality by using a sliding window, for multi-modal feature fusion, and outputting the multi-modal feature fusion confidence based on a weighted geometric algorithm, Step 5: time and space verification and optimization of the multi-modal feature fusion through the verification module: calculating the time and space consistency degree of the current frame and the historical image frame state and scoring, and adjusting the multi-modal fusion weight according to the score, Step 6: integrating the multi-modal features according to the adjusted weight through the output module, recognizing the micro-motion category according to the integrated multi-modal features through the classifier, and outputting the structured results.

2. The multi-modal feature analysis method for micro-motion recognition according to claim 1, characterized in that The preprocessing and feature extraction module in step 2 includes a parallel processing flow construction module. The parallel processing flow construction module uses a three-channel memory mapping method to generate image frames according to the input video stream, and inputs the image frames into each model for parallel processing to extract facial, hand, and posture features. The parallel processing flow construction module creates three physically continuous but logically independent memory mapping areas for each frame of the input video stream, and stores a facial image copy I face , a hand image copy I hand , and a posture image copy I pose , respectively. Each image copy retains the color space and resolution attributes of the original image, and the three image copies are loaded into the corresponding models for parallel extraction of facial, hand, and posture features.

3. The multi-modal feature analysis method for micro-motion recognition according to claim 1, The synchronization and alignment module in step 3 comprises a timestamp synchronization module, a data alignment verification module, a coordinate system conversion module, a space alignment module and a deviation correction module, The synchronization state of each modality feature is determined by the timestamp synchronization module: a timestamp difference matrix is constructed t face , t hand , t pose are respectively the face feature timestamp, the hand feature timestamp and the pose feature timestamp of the same frame; a synchronization threshold δ t is set, according to the formula: delta t = 1.2 x (1 / fps) real-time adjustment delta t , fps is the frame rate of the video stream, dynamically changing between 30 to 60 frames, if the maximum value of all elements in the timestamp difference matrix D of the last three frames satisfies max(D) < delta t , the corresponding feature group is determined to be in a synchronous state, The data alignment verification module combines the corresponding features in time sequence according to the synchronization state for timing alignment: first, check whether the timestamp difference is within the synchronization threshold range; adopt a double-pointer ring queue design, set up face feature queue Q face , hand feature queue Q hand and posture feature queue Q pose , each queue has a depth of 8 frames, and when synchronization failure is detected, the queue is cleaned until each feature reaches a synchronization state; combine the synchronized features in time sequence into a feature tensor T ∈ R (N×3×K) , where N is the total number of feature points, 3 represents the three-dimensional coordinate dimension, K is the time series length, and the output is a multi-modal feature data packet with a timestamp; The normalized coordinates output by each model are converted into a unified pixel coordinate system through the coordinate system conversion module: establishing an image pixel coordinate system as a benchmark reference system, and converting the normalized coordinates output by each model into pixel coordinates by using a homogeneous coordinate transformation method; The spatial offset of each modal feature is calculated through the space alignment module: clustering analysis of feature points, grouping and classifying facial, hand and body posture feature points output by different models; based on a Euclidean distance similarity measurement function, the spatial offset between different feature point sets is calculated; the iterative closest point algorithm is used to gradually reduce the positional deviation between the feature point sets through multiple rotation and translation transformations; The space alignment is realized by correcting the model output difference according to the spatial offset through the deviation correction module: taking the smaller value of the image width and height as the depth scaling benchmark parameter, and correcting the model output difference according to the spatial offset.

4. The multi-modal feature analysis method for micro-motion recognition according to claim 1, characterized in that The fusion module in step 4 comprises a facial reference system construction module, a spatial transformation calculation module, a semantic association modeling module and a confidence fusion calculation module, The face reference system construction module takes the screened face features as anchor points as reference benchmarks: through real-time video stream detection of face key points, the nose tip point P nose , the left outer corner of the eye The right outer corner of the eye Three feature points; The spatial positions of other modalities are dynamically adjusted through the spatial transformation calculation module to adapt to the coordinate changes caused by head movement; The semantic inter-modal mapping is established by a semantic correlation modeling module: a mapping relationship between the finger feature point coordinates P finge r and the face feature point coordinates P face is established; the Euclidean distance d = ||P finger -P face || therebetween is calculated, the contact relationship determination is triggered when it is detected that d < 15 pixels, and a spatial correlation marker is generated, and a semantic correlation function is constructed by using a Gaussian kernel function: f semantic = exp(-d 2 / 144.5), where the scale parameter σ = 8.5 pixels is determined by calibration experiments, and the output value range of the function is [0, 1], which is used to quantify the spatial correlation strength; The confidence fusion calculation module adopts a sliding window to dynamically adjust the fusion weights of each modality, is used for multi-modal feature fusion, and outputs a multi-modal feature fusion confidence based on a weighted geometric algorithm: the fusion weight is dynamically adjusted by a weight update formula w i (t) = 0.85·w i (t-1) + 0.15·(acc i / ∑acc j ), ensuring the constraint condition ∑w i = 1, acc i is the historical accuracy of the i th modality in the sliding window, and ∑acc j represents the sum of the historical accuracies of all modalities in the sliding window; the formula: C fused represents the comprehensive confidence after fusion, c i is the original confidence of the i th modality, and when C fused > 0.6, a micro-motion effective determination signal is generated, and the fusion confidence value is output.

5. The multi-modal feature analysis method for micro-motion recognition according to claim 1, characterized in that The verification module in step 5 comprises a conflict arbitration execution module, a historical state maintenance module, a time and space consistency measurement module and a conflict resolution execution module, Conflict degree measurement of each modality feature by conflict arbitration execution module: conflict degree measurement formula is adopted: Conflict = 1 - ∏(1 - |c i -c j |) quantifies inconsistency, (i,j) is all modality pairs, c i and c j are confidence degrees of different modalities, when Conflict>0.7 is detected, arbitration is triggered; modality optimization under conflict scene is carried out based on a space-time consistency score formula S consistency =0.6·spatial sim +0.4·temporal sim , wherein spatial similarity spatial_sim measures inter-modality spatial information through feature point distribution consistency, time similarity temporal sim is calculated by measuring the synchronization of inter-modality time information through action time sequence alignment, finally the S consistency highest modality is selected as the arbitration result, The historical image frame data buffer is managed through a historical state maintenance module. This buffer stores the fused feature state sequence and always retains valid state data from the most recent L frames. The default buffer length L is set to 10 frames. A timestamp is added to the feature state of each frame, and a temporal index relationship is established. Feature state retrieval is achieved by constructing a mapping table between the time dimension and the feature state. The spatiotemporal consistency measurement module calculates the degree of spatiotemporal consistency between the current frame and historical image frames: Spatial dispersion is calculated based on the feature vectors of each modality in the current frame, and the standardized spatial variance index is obtained by dividing the sum of squared Euclidean distances between features by the number of modalities; the cumulative temporal difference between the features of the current frame and the features of the previous L-1 frames in the historical buffer is calculated using a sliding window approach, and the mean square error of feature changes is used as a temporal consistency measure; the spatial and temporal variance indices are fused, and a comprehensive consistency score is generated through a linear weighting method, where the balance coefficients α = 0.4 and β = 0.

3. When the score is below 0.6, a conflict resolution process is triggered. The multimodal fusion weights are adjusted through the conflict resolution execution module: when the conflict resolution process is triggered, an exponential decay strategy is used to reduce the weights of inconsistent modes, and the product of the original weights and the consistency scores is used as the new weight values; when it is detected that the consistency scores of all modes are lower than the threshold, a status flag is generated and the decision output of the current frame is paused; the weight values ​​released by the reduction are distributed proportionally to the highly consistent modes through normalization processing to ensure that the total weight remains constant, while setting an upper limit for the weight adjustment range to prevent drastic weight fluctuations.

6. A multi-modal feature analysis system for micro-motion recognition, characterized by It includes a preprocessing and feature extraction module, a synchronization and alignment module, a fusion module, a verification module, and an output module. The preprocessing and feature extraction module performs multimodal data preprocessing and feature extraction: it generates image frames by copying the input video stream, inputs these image frames into various models for parallel processing, and extracts facial, hand, and pose features respectively. The synchronization and alignment module performs temporal synchronization and spatial alignment of each modal feature: first, it determines the synchronization state of each modal feature, and then combines the corresponding features in chronological order for temporal alignment based on the synchronization state; second, it transforms the normalized coordinates of each model output to a unified pixel coordinate system, calculates the spatial offset of each modal feature, and corrects the differences in model output based on the spatial offset to achieve spatial alignment. The fusion module performs cross-modal association and fusion: facial features are selected as anchor points as reference benchmarks, and the spatial positions of other modalities are dynamically adjusted to adapt to coordinate changes caused by head movements; a semantic mapping between modalities is established, and a sliding window is used to dynamically adjust the fusion weights of each modality for multimodal feature fusion; and a multimodal feature fusion confidence score is output based on a weighted geometric algorithm. The verification module performs spatiotemporal verification and optimization after multimodal feature fusion: it calculates and scores the spatiotemporal consistency between the current frame and historical image frames, and adjusts the multimodal fusion weights based on the scores. The output module integrates multimodal features based on the adjusted weights, and the classifier identifies the micro-motion categories based on the integrated multimodal features, outputting structured results.

7. The multi-modal feature analysis system for micro-motion recognition according to claim 6, characterized in that The preprocessing and feature extraction module comprises a parallel processing flow construction module. The parallel processing flow construction module adopts a three-channel memory mapping method to generate image frames according to input video streams, and inputs the image frames into each model for parallel processing to extract facial, hand and posture features. The parallel processing flow construction module creates three physically continuous but logically independent memory mapping areas for each frame of the input video stream, and stores a facial image copy I face , a hand image copy I hand and a posture image copy I pose , respectively. Each image copy retains the color space and resolution attributes of the original image, and the three image copies are loaded into corresponding models for parallel extraction of facial, hand and posture features.

8. The multi-modal feature analysis system for micro-motion recognition according to claim 6, characterized by synchronization The alignment module comprises a timestamp synchronization module, a data alignment verification module, a coordinate system conversion module, a spatial alignment module, and a deviation correction module, The synchronization state of each modality feature is determined by the timestamp synchronization module: a timestamp difference matrix is constructed t face , t hand , t pose are respectively the face feature timestamp, the hand feature timestamp and the pose feature timestamp of the same frame; a synchronization threshold δ t is set, according to the formula: delta t = 1.2 x (1 / fps) real-time adjustment delta t , fps is the frame rate of the video stream, dynamically changing between 30 to 60 frames, if the maximum value of all elements in the timestamp difference matrix D of the last three frames satisfies max(D) < delta t , the corresponding feature group is determined as a synchronous state, The data alignment verification module combines the corresponding features in time sequence according to the synchronization state for timing alignment: first, check whether the timestamp difference is within the synchronization threshold range; adopt a double-pointer ring queue design, set the face feature queue Q face , the hand feature queue Q hand and the posture feature queue Q pose , each queue has a depth of 8 frames, and when synchronization failure is detected, the queue is cleaned until each feature reaches a synchronization state; combine the synchronized features in time sequence into a feature tensor T ∈ R (N×3×K) , where N is the total number of feature points, 3 represents the three-dimensional coordinate dimension, K is the time series length, and the output is a multi-modal feature data packet with a timestamp; The normalized coordinates output by each model are converted to a unified pixel coordinate system by the coordinate system conversion module: an image pixel coordinate system is established as a reference frame, and the normalized coordinates output by each model are converted to pixel coordinates by using a homogeneous coordinate transformation method; The spatial alignment module is used to calculate the spatial offset of each modality feature space: the feature points are clustered and analyzed, and the face, hand, and body posture feature points output by different models are grouped and classified; based on the similarity measurement function of the Euclidean distance, the spatial offset between different feature point sets is calculated; the iterative closest point algorithm is used to gradually reduce the positional deviation between the feature point sets through multiple rotation and translation transformations; The deviation correction module is used to correct the differences in model output according to the spatial offset, so as to realize spatial alignment: the smaller value of the image width and height is taken as the depth scaling reference parameter, and the differences in model output are corrected according to the spatial offset.

9. The multi-modal feature analysis system for micro-motion recognition according to claim 6, characterized in that The fusion module comprises a face reference system construction module, a spatial transformation calculation module, a semantic association modeling module, and a confidence fusion calculation module, The face reference system construction module takes the screened face features as anchor points as reference benchmarks: through real-time video stream detection of face key points, the nose tip point P nose , the left outer corner of the eye the right outer corner of the eye three feature points; The spatial transformation calculation module is used to dynamically adjust the spatial positions of other modalities to adapt to the coordinate changes caused by head movement; The semantic inter-modal mapping is established by a semantic correlation modeling module: a mapping relationship between the finger feature point coordinates P finge r and the face feature point coordinates P face is established; the Euclidean distance d = ||P finger -P face || therebetween is calculated, the contact relationship is determined when it is detected that d < 15 pixels, and a spatial correlation mark is generated; and a semantic correlation function is constructed by using a Gaussian kernel function: f semantic = exp(-d 2 / 144.5), where the scale parameter σ = 8.5 pixels is determined by calibration experiments, and the output value range of the function is [0, 1], which is used to quantify the spatial correlation strength; The confidence fusion calculation module adopts a sliding window to dynamically adjust the fusion weights of each modality, is used for multi-modal feature fusion, and outputs a multi-modal feature fusion confidence based on a weighted geometric algorithm: the fusion weight is dynamically adjusted by a weight update formula w i (t) = 0.85·w i (t-1) + 0.15·(acc i / ∑acc j ), ensuring the constraint condition ∑w i = 1, acc i is the historical accuracy of the i th modality in the sliding window, and ∑acc j represents the sum of the historical accuracies of all modalities in the sliding window; the formula: C fused represents the integrated confidence after fusion, c i is the original confidence of the i th modality, and when C fused > 0.6, a micro-motion effective determination signal is generated, and the fusion confidence value is output.

10. The multi-modal feature analysis system for micro-motion recognition according to claim 6, characterized in that The verification module comprises a conflict arbitration execution module, a historical state maintenance module, a spatiotemporal consistency measurement module, and a conflict resolution execution module, Conflict degree measurement of each modality feature by conflict arbitration execution module: conflict degree measurement formula is adopted: Conflict = 1 - ∏(1 - |c i - c j |) quantifies inconsistency, (i,j) is all modality pairs, c i and c j are confidence degrees of different modalities, when Conflict>0.7 is detected, arbitration is triggered; modality optimization under conflict scene is carried out based on a space-time consistency score formula S consistency =0.6·spatial sim +0.4·temporal sim , wherein spatial similarity spatial_sim measures inter-modality spatial information through feature point distribution consistency, time similarity temporal sim is calculated by measuring the synchronization of inter-modality time information through action time sequence alignment, finally the S consistency highest modality is selected as the arbitration result, The historical state maintenance module is used to manage the fixed-length historical image frame data buffer: the historical image frame data buffer is used to store the fusion feature state sequence, the historical image frame data buffer always retains the effective state data of the most recent L frames, the default buffer length L is set to 10 frames, a timestamp label is added to each frame of feature state, and a time sequence index relationship is established, feature state retrieval is realized by constructing a mapping table of the time dimension and the feature state, The spatiotemporal consistency measurement module is used to calculate the spatiotemporal consistency degree of the current frame and the historical image frame state: the spatial dispersion degree is calculated based on the feature vectors of each modality of the current frame, the sum of the squares of the Euclidean distances between the features is divided by the number of modalities to obtain the standardized spatial variance index; the cumulative time difference between the features of the current frame and the features of the previous L-1 frames in the historical buffer is calculated by using a sliding window method, and the mean square error of the feature change amount is used as the time consistency measurement; the spatial and temporal variance indexes are fused to generate a comprehensive consistency score by linear weighting, where the balance coefficients α=0.4 and β=0.3, and the conflict resolution process is triggered when the score is lower than 0.6, The conflict resolution execution module is used to adjust the multi-modality fusion weight: when the conflict resolution process is triggered, an exponential decay strategy is used to reduce the weight of the inconsistent modality, and the product of the original weight and the consistency score is taken as the new weight value; when it is detected that all modality consistency scores are lower than the threshold, a state identifier is generated and the decision output of the current frame is paused; the weight value released by the weight reduction is proportionally distributed to the high-consistency modality by normalization processing, ensuring that the total weight sum remains constant, and an upper limit is set for the weight adjustment amplitude to prevent drastic fluctuations in the weight.

Images