A cross-domain video micro-motion capture and modeling method

Abstract

The application discloses a cross-domain video micro-motion capture and modeling method, relates to the technical field of micro-motion capture and modeling, extracts 2D pose data of each joint of a human body in a frame image, converts the 2D pose data into a heat map, splices the heat map of all joints in a frame according to a channel dimension, stacks the heat maps of all frames along a time dimension to form a 3D heat map sequence, compresses the 3D heat map sequence, clips the heat map by using a bounding box capable of surrounding all pose data in all frame image sequences, extracts features and classifies the 3D heat map sequence by using 3D-CNN, removes spatial down-sampling operation of a shallow layer at the front end of the network, and reduces the number of residual stack layers and the number of basic convolution channels, and the model adopts an asymmetric joint training strategy for cross-domain training, and a source domain training set participates in training in full amount and a core subset of a target domain training set participates in training. The application solves the problem of low recognition accuracy of traditional cross-domain micro-motion capture.

Description

Technical Field

[0001] This invention relates to the field of micro-motion capture and modeling technology, and in particular to a cross-domain video micro-motion capture and modeling method. Background Technology

[0002] Micro-motion capture and modeling technology is an interdisciplinary research area encompassing computer vision, pattern recognition, and multimodal behavior analysis. It primarily studies how to automatically identify and model minute, short-duration movements from video or other sensor data. Since micro-motions often contain psychological cues such as emotional changes, cognitive load, stress levels, and self-regulation behaviors, they are valuable in scenarios such as mental health recognition, micro-expression analysis, human-computer interaction, behavioral understanding, medical rehabilitation monitoring, and driver status monitoring. However, existing video micro-motion capture methods typically rely on a single dataset to build models, resulting in high recognition performance on the same dataset but insufficient generalization ability on new datasets.

[0003] Meanwhile, current micro-motion datasets exhibit significant differences in device type, resolution, frame rate, lighting conditions, shooting distance, background environment, and video compression methods. These differences lead to inconsistent visual representations of micro-motion signals across different data sources. Because micro-motions are characterized by extremely small amplitude, short duration, and localized spatial regions, models are highly sensitive to the quality of the input signal. When applied across different datasets, problems such as detection failure, feature distortion, or a significant drop in recognition accuracy frequently occur. Summary of the Invention

[0004] To overcome the shortcomings of traditional technologies in terms of accuracy in capturing micro-motions across domains, this invention provides a cross-domain video micro-motion capture and modeling method.

[0005] To achieve the above objectives, the present invention adopts the following technical solution, including: A cross-domain video micro-motion capture and modeling method includes the following steps: S1, Input frame image sequence, extract human posture data from the frame image, the posture data includes 2D posture data of each joint point; S2, convert the 2D pose data of each joint into a heatmap, stitch together the heatmaps of all relevant nodes in a frame according to the channel dimension to obtain a heatmap of a frame, and then stack the heatmaps of all frames along the time dimension to form a 3D heatmap sequence. S3, compress the 3D heatmap sequence by using bounding boxes that can surround all pose data in all frame images to crop the heatmaps of all frames and obtain the compressed 3D heatmap sequence. S4 uses an optimized 3D convolutional neural network as a micro-motion recognition model to extract features and classify the compressed 3D heatmap sequence, and output the predicted probability of each micro-motion category. The optimization of the 3D convolutional neural network includes: removing the spatial downsampling operation of the shallow layer at the front end of the network, and reducing the number of residual stacking layers and the number of basic convolution channels. The micro-action recognition model employs an asymmetric joint training strategy for cross-domain training, with the entire source domain training set participating in the training and a core subset of the target domain training set participating in the training.

[0006] Preferably, in step S1, the attitude data consists of triples for each joint. Composition, in which It is the confidence score of the k-th key point. These are the two-dimensional coordinates of the k-th joint.

[0007] Preferably, in step S2, the triples of each joint point are converted into a heatmap, and a graph is generated at each joint point location. A 2D Gaussian heatmap centered on the first point, The formula for calculating the heatmap of each joint is: ;

[0008] in, It controls the variance of the Gaussian heatmap; Indicates the row and column number of a pixel on the heatmap; The pixel position in the heatmap of the kth joint ( The normalized response intensity value on ) By stitching together the heatmaps of all relevant nodes in a frame according to their channels, with each node representing one channel, a heatmap of size [size missing] is obtained. ;in, The pixel dimensions of the heatmap for the joint. Image height, Image width, This refers to the number of key points; Stack the heatmaps of all frames along the time dimension to form a final size of 3D heatmap sequence; among which, This represents the number of frames.

[0009] Preferably, in step S3, in the spatial dimension, the smallest bounding box that can surround all pose data in the entire frame image sequence is found as the global bounding box; the heatmaps of all frames are cropped according to the global bounding box and scaled to the target size to obtain the compressed 3D heatmap sequence.

[0010] Preferably, in step S4, a three-dimensional convolutional neural network SlowOnly is used as the original network, and the original network is optimized.

[0011] Preferably, for the target domain training set, a stratified random sampling method is used to obtain the core subset.

[0012] Preferably, during the training iteration process, an asymmetric micro-batch mechanism is used, where the total number of samples in the micro-batch is... ,in, Each sample is taken from the full source domain training set. Each sample is taken from the core subset of the target domain training set. .

[0013] Preferably, a domain-independent evaluation mechanism is used for the trained model. After training, the model weights are frozen, and the model is evaluated separately on the source domain test set. and target domain test set Perform independent reasoning.

[0014] The present invention provides a readable storage medium having a computer program stored thereon, which, when executed, implements the cross-domain video micro-motion capture and modeling method.

[0015] The present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-domain video micro-motion capture and modeling method described above.

[0016] The advantages of this invention are: (1) This invention abandons the original RGB pixel dependency and eliminates the domain offset problem caused by different datasets in terms of shooting device, resolution, frame rate, lighting, background and video compression noise; it only retains the coordinates and confidence of human joints to construct Gaussian heatmaps, filters invalid information such as color, texture and cluttered background, and improves the basic ability of the model to generalize across datasets from the input end.

[0017] (2) This invention only stores the triplet of the key points, without the need to save the high-resolution original image / feature map; it can generate heat maps online in real time, which greatly saves disk storage and data read and write bandwidth, and reduces the engineering deployment cost.

[0018] (3) By changing the composition of the training dataset, this invention alleviates the differences between different datasets, improves the generalization ability of the model, and can effectively improve the accuracy of cross-domain micro-motion capture.

[0019] (4) This invention finds the smallest bounding box that can surround all pose data in the entire video frame image sequence, and crops and scales the heatmaps of all frames. On the one hand, it removes invalid backgrounds, compresses redundant computing power, concentrates and retains the effective area of ​​human poses, filters irrelevant background interference over a large area, reduces input spatial resolution, simplifies the computation of subsequent networks, and takes into account both feature purity and inference efficiency. On the other hand, it avoids inter-frame jitter interference, avoids micro-motion offset distortion caused by frame-by-frame bounding box fluctuations, ensures the consistency of temporal spatial benchmarks, and lays the foundation for stable modeling of micro-motion temporal changes in 3D convolution.

[0020] (5) This invention removes the early spatial downsampling operation of the original 3D-CNN front end to adapt to the low spatial resolution heatmap input after cropping; avoids the smoothing of small joint displacements and local micro-motion details due to over-downsampling, and ensures that 3D convolution can effectively capture fine spatial pose changes.

[0021] (6) Since the input heatmap is already a structured mid-level pose feature, there is no need for the network to extract the basic information of bottom edge, texture and light and shadow. Therefore, this invention significantly reduces the number of model parameters and FLOPs by reducing the number of residual stacking layers and compressing the number of basic convolution channels, thereby improving the training convergence speed and real-time inference performance, while suppressing the risk of overfitting of complex deep networks in small sample and cross-domain scenarios.

[0022] (7) This invention does not require full merging of multi-domain datasets for training, nor does it require traditional full fine-tuning; it only uses the full source domain data and hierarchical sampling of the core subset of the target domain to construct asymmetric micro-batches, with almost no additional large-scale computational overhead, achieving low-cost cross-domain feature alignment and balancing training cost and cross-domain generalization ability. Most source domain samples dominate the training to solidify the basic feature space, while a small number of target domain samples are used for consistency regularization constraints; this guides the network to learn robust pose and motion features shared between different domains, while ensuring that the model's basic recognition ability in the source domain does not degrade, suppressing catastrophic forgetting, and taking into account the accuracy of both the source and target domains.

[0023] (8) The hierarchical random sampling of the present invention ensures that the core subset of the target domain strictly replicates the prior distribution of the original target domain dataset categories, avoiding feature alignment failure caused by sampling skew; combined with the domain independent evaluation mechanism, the cross-domain transfer effect of the model can be quantitatively verified, and the algorithm iteration has strong controllability.

[0024] (9) This invention forms a technical closed loop from the entire link of input representation optimization, spatial normalization preprocessing, lightweight transformation of backbone network, and cross-domain training strategy optimization. On the one hand, it completely solves the shortcomings of existing micro-motion recognition models, such as reliance on a single dataset, poor cross-domain generalization, severe interference from environmental devices, and easy loss of micro-motion features. On the other hand, under the premise of significantly reducing storage costs, computing overhead and training costs, it significantly improves the accuracy of micro-motion capture and classification recognition in complex cross-domain scenarios. It can be stably adapted to practical application scenarios such as mental health analysis, human-computer interaction, medical rehabilitation monitoring, and driver status assessment. It has strong engineering practicality and technical versatility. Attached Figure Description

[0025] Figure 1 This is an architecture diagram of a cross-domain video micro-motion capture and modeling method according to the present invention.

[0026] Figure 2 This is a flowchart of a cross-domain video micro-motion capture and modeling method according to the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] This invention provides a cross-domain video micro-motion capture and modeling method, primarily based on 3D-CNN (3D Convolutional Neural Network). The overall process is as follows: Figure 1 As shown, it specifically includes the following content.

[0029] (1) Micro-motion recognition process (by Figure 2 (As shown) S1, Input a sequence of frame images from a video, and use a 2D Top-Down pose estimation method to extract pose data of the human body in the frame images. The pose data includes 2D pose data of each joint.

[0030] First, the human body in the frame image is selected using the Faster-RCNN object detector. Then, the pose estimator HRNet is used to extract pose data (2D pose data of each joint). To save disk space, the extracted pose data is not directly saved as feature maps, but rather as triples of joints. .in, It is the highest confidence score of the k-th key point. These are the two-dimensional coordinates of the k-th joint.

[0031] S2, convert the triples of each joint into heatmaps, stitch together the heatmaps of all relevant nodes in a frame of image by channel to obtain a heatmap of a frame, and then stack the heatmaps of all frames along the time dimension to form a 3D heatmap sequence.

[0032] First, using the saved triples of each joint, generate a set of values ​​at each joint position. A 2D Gaussian heatmap centered on the first point, The formula for calculating the Gaussian heatmap of each key point is: ; in, It controls the variance of the Gaussian heatmap; Indicates the row and column number of a pixel on the heatmap; The pixel position in the heatmap of the kth joint ( The normalized response intensity value on ).

[0033] Next, the heatmaps of all nodes in a frame are stitched together by channel (each node is a separate channel) to obtain a heatmap of a single frame with a size of [size missing]. .in, The pixel dimensions of the heatmap for the joint. Image height, Image width, This represents the number of joints.

[0034] Then, the heatmaps of all frames are stacked along the time dimension to form a final size of A sequence of 3D heatmaps. Among them, This represents the number of frames.

[0035] S3 compresses the 3D heatmap sequence.

[0036] First, in the spatial dimension, find the smallest bounding box that can enclose all pose data in the entire video frame image sequence, and use it as the global bounding box.

[0037] Then, the heatmaps of all frames are cropped based on the global bounding box and scaled to the target size to remove irrelevant background areas, resulting in a compressed 3D heatmap sequence.

[0038] S4 employs an optimized 3D-CNN as the micro-action recognition model to extract features and classify the compressed 3D heatmap sequence, outputting the predicted probability of each micro-action category. The optimizations to the 3D-CNN include: removing spatial downsampling operations from the shallow layers of the network front-end, and reducing the number of residual stacking layers and the number of basic convolutional channels.

[0039] In this embodiment, a three-dimensional convolutional neural network, SlowOnly, which is widely used in action recognition tasks based on RGB data, is selected as the original network.

[0040] Then, based on the optimization of the original network, the following two aspects of optimization are included: 1. Since the spatial resolution of the 3D heatmap sequence has been cropped and is much smaller than that of the original RGB video frames, the early spatial downsampling operation of the network was removed.

[0041] 2. Since the input heatmap is already extracted mid-level features, there is no need for an extremely deep network to extract the underlying texture. Therefore, a network structure with fewer layers and fewer channels is adopted to reduce computational complexity.

[0042] In summary, during the forward propagation of the optimized network, the spatial resolution of the feature maps gradually decreases with the stacking of residual layers, while the channel dimension increases accordingly. The high-dimensional features are compressed into a one-dimensional full-range semantic vector via a global average pooling layer and then fed into a fully connected layer to map the predicted probabilities of each micro-action category.

[0043] (2) Cross-domain training strategy of the model Traditional cross-domain fine-tuning often leads to severe catastrophic forgetting of the model in the source domain, while merging the two datasets (source domain dataset and target domain dataset) for training introduces huge computational overhead.

[0044] To achieve the optimal balance between training cost and cross-domain generalization performance, this invention proposes an efficient asymmetric joint training strategy. This strategy maintains the source domain training set... With the main training set fully involved, a core subset of the target domain training set is strategically introduced to guide the model to learn robust feature representations with cross-domain alignment without increasing additional training overhead.

[0045] To strictly control training costs, the training set in the target domain is not used. Instead of performing a full iteration, stratified random sampling is used to construct a very small core subset of the target domain. Hierarchical random sampling ensures the selection of the core subset of the target domain. It can perfectly preserve the prior class distribution of the original target domain dataset.

[0046] During the training iterations, an asymmetric mini-batch mechanism is employed. Let the total capacity (total number of samples) of the mini-batch be... It is divided into two parts: the vast majority of samples (sample size) Sampled from the full source domain training set Very few samples (sample size) Sampled from the core subset of the target domain training set ,Right now Much larger This asymmetric batch construction method not only ensures the stability of the main task feature space, but also effectively performs consistency regularization on the feature manifold using the target domain data.

[0047] To fully verify the effectiveness of this asymmetric joint training strategy, a domain-independent evaluation mechanism was adopted. After training, the model weights were frozen, and evaluation was performed on the source domain test set. and target domain test set Independent reasoning is performed on the model. This evaluation mechanism can intuitively quantify the model's ability to absorb knowledge from the target domain (manifested as...). While improving accuracy, the degree of preservation of the inherent knowledge of the source domain (manifested as...) (Accuracy showed no significant degradation).

[0048] Example 1 To verify the effectiveness of this invention, this embodiment was tested on the constructed datasets. Specifically, dataset MA-52 was constructed using interview videos under natural conditions, and dataset MA-82 was constructed using videos of subjects performing under controlled conditions. The two datasets were divided into training and testing sets according to a set ratio. Using the above method, training was performed on the MA-82 training set as a baseline, and benchmark results were obtained by testing on both the MA-82 and MA-52 testing sets. For comparison, a core subset of the MA-52 training set was added to the MA-82 training set as a new training set, and results were obtained by testing on both the MA-82 and MA-52 testing sets. Simultaneously, training was also performed on MA-52 as a baseline using the above method, and all results are shown in Table 1.

[0049] Table 1. Classification Accuracy Comparison Table

[0050] Experimental results show that the training strategy proposed in this invention achieves the best balance between training cost and cross-domain generalization performance.

[0051] This invention proposes a cross-domain video micro-motion capture and modeling method, aiming to improve the accuracy of cross-domain video micro-motion capture by utilizing an asymmetric joint training strategy. This strategy has a wide range of applications, including mental health assessment, human-computer interaction, sports training, behavior analysis, and security monitoring, and has significant practical value and broad application prospects.

[0052] In mental health assessment scenarios, this invention can accurately capture subtle body movements from interview videos, psychological consultation recordings, and natural interaction scenarios. By enhancing the modeling of micro-displacements of key parts such as the head, fingers, toes, and shoulders, the model can identify micro-movement features such as "toe touching the ground and shaking", "finger tapping", and "torso leaning forward and backward".

[0053] In human-computer interaction and intelligent systems, this invention can serve as a micro-motion level motion perception layer, obtaining consistent micro-motion representations from different cameras, terminal devices, or user scenarios. The system can capture subtle gestures, micro-expressions, or posture changes of users in real time, driving various interaction modes such as intelligent assistants, emotion response systems, and wearable device feedback.

[0054] In behavioral monitoring and safety early warning applications, this invention can capture abnormal micro-movements in real time, such as involuntary shaking of hospital patients, slight pre-fall movements of the elderly, or fatigue-related micro-behaviors of drivers (such as slight head nodding or relaxed hands). Through cross-domain migration capabilities, the system can maintain consistent motion detection performance in different deployment environments, thereby significantly improving the universality and reliability of the safety early warning system.

[0055] In summary, the cross-domain video micro-motion capture and modeling method proposed in this invention can achieve stable micro-motion recognition and parametric modeling under multiple scenarios and data sources. This method not only significantly improves the accuracy and robustness of cross-domain micro-motion detection, but also provides reliable technical support for multiple industries such as psychological research, intelligent interactive systems, sports training optimization, and public safety monitoring, demonstrating significant application value and broad market prospects.

[0056] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-domain video micro-motion capture and modeling method, characterized in that, Includes the following steps: S1, Input frame image sequence, extract human posture data from the frame image, the posture data includes 2D posture data of each joint point; S2, convert the 2D pose data of each joint into a heatmap, stitch together the heatmaps of all relevant nodes in a frame according to the channel dimension to obtain a heatmap of a frame, and then stack the heatmaps of all frames along the time dimension to form a 3D heatmap sequence. S3, compress the 3D heatmap sequence by using bounding boxes that can surround all pose data in all frame images to crop the heatmaps of all frames and obtain the compressed 3D heatmap sequence. S4 uses an optimized 3D convolutional neural network as a micro-motion recognition model to extract features and classify the compressed 3D heatmap sequence, and output the predicted probability of each micro-motion category. The optimization of the 3D convolutional neural network includes: removing the spatial downsampling operation of the shallow layer at the front end of the network, and reducing the number of residual stacking layers and the number of basic convolution channels. The micro-action recognition model employs an asymmetric joint training strategy for cross-domain training, with the entire source domain training set participating in the training and a core subset of the target domain training set participating in the training.

2. The cross-domain video micro-motion capture and modeling method according to claim 1, characterized in that, In step S1, the attitude data consists of triples for each joint. Composition, in which It is the confidence score of the k-th key point. These are the two-dimensional coordinates of the k-th joint.

3. The cross-domain video micro-motion capture and modeling method according to claim 2, characterized in that, In step S2, the triples of each joint point are converted into heatmaps, and a graph is generated at each joint point location. A 2D Gaussian heatmap centered on the first point, The formula for calculating the heatmap of each joint is: in, It controls the variance of the Gaussian heatmap; Indicates the row and column number of a pixel on the heatmap; The pixel position in the heatmap of the kth joint ( The normalized response intensity value on ) By stitching together the heatmaps of all relevant nodes in a frame according to their channels, with each node representing one channel, a heatmap of size [size missing] is obtained. ;in, The pixel dimensions of the heatmap for the joint. Image height, Image width, This refers to the number of key points; Stack the heatmaps of all frames along the time dimension to form a final size of 3D heatmap sequence; among which, This represents the number of frames.

4. The cross-domain video micro-motion capture and modeling method according to claim 1, characterized in that, In step S3, in the spatial dimension, the smallest bounding box that can enclose all pose data in the entire frame image sequence is found as the global bounding box; the heatmaps of all frames are cropped according to the global bounding box and scaled to the target size to obtain the compressed 3D heatmap sequence.

5. The cross-domain video micro-motion capture and modeling method according to claim 1, characterized in that, In step S4, a three-dimensional convolutional neural network, SlowOnly, is used as the original network to optimize the original network.

6. The cross-domain video micro-motion capture and modeling method according to claim 1, characterized in that, For the target domain training set, a stratified random sampling method is used to obtain the core subset.

7. The cross-domain video micro-motion capture and modeling method according to claim 1, characterized in that, During the training iteration process, an asymmetric micro-batch mechanism is employed, with the total number of samples in each micro-batch set being... ,in, Each sample is taken from the full source domain training set. Each sample is taken from the core subset of the target domain training set. .

8. The cross-domain video micro-motion capture and modeling method according to claim 1, characterized in that, The trained model is evaluated using a domain-independent mechanism. After training, the model weights are frozen, and the model is evaluated on the source domain test set. and target domain test set Perform independent reasoning.

9. A readable storage medium, characterized in that, It stores a computer program, which, when executed, implements a cross-domain video micro-motion capture and modeling method as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a cross-domain video micro-motion capture and modeling method as described in any one of claims 1 to 8.

Images