A micro-motion recognition method based on a mask autoencoder and a hybrid expert model

Abstract

The application provides a micro-motion recognition method based on a mask autoencoder and a hybrid expert model, which comprises the following steps: encoding acquired bone key point data into a multidimensional heat map; feature extraction: inputting the multidimensional heat map into a pre-trained heat map mask autoencoder module, and extracting fine-grained motion features through the encoder of the mask autoencoder module; hand-dependent gate control; expert processing: after the multidimensional heat map is subjected to hand-related and hand-unrelated feature processing, the multidimensional heat map is input into a first expert model and a second expert model respectively, and first expert output and second expert output are obtained; prediction fusion: according to first gate weights and second gate weights, the first expert output and the second expert output are weighted and fused to obtain expert fusion results. Through the cooperation between the expert models, more accurate recognition can be realized, the 'hand interference' in micro-motion recognition can be effectively alleviated, and the accuracy and robustness are improved.

Description

Technical Field

[0001] This application relates to the field of micro-motion recognition technology, specifically a micro-motion recognition method based on mask autoencoder and hybrid expert model. Background Technology

[0002] Micromotors (MAs) are rapid, subtle behaviors exhibiting extremely low intensity. Unlike regular movements, micromotors are not used for illustrative or communicative purposes, but rather occur as spontaneous or involuntary bodily responses to specific stimuli, especially negative ones. Therefore, micromotor recognition (MAR) plays a crucial role in emotion recognition and psychological assessment. However, due to their small amplitude and subtle interclass differences, micromotors are highly susceptible to interference from other movements (especially hand movements), making micromotor recognition a very challenging task.

[0003] Micro-motion recognition methods can be broadly categorized into two types: RGB video-based methods and skeleton-based methods. RGB video-based methods typically use convolutional neural networks (CNNs) or Transformer-based architectures to directly model fine-grained motion patterns from visual observations, demonstrating powerful capabilities in this task. However, their reliance on raw appearance information (such as face and clothing) inevitably raises privacy concerns, limiting their deployment in privacy-sensitive applications such as mental health monitoring. In contrast, skeleton-based techniques, due to their efficiency and privacy-preserving characteristics, have become an important research direction for identity-agnostic micro-motion recognition. These methods use only the coordinates of human skeletal joints to abstract human motion into structured joint representations and perform spatiotemporal feature learning based on graph convolutional networks (GCNs) or PoseConv3D architectures. This paradigm not only exhibits good recognition accuracy but also successfully avoids the privacy risks associated with facial and biometric information, thus achieving a balance between performance and privacy security.

[0004] Nevertheless, current skeleton-based methods still have inherent limitations in recognizing local micro-motions. GCN-based methods are limited in robustness and scalability, while PoseC3D-based methods are constrained by the local receptive field mechanism of convolution and cannot model complex relationships between long-distance joints. Furthermore, micro-motions are highly susceptible to interference from hand movements; methods employing a uniform holistic modeling strategy often mistake irrelevant hand movements for the micro-motions to be recognized, leading to erroneous results. Therefore, how to better capture the fine-grained features between skeletons in micro-motions and how to effectively suppress hand interference remain significant challenges in the field of micro-motion recognition. Summary of the Invention

[0005] The problem addressed by this invention is how to better capture the fine-grained features between micro-movement skeletons and how to effectively suppress hand interference.

[0006] To address the aforementioned problems, this invention provides a micro-motion recognition method, recognition system, electronic device, and storage medium based on a mask autoencoder and a hybrid expert model.

[0007] In a first aspect, the present invention provides a micro-action recognition method based on a masked autoencoder and a hybrid expert model, comprising: Data preprocessing: The acquired skeletal keypoint data is encoded into a multidimensional heatmap; Feature extraction: The multidimensional heatmap is input into a pre-trained heatmap mask autoencoder module, and fine-grained action features are extracted by the encoder of the mask autoencoder module; Hand-dependent gating: The fine-grained motion features are input into a hand-dependent hybrid expert module. Through a hand-aware gating strategy, a first gating weight and a second gating weight are generated based on the dependency relationship between the motion category and hand movement. Expert processing: The multidimensional heatmap is processed with features related to and unrelated to the hand, and then input into the first expert model and the second expert model respectively to obtain the first expert output and the second expert output; Predictive fusion: Based on the first gating weight and the second gating weight, the first expert output and the second expert output are weighted and fused to obtain the expert fusion result. Based on the expert fusion result and the fine-grained action features, the final micro-action recognition result is generated.

[0008] Optionally, the data preprocessing step includes: The skeletal key point data is uniformly sampled to a predetermined number of frames; The sampled skeletal keypoint data is encoded into a format of size [size missing]. heatmap, in which Indicates the number of joints. Indicates the number of frames. and This indicates the height and width of each heatmap frame; Among them, for the first Each joint, and its corresponding joint thermogram Generated based on a Gaussian distribution centered on the joint coordinates; for connecting joints and The Each limb, and its corresponding limb heat map Based on the points on the heatmap and the connecting joints and The distance of the line segment is generated.

[0009] Optionally, the mask autoencoder module is pre-trained using an asymmetric encoder-decoder architecture; During the pre-training phase, a channel masking strategy is applied to the input multidimensional heatmap, randomly selecting some channels for masking. The masked heatmap is then input into the encoder, and the decoder reconstructs the masked channels. The encoder is retained after pre-training and used to extract the fine-grained action features.

[0010] Optionally, the channel masking strategy specifically involves randomly selecting several channels in the channel dimension of the multidimensional heatmap and setting all element values ​​within the selected channels to zero.

[0011] Optionally, during the pre-training phase, a weighted mean square error loss function is used to constrain the reconstruction result of the decoder, wherein the loss function is: ; in Indicates the actual value. Indicates the predicted value. Indicates the weighting factor. The total number of elements contained in the masked channel.

[0012] Optionally, the feature extraction further includes: The output of the encoder is fed into a linear classification layer, and the encoder and the linear classification layer are subjected to supervised fine-tuning using labeled data. The linear classification layer outputs the fine-grained action features, which are fine-grained category probability distributions. .

[0013] Optionally, the hand-sensing gating strategy includes: Based on prior knowledge, the fine-grained action categories to be identified are divided into two groups: those related to the hand and those unrelated to the hand. Based on the fine-grained category probability distribution The sum of probabilities belonging to all categories in the hand-related group is calculated and used as the first gating weight for hand-related groups. And calculate the sum of probabilities of all categories belonging to the hand-independent group, as the second hand-independent gating weight. .

[0014] Optionally, the expert processing includes: The first expert model is used to handle hand-related micro-movements, and the second expert model is used to handle hand-independent micro-movements. Before inputting the multidimensional heatmap into the first expert model, the channels corresponding to hand joints in the multidimensional heatmap are enhanced. Before inputting the multidimensional heatmap into the second expert model, the channels corresponding to hand joints in the multidimensional heatmap are suppressed.

[0015] Optionally, the prediction fusion includes: Multiply the first expert output by the first gating weight, multiply the second expert output by the second gating weight, and then concatenate the two results to obtain the expert fusion result. ; The expert fusion results The micro-motion recognition result is generated by fusing the fine-grained motion features.

[0016] Secondly, embodiments of the present invention provide a micro-action recognition system based on a mask autoencoder and a hybrid expert model, comprising: The data preprocessing module is used to encode the acquired skeletal keypoint data into a multidimensional heatmap; The feature extraction module includes a pre-trained masked autoencoder for extracting fine-grained action features from the multidimensional heatmap. The hybrid expert module includes a hand-sensing gating unit, a first expert model, and a second expert model; The hand-sensing gating unit is used to generate a first gating weight and a second gating weight based on the dependency relationship between the fine-grained motion features and hand movements. The first expert model is used to process the multidimensional heatmap after hand feature enhancement and output the first expert result; The second expert model is used to process the multidimensional heatmap after hand feature suppression and output the second expert result; and, The prediction fusion module is used to perform weighted fusion of the first expert result and the second expert result according to the first gating weight and the second gating weight, and generate the final micro-action recognition result based on the fusion result and the fine-grained action features.

[0017] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.

[0018] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the micro-motion recognition method based on a mask autoencoder and a hybrid expert model as described in the first aspect.

[0019] The beneficial effects of the micro-action recognition method based on mask autoencoder and hybrid expert model of the present invention are as follows: This invention, through a heatmap mask autoencoder module and employing the proposed channel masking strategy and weighted mean square loss, enhances the model's perception and capture of micro-motion skeletal features, extracting more fine-grained and generalized motion feature representations. Furthermore, by introducing a hand-dependent hybrid expert module, micro-motions are classified into different sets based on their hand dependence, and more refined recognition is achieved through collaboration among expert models. This effectively mitigates "hand interference" in micro-motion recognition, improving accuracy and robustness. Attached Figure Description

[0020] Figure 1 This is a flowchart of the micro-action recognition method based on mask autoencoder and hybrid expert model in an embodiment of the present invention; Figure 2 This is a model diagram of the micro-action recognition method based on mask autoencoder and hybrid expert model in an embodiment of the present invention. Figure 3 This is a comparison diagram of the channel masking strategy with the frame masking strategy, random masking strategy and pipeline masking strategy in the embodiments of the present invention; Figure 4 This is a structural block diagram of the micro-motion recognition system based on a mask autoencoder and a hybrid expert model in an embodiment of the present invention; Figure 5 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation

[0021] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0023] like Figures 1 to 2 As shown in the figure, a micro-action recognition method based on a mask autoencoder and a hybrid expert model provided in this embodiment of the invention includes: S1. Data preprocessing: Encode the acquired skeletal key point data into a multidimensional heatmap; Data preprocessing steps include: Sampling of skeletal keypoint data to a predetermined number of frames; The sampled skeletal keypoint data is encoded into a format of size [size missing]. heatmap, in which Indicates the number of joints. Indicates the number of frames. and This indicates the height and width of each heatmap frame; Among them, for the first Each joint, and its corresponding joint thermogram Generated based on a Gaussian distribution centered on the joint coordinates; for connecting joints and The Each limb, and its corresponding limb heat map Based on the points on the heatmap and the connecting joints and The distance of the line segment is generated.

[0024] For example, for RGB video input, a human skeleton keypoint detection algorithm (such as HRNet) is used to extract 2D keypoints, resulting in a coordinate triplet (x, y, c) for each skeleton keypoint, where (x, y) represents the keypoint coordinates and c represents the keypoint confidence. For 3D skeleton keypoint input, only the x and y axis coordinates are retained, and c is set to 1.

[0025] Downsample each micro-motion sample uniformly to The frames are then subjected to data augmentation (including cropping, scaling, and horizontal flipping). The augmented 2D skeletal keypoints are encoded into frames of size [size missing]. The heatmap, and the height and width of each frame. For the first For each skeletal joint, a joint heatmap is obtained by combining a Gaussian mapping centered on each node. For the first Draw a body heat map for each limb. The formula is as follows: ; ; In this embodiment, Controlling the variance of the Gaussian mapping, , Represents the coordinates of all elements within the region. Indicates the first A set of ternary coordinates for each skeletal joint. and Indicates the first The two end joints of a limb Indicates the first The connection between limbs, the function Calculation points to line segment distance, They represent the first Confidence level of the joints at both ends of a limb.

[0026] S2, Feature Extraction: Input the multidimensional heatmap into the pre-trained heatmap mask autoencoder module, and extract fine-grained action features through the encoder of the mask autoencoder module; The mask autoencoder module is pre-trained using an asymmetric encoder-decoder architecture; During the pre-training phase, a channel masking strategy is applied to the input multidimensional heatmap, randomly selecting some channels for masking. The masked heatmap is then input into the encoder, and the decoder reconstructs the masked channels. The encoder is retained after pre-training and used to extract fine-grained action features.

[0027] Furthermore, the channel masking strategy is as follows: on the channel dimension of the multidimensional heatmap, several channels are randomly selected, and all element values ​​in the selected channels are set to zero.

[0028] Specifically, the heatmap mask autoencoder module in this invention employs an asymmetric encoder-decoder architecture with Vision Transformer (ViT) as its backbone to process heatmaps. During pre-training, the model performs mask reconstruction on unlabeled data to learn the spatiotemporal representation of the skeleton heatmap. Specifically, given a heatmap input... First, a joint spatiotemporal cube embedding ( (3D convolution) reshapes it into Each dimension is The token is denoted as ,in , This represents the number of output channels for the 3D convolution. Then, for... Add location encoding.

[0029] Furthermore, this invention proposes a channel masking strategy to mask the heatmap during the pre-training phase. For example... Figure 3 As shown, unlike existing masking strategies (such as random masking, tubular masking, and frame masking), several heatmap channels are randomly selected and set to zero. The masked heatmap is then input into the encoder. The decoder predicts the original values ​​of the masked channels; that is, only the masked portions participate in the calculation of the decoder's loss function. To encourage the decoder to focus on information-rich regions, a weighted mean squared error loss is used, the formula of which is as follows: ; in, The total number of elements contained in the masked channel. Indicates the actual value. Indicates the predicted value. This represents the weighting factor, which is set to 10 in this embodiment.

[0030] Feature extraction further includes: The encoder's output is fed into a linear classification layer, and supervised fine-tuning of both the encoder and the linear classification layer is performed using labeled data. The linear classification layer outputs fine-grained action features, which are fine-grained class probability distributions. .

[0031] Specifically, in the fine-tuning stage, only the structure and weights of the encoder are retained, and a linear classification layer is connected to the back end of the encoder output to perform supervised fine-tuning using cross-entropy loss.

[0032] S3, Hand-dependent gating: Fine-grained motion features are input into a hand-dependent hybrid expert module. Through a hand-aware gating strategy, a first gating weight and a second gating weight are generated based on the dependency relationship between the motion category and hand movement. Hand-sensing gating strategies include: Based on prior knowledge, the fine-grained action categories to be identified are divided into two groups: those related to the hand and those unrelated to the hand. Based on the probability distribution of fine-grained categories The sum of probabilities belonging to all categories in the hand-related group is calculated and used as the first gating weight for hand-related groups. And calculate the sum of probabilities of all categories belonging to the hand-independent group, as the second hand-independent gating weight. .

[0033] S4. Expert Processing: After performing feature processing on the multidimensional heatmap that is related to the hand and that is not related to the hand, the corresponding inputs are fed into the first expert model and the second expert model to obtain the first expert output and the second expert output. Expert handling includes: The first expert model is used to handle hand-related micro-movements, and the second expert model is used to handle hand-independent micro-movements. Before inputting the multidimensional heatmap into the first expert model, the channels corresponding to the hand joints in the multidimensional heatmap are enhanced. Before inputting the multidimensional heatmap into the second expert model, the channels in the multidimensional heatmap corresponding to the hand joints are suppressed.

[0034] Specifically, another module of the present invention is a hand-dependent hybrid expert module, which mainly includes a hand-aware gating strategy and an expert group module. The hand-aware gating strategy, based on statistical analysis of the dataset, categorizes fine-grained actions into hand-related and hand-independent categories. Subsequently, the generated fine-grained category distribution... Based on this, the probabilities of fine-grained categories within each group are aggregated to calculate the hand-related probabilities. : ; in, Map each fine-grained category to its hand dependency group (i.e. whether it depends on hand movements). Represents all functions Mapping to group Fine-grained category index The set, Indicates fine-grained action categories The original predicted probability, Indicates the first Aggregate probability (gating weight) of each hand-dependent group.

[0035] The samples are then fed into an expert group consisting of two expert models with the same structure as the encoder: one for hand-related micro-movements and the other for hand-independent micro-movements. Hand features are explicitly enhanced by amplifying the values ​​of the hand-related channels before being fed into the experts. Conversely, for hand-independent samples… Before expert reasoning, potential interference from irrelevant hand movements is suppressed by setting the hand-related heatmap channels to zero, as shown in the following formula: , ; in, Indicates the first One channel, It is a set of channels corresponding to the joints of the hand. and Let the first and second hands be the first and third hands respectively, representing the first and second hands of the hand-related samples. aisle, and The number of the processed sample One channel. Indicates a feature magnification operation; This indicates a feature clearing operation.

[0036] S5. Predictive Fusion: Based on the first gating weight and the second gating weight, the first expert output and the second expert output are weighted and fused to obtain the expert fusion result. Based on the expert fusion result and fine-grained action features, the final micro-action recognition result is generated.

[0037] Predictive fusion includes: Multiply the first expert's output by the first gating weight, multiply the second expert's output by the second gating weight, and then concatenate the two results to obtain the expert fusion result. ; fusion of expert results The data is then fused with fine-grained motion features to generate the final micro-motion recognition result.

[0038] Specifically, based on S4, the output of the expert model can be obtained by weighted concatenation, as shown in the following formula: ; in, and These represent the first expert model and the second expert model, respectively. and These are the first and second gating weights at the category level, obtained from the hand-sensing gating module. and Indicates the input sample. This indicates a splicing operation. Finally, the expert group's output will be... The fine-grained motion features fused with the output of step S4 are used to obtain the final prediction, further improving the overall performance and robustness of MAR.

[0039] The following table shows the recognition results using the public datasets MA-52, iMiGUE, and SMG as examples: Table 1

[0040] As can be seen from the technical solution provided by the present invention, by using the heatmap mask autoencoder module and the channel masking strategy and weighted mean square loss proposed in this invention, the model's perception and capture of micro-motion skeletal features can be improved, and more fine-grained and generalized motion feature representations can be extracted. Furthermore, by introducing a hand-dependent hybrid expert module, micro-motions are classified into different sets according to their dependence on the hand, and more refined recognition is achieved through collaboration between expert models. This effectively alleviates "hand interference" in micro-motion recognition, improving accuracy and robustness. Compared to the baseline ST-GCN, the present invention achieves significant improvements in the MA-52, iMiGUE, and SMG datasets, with Top-1 accuracy improvements of 2.98%, 8.87%, and 9.51%, respectively, and Top-5 accuracy improvements of 4.51%, 4.23%, and 0.16%, demonstrating the superiority of the present invention.

[0041] As can be seen from the technical solution provided by this invention, by using the heatmap mask autoencoder module and the channel masking strategy and weighted mean square loss proposed in this invention, the model's perception and capture of micro-motion skeletal features can be improved, and more fine-grained and generalized motion feature representations can be extracted. Furthermore, by introducing a hand-dependent hybrid expert module, micro-motions are classified into different sets according to their dependence on the hand, and more refined recognition is achieved through collaboration between expert models. This effectively alleviates "hand interference" in micro-motion recognition, improving accuracy and robustness.

[0042] This invention also provides a micro-motion recognition system based on a mask autoencoder and a hybrid expert model. This system is used to implement the above-described method embodiments, and details already described will not be repeated. The terms "module," "unit," and "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.

[0043] like Figure 4 As shown, Figure 4This is a block diagram of the micro-action recognition system based on mask autoencoder and hybrid expert model in this invention. The system includes: The data preprocessing module 101 is used to encode the acquired skeletal key point data into a multidimensional heatmap; The feature extraction module 102 includes a pre-trained masked autoencoder for extracting fine-grained action features from the multidimensional heatmap. The hybrid expert module 103 includes a hand-sensing gating unit, a first expert model, and a second expert model; The hand-sensing gating unit is used to generate a first gating weight and a second gating weight based on the dependency relationship between the fine-grained motion features and hand movements. The first expert model is used to process the multidimensional heatmap after hand feature enhancement and output the first expert result; The second expert model is used to process the multidimensional heatmap after hand feature suppression and output the second expert result; and, The prediction fusion module 104 is used to perform weighted fusion of the first expert result and the second expert result according to the first gating weight and the second gating weight, and generate the final micro-action recognition result based on the fusion result and the fine-grained action features.

[0044] like Figure 5 As shown in the figure, an electronic device provided by an embodiment of the present invention includes: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following method: Data preprocessing: The acquired skeletal keypoint data is encoded into a multidimensional heatmap; Feature extraction: The multidimensional heatmap is input into a pre-trained heatmap mask autoencoder module, and fine-grained action features are extracted by the encoder of the mask autoencoder module; Hand-dependent gating: The fine-grained motion features are input into a hand-dependent hybrid expert module. Through a hand-aware gating strategy, a first gating weight and a second gating weight are generated based on the dependency relationship between the motion category and hand movement. Expert processing: The multidimensional heatmap is processed with features related to and unrelated to the hand, and then input into the first expert model and the second expert model respectively to obtain the first expert output and the second expert output; Predictive fusion: Based on the first gating weight and the second gating weight, the first expert output and the second expert output are weighted and fused to obtain the expert fusion result. Based on the expert fusion result and the fine-grained action features, the final micro-action recognition result is generated.

[0045] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0046] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0047] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A micro-motion recognition method based on masked autoencoder and hybrid expert model, characterized in that, include: Data preprocessing: The acquired skeletal keypoint data is encoded into a multidimensional heatmap; Feature extraction: The multidimensional heatmap is input into a pre-trained heatmap mask autoencoder module, and fine-grained action features are extracted by the encoder of the mask autoencoder module; Hand-dependent gating: The fine-grained motion features are input into a hand-dependent hybrid expert module. Through a hand-aware gating strategy, a first gating weight and a second gating weight are generated based on the dependency relationship between the motion category and hand movement. Expert processing: The multidimensional heatmap is processed with features related to and unrelated to the hand, and then input into the first expert model and the second expert model respectively to obtain the first expert output and the second expert output; Predictive fusion: Based on the first gating weight and the second gating weight, the first expert output and the second expert output are weighted and fused to obtain the expert fusion result. Based on the expert fusion result and the fine-grained action features, the final micro-action recognition result is generated.

2. The method according to claim 1, characterized in that, The data preprocessing includes: The skeletal key point data is uniformly sampled to a predetermined number of frames; The sampled skeletal keypoint data is encoded into a format of size [size missing]. heatmap, in which Indicates the number of joints. Indicates the number of frames. and This indicates the height and width of each heatmap frame; Among them, for the first Each joint, and its corresponding joint thermogram Generated based on a Gaussian distribution centered on the joint coordinates; for connecting joints and The Each limb, and its corresponding limb heat map Based on the points on the heatmap and the connecting joints and The distance of the line segment is generated.

3. The method according to claim 1, characterized in that, The mask autoencoder module is pre-trained using an asymmetric encoder-decoder architecture; During the pre-training phase, a channel masking strategy is applied to the input multidimensional heatmap, randomly selecting some channels for masking. The masked heatmap is then input into the encoder, and the decoder reconstructs the masked channels. The encoder is retained after pre-training and used to extract the fine-grained action features.

4. The method according to claim 3, characterized in that, The channel masking strategy is as follows: on the channel dimension of the multidimensional heatmap, several channels are randomly selected, and all element values ​​in the selected channels are set to zero.

5. The method according to claim 3, characterized in that, In the pre-training phase, a weighted mean square error loss function is used to constrain the reconstruction results of the decoder. The loss function is: ； in Indicates the actual value. Indicates the predicted value. Indicates the weighting factor. The total number of elements contained in the masked channel.

6. The method according to claim 1, characterized in that, The feature extraction further includes: The output of the encoder is fed into a linear classification layer, and the encoder and the linear classification layer are fine-tuned in a supervised manner using labeled data. The linear classification layer outputs the fine-grained action features, which are fine-grained category probability distributions.

7. The method according to claim 6, characterized in that, The hand-sensing gating strategy includes: Based on prior knowledge, the fine-grained action categories to be identified are divided into two groups: those related to the hand and those unrelated to the hand. Based on the fine-grained category probability distribution, the sum of probabilities belonging to all categories in the hand-related group is calculated as the first gating weight for hand-related categories, and the sum of probabilities belonging to all categories in the hand-independent group is calculated as the second gating weight for hand-independent categories.

8. The method according to claim 1, characterized in that, The expert processing includes: The first expert model is used to handle hand-related micro-movements, and the second expert model is used to handle hand-independent micro-movements. Before inputting the multidimensional heatmap into the first expert model, the channels corresponding to hand joints in the multidimensional heatmap are enhanced. Before inputting the multidimensional heatmap into the second expert model, the channels corresponding to hand joints in the multidimensional heatmap are suppressed.

9. The method according to claim 1, characterized in that, The predictive fusion includes: Multiply the first expert output by the first gating weight, multiply the second expert output by the second gating weight, and concatenate the two results to obtain the expert fusion result; The expert fusion result is fused with the fine-grained action features to generate the final micro-action recognition result.

10. A micro-motion recognition system based on a mask autoencoder and a hybrid expert model, characterized in that, include: The data preprocessing module is used to encode the acquired skeletal keypoint data into a multidimensional heatmap; The feature extraction module includes a pre-trained masked autoencoder for extracting fine-grained action features from the multidimensional heatmap. The hybrid expert module includes a hand-sensing gating unit, a first expert model, and a second expert model; The hand-sensing gating unit is used to generate a first gating weight and a second gating weight based on the dependency relationship between the fine-grained motion features and hand movements. The first expert model is used to process the multidimensional heatmap after hand feature enhancement and output the first expert result; The second expert model is used to process the multidimensional heatmap after hand feature suppression and output the second expert result; and, The prediction fusion module is used to perform weighted fusion of the first expert result and the second expert result according to the first gating weight and the second gating weight, and generate the final micro-action recognition result based on the fusion result and the fine-grained action features.

Images