Multi-modal human body action recognition method, system and equipment based on gating fusion and medium

Abstract

The invention belongs to the technical field of action recognition, and discloses a multi-modal human body action recognition method, system and device based on gating fusion and a medium, and the method comprises the steps: obtaining multi-modal sensing data, carrying out the filtering processing and feature extraction of the multi-modal sensing data, obtaining the preprocessing data corresponding to each modal, and carrying out the feature extraction of the preprocessing data; performing time sequence alignment on the preprocessed data of each modal to obtain a multi-modal sample; inputting the multi-modal sample into a human body action recognition model for classification prediction to obtain an action recognition result; wherein the human body action recognition model is constructed based on a time sequence Transform network, and comprises a multi-modal coding module, a gating fusion module and a classification head which are connected in sequence. According to the method, the problems that a single mode is easily blocked and interfered by noise and a traditional multi-mode fusion strategy is simple and extensive are solved, and high-precision and high-robustness human body action recognition in a complex scene is realized.

Description

Technical Field

[0001] This invention belongs to the field of motion recognition technology, and in particular relates to a multimodal human motion recognition method, system, device and medium based on gating fusion. Background Technology

[0002] Human motion recognition is a core foundational technology in ubiquitous computing, intelligent sensing, and human-computer interaction, with wide-ranging applications in smart healthcare, security monitoring, industrial inspection, and emergency rescue. Existing technologies primarily rely on single or a few modalities for motion perception, such as vision-based pose estimation, motion analysis based on inertial measurement units (IMUs), audio-based event detection, or contactless sensing based on Wi-Fi channel state information (CSI). However, in real-world open environments, single-modality approaches face numerous inherent limitations: visual modalities are susceptible to changes in lighting, viewpoint occlusion, and privacy constraints; IMU modalities depend on device wear and are sensitive to individual differences; audio modalities are easily affected by environmental noise; and Wi-Fi CSI modalities are significantly affected by multipath effects and device layout, making it difficult to meet the requirements of highly reliable applications in terms of recognition accuracy and stability.

[0003] To improve recognition robustness, multimodal fusion technology has emerged. Existing fusion methods often employ feature splicing, decision-level voting, or simple attention weighting mechanisms, lacking explicit and refined modeling of the temporal dynamics, signal-to-noise ratio differences, and complementary or redundant relationships between different modal signals. Especially in extreme cases where some modalities fail (e.g., camera obstruction, IMU not being worn, excessive environmental noise), traditional fusion strategies often experience a sharp performance decline, failing to achieve adaptive and robust multi-source information integration. Therefore, a novel recognition framework is urgently needed that can deeply fuse heterogeneous temporal modalities and possess the ability to dynamically evaluate modal reliability and adaptively weight them, in order to overcome the performance bottleneck of action recognition in complex scenarios and meet the practical application requirements of highly reliable intelligent interaction. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal human motion recognition method, system, device, and medium based on gating fusion to solve the problems existing in the prior art.

[0005] To achieve the above objectives, this invention provides a multimodal human action recognition method based on gated fusion, comprising: Step S1: Acquire synchronous timing data collected by a multimodal sensor array consisting of a visual skeleton, Wi-Fi channel status information, ambient audio, and a wristband IMU; Step S2: Perform targeted preprocessing and time-series alignment on the multimodal time-series data to construct unified multimodal samples; Step S3: Input the multimodal samples into a multimodal temporal Transformer network, which includes modules such as a modality-specific encoder and a gating fusion module; the modality-specific encoder extracts temporal features from the skeleton, audio, IMU, and Wi-Fi CSI modalities respectively; the gating fusion module adaptively weights and fuses the features of each modality using learnable gating vectors. Step S4: Based on the fused features, combined with the modality-level Dropout training strategy, output the human action recognition results through the classification head.

[0006] Optionally, in step S2, the targeted preprocessing includes: The human skeletal data was denoised using Discrete Wavelet Transform (DWT), and the three-dimensional coordinate sequences of 33 key joints were extracted. : ; in Indicates the time step index; This represents the joint index, with values ​​ranging from 1 to 33. Indicates the first The time step, the first Each joint in the three-dimensional coordinate system The coordinate values ​​of the axis; Indicates the first The time step, the first Each joint in the three-dimensional coordinate system The coordinate values ​​of the axis; Indicates the first The time step, the first Each joint in the three-dimensional coordinate system The coordinate values ​​of the axis; Wiener filtering was used to denoise the audio data, and the root mean square energy, MFCC-0~4, spectral centroid and spectral bandwidth features were extracted. The IMU inertial data (accelerometer, gyroscope, geomagnetism) was denoised using DWT and normalized to form a time-series feature sequence; Multi-channel interpolation and dynamic weighted fusion were used to extract 128-dimensional subcarrier amplitude sequences from Wi-Fi CSI data.

[0007] Optionally, in step S3, the modal encoder includes: The skeletal encoder, consisting of linear projection, position encoding, multi-layer Transformer encoder, and attention pooling, outputs a global representation of the skeleton. ; The IMU and audio modal encoder consist of linear projection, a single-layer Transformer encoder, and adaptive average pooling, outputting a global representation. ( (For IMU / audio); The Wi-Fi CSI modal encoder consists of a cascaded one-dimensional convolution and a lightweight Transformer, outputting a global representation. .

[0008] Optionally, in step S3, the gating fusion module includes: The global features output by each modal encoder By mapping to a unified feature space through linear transformation, a unified dimensional representation is obtained. The mapping relationship satisfies: ; in, For modal index, the value is { bone, audio, imu, wifi}, corresponding to the skeletal, audio, IMU, and Wi-Fi CSI modalities, respectively; For the first The modality linear mapping weight matrix, with dimension 1 ( To unify feature dimensions, For the first Modal original feature dimension).

[0009] Uniform dimensional representation for each modality Gated vectors are learned through fully connected layers and the sigmoid activation function. It is used to adaptively characterize modal reliability, and the calculation method is as follows: ; in, It is a Sigmoid activation function with an output range of [0,1]. For the first The learning weight matrix of the modal gating vector has dimensions of ; For the first The modal gating vector has a dimension of A larger value indicates a higher reliability and contribution priority for the corresponding mode.

[0010] The gate vector is gated by element-wise multiplication. With unified dimensional representation Weighted fusion is performed to obtain the gated weighted modal features. The calculation method is as follows: ; in, This represents an element-wise multiplication operation, which enables the gating vector to adjust the dimensional weights of modal features and suppress interference from low-reliability modes.

[0011] Gated weighted features of all modalities The features are concatenated along the feature dimension and then fed into a multilayer perceptron (MLP) for nonlinear integration to obtain a unified multimodal representation. The specific process is as follows: ; ; in, This is a feature dimension concatenation operation. The concatenated feature vector has a dimension of . (B represents the batch size); and These are the weight matrix and bias vector of the multilayer perceptron, with dimensions of respectively. and ReLU is a non-linear activation function. For layer normalization operation; For the final multimodal representation, the dimension is It is used to input the classification head to complete action recognition.

[0012] Optionally, in step S4, a modality-level dropout strategy is introduced during the model training phase to randomly block the input of one or more modalities with a preset probability, so as to improve the robustness of the model in the case of modality loss or failure.

[0013] On the other hand, to achieve the above objectives, the present invention provides a multimodal human motion recognition system based on gated fusion, comprising: The data acquisition module is used to acquire multimodal sensing data, including visual skeleton data, Wi-Fi CSI data, ambient audio, and wristband IMU data. The data preprocessing module is used to filter and extract features from the multimodal sensing data to obtain preprocessed data corresponding to each modality, and to perform time-series alignment on the preprocessed data of each modality to obtain multimodal samples. The action recognition module is used to input the multimodal samples into the human action recognition model for classification and prediction to obtain action recognition results. The human action recognition model is built based on a temporal Transformer network and includes a multimodal encoding module, a gating fusion module and a classification head connected in sequence. The multimodal encoding module includes encoders corresponding to each modality, and the encoders corresponding to each modality are set in parallel.

[0014] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes the aforementioned multimodal human motion recognition method based on gating fusion when executed by the processor.

[0015] The present invention also provides a computer-readable storage medium storing a computer program, the computer program being executed by a processor of the described multimodal human motion recognition method based on gating fusion.

[0016] The technical effects of this invention are as follows: 1. Multimodal deep fusion and adaptive weighting: Through a gated fusion module, the model can adaptively adjust the contribution of each modality according to the real-time quality of the input signal and its relevance to the task. When the modality is reliable, its role is enhanced, and when the modality is disturbed, its influence is suppressed, thus achieving true robust fusion.

[0017] 2. Strong temporal modeling capability: This invention uses Transformer as the basic encoder for each modality, which effectively captures the long-term temporal dependence and dynamic evolution pattern of human actions, and improves the representation capability of features.

[0018] 3. Strong robustness to modal loss: This invention explicitly simulates sensor failure scenarios through a modal-level Dropout training strategy, enabling the model to still output stably even when some modalities are missing.

[0019] 4. Edge Deployment Friendly: The model structure design of this invention takes into account both performance and efficiency, and can be accelerated and optimized through tools such as TensorRT. It is suitable for deployment on edge devices with limited computing power (such as the Jetson platform) and meets real-time requirements. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is an overall flowchart of the human motion recognition method in this embodiment of the invention; Figure 2This is a schematic diagram of the core architecture of the multimodal temporal Transformer network (RGMT) in an embodiment of the present invention; Figure 3 This is a flowchart of the gated fusion module in an embodiment of the present invention; Figure 4 This is a multi-class action recognition confusion matrix obtained by evaluating on a self-built test set in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the implementation of an embodiment of the present invention. Detailed Implementation

[0022] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0023] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0024] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0025] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] Example 1 like Figure 1 - Figure 5As shown, to address the aforementioned problems in the existing technology, this embodiment provides a multimodal human motion recognition method based on gated fusion. This embodiment constructs a four-modal sensor array including skeletal, audio, IMU (accelerometer, gyroscope, geomagnetic sensor), and Wi-Fi CSI sensors, and designs a multimodal temporal Transformer network (RobustGated Multi-modal Transformer, RGMT) to achieve efficient encoding and adaptive fusion of heterogeneous temporal signals. The method of this embodiment specifically includes: Step S1: Acquire synchronous timing data collected by a multimodal sensor array consisting of a visual skeleton, Wi-Fi channel status information, ambient audio, and a wristband IMU; Step S2: Perform targeted preprocessing and time-series alignment on the multimodal time-series data to construct unified multimodal samples; Step S3: Input the multimodal samples into a multimodal temporal Transformer network, which includes modules such as a modality-specific encoder and a gating fusion module; the modality-specific encoder extracts temporal features from the skeleton, audio, IMU, and Wi-Fi CSI modalities respectively; the gating fusion module adaptively weights and fuses the features of each modality using learnable gating vectors. Step S4: Based on the fused features and combined with the modality-level Dropout training strategy, output the human action recognition results through the classification head; The technical solution of this embodiment solves the problems of single-modal susceptibility to occlusion and noise interference, as well as the simplistic and crude nature of traditional multimodal fusion strategies. It achieves high-precision and robust human motion recognition in complex scenarios and can be widely applied to various application scenarios such as robot interaction, intelligent security, intelligent wearable devices, medical auxiliary monitoring, and drone motion recognition.

[0028] As one implementation of this embodiment, in step S1, a synchronous trigger signal or a high-precision timestamp is used to ensure that the data from the camera (extracting skeleton), Wi-Fi receiver array (acquiring CSI), smartphone (acquiring audio), and smart bracelet (acquiring IMU) are aligned in time. Since the original data sampling frequencies differ, they need to be unified to the same time reference through resampling or interpolation.

[0029] As one implementation of this embodiment, step S2 includes the following steps: S2-1: Perform discrete wavelet transform denoising on the coordinates of key points of the skeleton; perform Wiener filtering denoising on the audio signal and extract features such as MFCC; perform discrete wavelet transform denoising on the IMU three-axis data; perform interpolation completion and adaptive weighted fusion on multi-channel Wi-Fi CSI data.

[0030] S2-2: Using the timeline of the skeletal sequence as a reference, other modal data are aligned to each skeletal frame through nearest neighbor matching to form multivariate temporal samples with consistent time length. : ; in, Represents temporal subsamples of skeletal modalities; Represents audio modal temporal sub-samples; Represents IMU modal temporal sub-samples; This represents a timing subsample of the Wi-Fi CSI mode.

[0031] As one implementation of this embodiment, step S3 includes the following steps: S3-1: Design the dedicated encoders for each mode as follows: Skeletal modal encoder: Input sequence First, the linear layer and the layer normalization mapping are used to... After adding position encoding, the input is a multi-layer, multi-head self-attention Transformer encoder. Temporal modeling is performed, and finally, a global representation is obtained through an attention pooling layer. The calculation formula is: ; ; in, Indicates the first Skeletal features of the time step For learnable weight vectors, For the first Attention weights for time steps This represents the global characterization of the skeletal modality.

[0032] Audio and IMU mode encoders: Input sequences for audio, acceleration, gyroscope, and geomagnetic modes. After light linear projection, it passes through a single-layer Transformer encoder. Capture temporal dependencies, and then obtain a global representation through time-dimensional average pooling. .

[0033] Wi-Fi CSI Modal Encoder: Input Sequence ( (This refers to the number of subcarriers). First, local time-subcarrier patterns are extracted using a one-dimensional convolutional layer. Then, the output is reconstructed into a time series, input into a lightweight Transformer encoder, and finally, average pooling is used to obtain the global representation. .

[0034] S3-2: Calculate the global features output by each modal encoder. By mapping to a unified feature space through linear transformation, a unified dimensional representation is obtained. The mapping relationship satisfies: ; in, This is the modal index, with values ​​{bone, audio, imu, wifi}, corresponding to the skeletal, audio, IMU, and Wi-Fi CSI modalities, respectively. For the first The modality linear mapping weight matrix, with dimension 1 ( To unify feature dimensions, For the first Modal original feature dimension).

[0035] Uniform dimensional representation for each modality Gated vectors are learned through fully connected layers and the sigmoid activation function. It is used to adaptively characterize modal reliability, and the calculation method is as follows: ; in, It is a Sigmoid activation function with an output range of [0,1]. For the first The learning weight matrix of the modal gating vector has dimensions of ; For the first The modal gating vector has a dimension of A larger value indicates a higher reliability and contribution priority for the corresponding mode.

[0036] The gate vector is gated by element-wise multiplication. With unified dimensional representation Weighted fusion is performed to obtain the gated weighted modal features. The calculation method is as follows: ; in, This represents an element-wise multiplication operation, which enables the gating vector to adjust the dimensional weights of modal features and suppress interference from low-reliability modes.

[0037] Gated weighted features of all modalities The features are concatenated along the feature dimension and then fed into a multilayer perceptron (MLP) for nonlinear integration to obtain a unified multimodal representation. The specific process is as follows: ; ; in, This is a feature dimension concatenation operation. The concatenated feature vector has a dimension of . (B represents the batch size); and These are the weight matrix and bias vector of the multilayer perceptron, with dimensions of respectively. and ReLU is a non-linear activation function. For layer normalization operation; For the final multimodal representation, the dimension is It is used to input the classification head to complete action recognition.

[0038] As one implementation of this embodiment, step S4 includes the following steps: S4-1: Fusion Characterization Input classification head. The classification head consists of two fully connected layers, using ReLU activation and Dropout regularization in between, and finally outputs the logits corresponding to the preset action category (such as standing, walking, etc.), and obtains the predicted probability distribution through the softmax function.

[0039] S4-2: During model training, each training batch is given a probability. The input sequence of a certain modality is randomly set to zero. This strategy forces the model to learn independently of any single modality and automatically adjusts the dependency weights on the remaining available modalities using a gated fusion mechanism, thereby greatly improving the model's robustness in situations such as sensor failure or poor signal quality.

[0040] To verify the effectiveness of the method in this embodiment, tests were conducted on a self-built dataset containing 10 action categories. The resulting confusion matrix is ​​shown below. Figure 4 As shown. The method in this embodiment (RGMT model) achieved a recognition accuracy of nearly 100% for most actions, with very little confusion only for similar movement patterns such as running and walking, turning and jumping. The overall accuracy reached 98.82%, which is significantly better than single-modality and traditional fusion baselines.

[0041] This embodiment constructs an end-to-end multimodal temporal action recognition framework by designing a modal-specific encoder and a gating adaptive fusion mechanism. This framework not only fully leverages the complementary spatiotemporal information of four heterogeneous modalities—skeleton, IMU, audio, and Wi-Fi CSI—but also dynamically responds to changes in modal quality through a gating mechanism and addresses modal loss through a modal dropout strategy. This enables high-precision and robust human action recognition in complex real-world scenarios, providing a reliable technical foundation for applications such as intelligent monitoring and human-computer interaction.

[0042] This embodiment also provides a multimodal human motion recognition system based on gating fusion, including: The data acquisition module is used to acquire multimodal sensing data, including visual skeleton data, Wi-Fi CSI data, ambient audio, and wristband IMU data. The data preprocessing module is used to filter and extract features from the multimodal sensing data to obtain preprocessed data corresponding to each modality, and to perform time-series alignment on the preprocessed data of each modality to obtain multimodal samples. The action recognition module is used to input the multimodal samples into the human action recognition model for classification and prediction to obtain action recognition results. The human action recognition model is built based on a temporal Transformer network and includes a multimodal encoding module, a gating fusion module and a classification head connected in sequence. The multimodal encoding module includes encoders corresponding to each modality, and the encoders corresponding to each modality are set in parallel.

[0043] This embodiment also provides an electronic device, including a memory and a processor. The memory stores a computer program that is executed by the processor. When the computer program is run by the processor, it executes the aforementioned multimodal human motion recognition method based on gating fusion.

[0044] This embodiment also provides a computer-readable storage medium storing a computer program, which is executed by a processor to provide the described multimodal human motion recognition method based on gating fusion.

[0045] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal human motion recognition method based on gating fusion, characterized in that, include: Acquire multimodal sensing data, including visual skeleton data, Wi-Fi CSI data, ambient audio, and wristband IMU data; The multimodal sensing data are filtered and feature extracted to obtain preprocessed data corresponding to each modality. The preprocessed data of each modality are then time-aligned to obtain multimodal samples. The multimodal samples are input into the human action recognition model for classification and prediction to obtain action recognition results. The human action recognition model is built based on a temporal Transformer network and includes a multimodal encoding module, a gating fusion module and a classification head connected in sequence. The multimodal encoding module includes encoders corresponding to each modality, and the encoders corresponding to each modality are set in parallel.

2. The multimodal human motion recognition method based on gating fusion according to claim 1, characterized in that, The process of acquiring the multimodal samples specifically includes: The visual skeleton data is denoised using the discrete wavelet transform algorithm, and the three-dimensional coordinate sequence of key joints is extracted. The Wiener filtering algorithm is used to denoise ambient audio and extract root mean square energy, MFCC features, spectral centroid and spectral bandwidth features. The wristband IMU data is denoised and normalized using the DWT denoising algorithm to obtain a time-series feature sequence. The Wi-Fi CSI data is subjected to multi-channel interpolation and dynamic weighted fusion to extract a 128-dimensional subcarrier amplitude sequence; Based on the three-dimensional coordinate sequence corresponding to the visual skeleton data, the preprocessed modal data are aligned to each skeleton frame using the nearest neighbor matching method to obtain multimodal samples.

3. The multimodal human motion recognition method based on gating fusion according to claim 1, characterized in that, The training process of the human motion recognition model specifically includes: Acquire training data, which includes multimodal sensing training data and corresponding human motion labels; An initial human motion recognition model is constructed, and the training data is input into the initial human motion recognition model for classification and prediction. The initial human motion recognition model is then trained based on a preset target loss function to obtain a trained human motion recognition model. Based on the trained human motion recognition model, a multimodal sample human motion recognition task is performed.

4. The multimodal human motion recognition method based on gating fusion according to claim 1, characterized in that, During the initial training of the human motion recognition model, a modal-level dropout strategy is introduced to simulate sensor failure scenarios. In each training batch, one or more modal inputs are randomly masked with a preset probability.

5. The multimodal human motion recognition method based on gating fusion according to claim 1, characterized in that, The processing procedure of the human motion recognition model specifically includes: Samples from each modality are input into the corresponding encoder for feature extraction to obtain global features for each modality; wherein, the encoder includes an IMU and an audio modality encoder, a skeletal encoder, and a Wi-Fi CSI modality encoder; The global features of each modality are input into the gated fusion module, and the global features of each modality are mapped to a unified feature space through linear transformation to obtain a unified dimensional representation. For a unified dimensional representation of each modality, a gating vector is learned through a fully connected layer and a sigmoid activation function; the gating vector and the unified dimensional representation are then weighted and fused through element-wise multiplication to obtain gated weighted features. The gated weighted features of each modality are concatenated along the feature dimension and input into a multilayer perceptron for nonlinear integration to obtain a unified multimodal representation. The unified multimodal representation is used as the input classification head for classification prediction, and the corresponding action recognition result is output.

6. A multimodal human motion recognition method system based on gating fusion, characterized in that, include: The data acquisition module is used to acquire multimodal sensing data, including visual skeleton data, Wi-Fi CSI data, ambient audio, and wristband IMU data. The data preprocessing module is used to filter and extract features from the multimodal sensing data to obtain preprocessed data corresponding to each modality, and to perform time-series alignment on the preprocessed data of each modality to obtain multimodal samples. The action recognition module is used to input the multimodal samples into the human action recognition model for classification and prediction to obtain action recognition results. The human action recognition model is built based on a temporal Transformer network and includes a multimodal encoding module, a gating fusion module and a classification head connected in sequence. The multimodal encoding module includes encoders corresponding to each modality, and the encoders corresponding to each modality are set in parallel.

7. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program that is executed by the processor, and the computer program, when executed by the processor, performs a multimodal human motion recognition method based on gating fusion as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is executed by a processor as described in any one of claims 1-5: a multimodal human motion recognition method based on gating fusion.

Images