Human motion posture recognition method and system based on multi-modal data fusion

Abstract

The application discloses a human motion posture recognition method and system based on multi-modal data fusion, relates to the technical field of posture recognition, and first synchronously acquires human motion posture video frames and IMU segments; then, features of the two kinds of heterogeneous data are extracted in a branch-parallel manner, and spatial posture information of vision and dynamic inertia features of IMU are captured respectively. Subsequently, the features specific to the modes are uniformly represented through a mixed Tokenization and embedding mechanism, a spatial-inertia collaborative attention fusion mechanism is further introduced to realize dynamic association and deep fusion of cross-modal information, and finally, the multi-modal fusion feature vector obtained through fusion is classified to realize accurate recognition of human motion postures. In this way, the defect that traditional fusion is insufficient in capturing subtle action differences can be overcome, and the accuracy and stability of action recognition in a complex scene are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gesture recognition, and more specifically, to a human motion gesture recognition method and system based on multi-modal data fusion. BACKGROUND

[0002] With the rapid development of artificial intelligence and Internet of Things technology, accurate and robust intelligent decision analysis of human motion gestures has become a core requirement in many frontier fields. For example, in rehabilitation training, doctors need to accurately assess the slight improvement of patients' gait, which requires higher recognition of fine-grained actions. Therefore, the limitations of a single data source make existing human motion analysis schemes difficult to meet the growing demand for fine-grained and intelligent decision-making in terms of accuracy, robustness and generalization ability in the face of the core challenge of fine-grained action recognition. Under this background, by integrating the complementary information of different modalities, multi-source data fusion technology that builds more comprehensive and robust motion representation shows great potential and becomes an important path to solve the above problems.

[0003] However, in the prior art, traditional multi-source data fusion strategies, such as early fusion, late fusion or intermediate layer fusion, each have their own shortcomings. Early fusion usually directly splices the raw data or preliminary features from different sensors, which can lead to high input dimension and difficulty in effectively capturing the complex nonlinear relationship and temporal dependence between different modal data. Especially in the fine-grained recognition scenario, the tiny effective signal may be overwhelmed by a large amount of redundant information or noise, making it difficult for the model to learn key discriminative features. Late fusion separately processes and extracts features from different modal data, and then fuses the results at the decision level, which cannot fully utilize the complementarity between different modal data to distinguish subtle action differences, and its fusion effect is often limited to simple superposition, making it difficult to achieve synergistic gain. In addition, some intermediate layer fusion methods attempt to fuse features at the intermediate layer of the deep learning model, but often use simple splicing, summation or weighted averaging, lacking dynamic perception and adaptive adjustment ability for the internal relevance and importance of different modal data. This makes it difficult for existing schemes to achieve ideal precision and robustness when dealing with complex, subtle or rapidly changing human motion.

[0004] Therefore, there is an urgent need for an optimized human motion gesture recognition method and system based on multi-modal data fusion. SUMMARY

[0005] To solve the above technical problems, the present application is proposed.

[0006] According to an aspect of the present application, a human motion posture recognition method based on multi-modal data fusion is provided, which comprises: acquiring human motion posture video frames and IMU segments aligned with the human motion posture video frames; performing branch-parallel feature extraction on the human motion posture video frames and the IMU segments to obtain a skeletal posture vector and an inertial feature matrix; performing mixed Tokenization and embedding on the skeletal posture vector and the inertial feature matrix to obtain a mixed Token sequence; performing spatial-inertial collaborative attention fusion on the mixed Token sequence to obtain a human motion posture multi-modal fusion representation vector; and inputting the human motion posture multi-modal fusion representation vector into a multi-layer perceptron classification head to obtain a final action label.

[0007] According to another aspect of the present application, a human motion posture recognition system based on multi-modal data fusion is provided, which comprises: a multi-modal data acquisition module for acquiring human motion posture video frames and IMU segments aligned with the human motion posture video frames; a branch-parallel feature extraction module for performing branch-parallel feature extraction on the human motion posture video frames and the IMU segments to obtain a skeletal posture vector and an inertial feature matrix; a mixed Tokenization and embedding module for performing mixed Tokenization and embedding on the skeletal posture vector and the inertial feature matrix to obtain a mixed Token sequence; a multi-modal fusion module for performing spatial-inertial collaborative attention fusion on the mixed Token sequence to obtain a human motion posture multi-modal fusion representation vector; and an action classification module for inputting the human motion posture multi-modal fusion representation vector into a multi-layer perceptron classification head to obtain a final action label.

[0008] Compared with the prior art, the human motion posture recognition method and system based on multi-modal data fusion provided by the present application first synchronously acquire human motion posture video frames and IMU segments, then perform feature extraction on the two kinds of heterogeneous data in a branch-parallel manner to capture spatial posture information and dynamic inertial features of IMU, respectively. Subsequently, the modal-specific features are uniformly represented through a mixed Tokenization and embedding mechanism, and a spatial-inertial collaborative attention fusion mechanism is further introduced to realize dynamic association and deep fusion of cross-modal information. Finally, the multi-modal fusion representation vector obtained through fusion is classified to realize accurate recognition of human motion posture. In this way, the defect of insufficient capture of subtle action differences in traditional fusion can be overcome, and the accuracy and stability of action recognition in complex scenes can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description thereof taken in conjunction with the accompanying drawings, in which: The accompanying drawings provide exemplary embodiments of the application and serve as an aid in understanding the application. They are a part of and constitute procedures for carrying out the application. They do not limit the scope of the application. In the drawings, like reference numerals refer to like elements or steps throughout.

[0010] Figure 1 A flow chart of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application.

[0011] Figure 2 A data flow chart of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application.

[0012] Figure 3 A flow chart of sub-step S1 of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application.

[0013] Figure 4 A flow chart of sub-step S2 of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application.

[0014] Figure 5 A flow chart of sub-step S3 of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application.

[0015] Figure 6 A flow chart of sub-step S4 of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application.

[0016] Figure 7 A block diagram of the human motion posture recognition system based on multi-modal data fusion according to the embodiment of the present application. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art. It should be understood that the drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

[0018] In view of the above background art, the present application proposes a human motion posture recognition method based on multi-modal data fusion. Figure 1 A flow chart of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application. Figure 2A data flow diagram of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the human motion posture recognition method based on multi-modal data fusion comprises the following steps: S1, acquiring a human motion posture video frame and an IMU segment aligned with the human motion posture video frame; S2, performing split parallel feature extraction on the human motion posture video frame and the IMU segment to obtain a skeletal posture vector and an inertial feature matrix; S3, performing mixed Tokenization and embedding on the skeletal posture vector and the inertial feature matrix to obtain a mixed Token sequence; S4, performing spatial-inertial collaborative attention fusion on the mixed Token sequence to obtain a human motion posture multi-modal fusion representation vector; and S5, inputting the human motion posture multi-modal fusion representation vector into a multi-layer perception machine classification head to obtain a final action label. Figure 1 and Figure 2 As shown in FIG. 1, the human motion posture recognition method based on multi-modal data fusion comprises the following steps: S1, acquiring a human motion posture video frame and an IMU segment aligned with the human motion posture video frame; S2, performing split parallel feature extraction on the human motion posture video frame and the IMU segment to obtain a skeletal posture vector and an inertial feature matrix; S3, performing mixed Tokenization and embedding on the skeletal posture vector and the inertial feature matrix to obtain a mixed Token sequence; S4, performing spatial-inertial collaborative attention fusion on the mixed Token sequence to obtain a human motion posture multi-modal fusion representation vector; and S5, inputting the human motion posture multi-modal fusion representation vector into a multi-layer perception machine classification head to obtain a final action label.

[0019] In the human motion posture recognition method based on multi-modal data fusion, the step S1, acquiring a human motion posture video frame and an IMU segment aligned with the human motion posture video frame. It can be understood that the human motion posture video frame can provide spatial posture information at the visual level, while the IMU segment can record dynamic inertial data such as acceleration and angular velocity of the motion. Based on this, the present application precisely aligns the human motion posture video frame and the IMU segment in the time dimension, ensures that the two modal data correspond to the same motion moment, obtains the human motion posture video frame and the human motion posture video frame aligned IMU segment that are synchronized in time, makes up for the limitations of a single modal, and enables the subsequent steps to accurately associate the visual spatial posture information and the dynamic inertial features, thereby creating conditions for subsequent deep fusion of discriminative information in the two heterogeneous data.

[0020] In particular, in one specific embodiment, Figure 3 A flowchart of the step S1 of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application is shown in FIG. 2. As shown in FIG. 2, the step S1 comprises the following steps: S11, acquiring the human motion posture video frame; and S12, aggregating all IMU data points falling within the time interval of the frame to obtain the IMU segment based on the timestamp of the human motion posture video frame. Figure 3

[0021] ​Specifically, the step S11 acquires the human motion posture video frame. Specifically, an image frame accurately reflecting the instantaneous spatial posture of human motion is acquired as the original data for visual modal feature extraction, and a continuous human motion posture video frame sequence is obtained, each human motion posture video frame containing a corresponding timestamp, completely recording the spatial posture change in the human motion process, ensuring that accurate human key point coordinates can be extracted from the video frame in the subsequent process, providing reliable visual basis for constructing the skeletal posture vector.

[0022] In particular, in one possible embodiment, the implementation process of the step S11 is as follows: in an indoor motion capture scene, a high-definition RGB camera with a resolution of 1920x1080 is deployed, the lens is directed at the human motion area, and the original video stream is collected at a fixed frame rate of 30 frames / second. The original video stream is extracted in real time by a video decoding tool, and the continuous video stream is split into independent image frames. Each extracted image frame is preprocessed, first using Gaussian filtering to remove image noise, and then uniformly adjusting the image size to 256x256 pixels to eliminate size differences between different frames. At the same time, the high-precision time synchronization module built-in the camera is used to record the acquisition timestamp of each image frame, accurate to the millisecond level, ensuring the accuracy of the time information. Finally, a human motion posture video frame sequence containing timestamps is generated, each frame of which can clearly display the spatial positions of key parts such as the head, torso, and limbs of the human body, providing standardized visual input for subsequent operations.

[0023] Specifically, the step S12 collects and integrates the inertial measurement unit (IMU) data corresponding in time to the human motion posture video frame based on the timestamp of the human motion posture video frame to obtain the IMU segment. In order to adapt to different application scenarios and hardware configurations, the collection and integration of IMU data can be flexibly configured. For example, the IMU data can come from sensors worn on one or more key parts of the human body (such as the waist, wrist, ankle, torso, etc.). In one implementation, the system can aggregate data from all deployed sensors to obtain the most comprehensive motion dynamic information. In another implementation, the system can selectively use the information of only one or a part of the preset sensors according to the specific action type to be recognized, for example, when analyzing gait, the data of the ankle sensor can be preferentially or only used, thereby reducing the complexity of data processing and enhancing the pertinence. In addition, the selection of sensors can also be dynamic, for example, the system can analyze the readings of each sensor in real time and select the sensor with the largest motion amplitude or the most significant data at the current time point as the effective data source at that time point.

[0024] In terms of time alignment of data collection, the time interval for aggregating IMU data points is also not fixed. The time interval can be flexibly set according to the analysis requirements, for example, it can be a backward time window starting from the video frame timestamp, or a center-symmetric time window around the timestamp, or a forward time window ending at the timestamp, so as to more accurately capture the inertial characteristics before, during or after a specific action. After determining the sensor source and the time window, the processing method of the IMU data points in the window is also diverse. The system can choose to aggregate all IMU data points in the time window to capture the complete dynamic process. Alternatively, the system can also perform selective aggregation, for example, only aggregate those data points whose acceleration or angular velocity values exceed a preset threshold, in this way, focus on the key events or high-intensity stages in the motion process, effectively filter out redundant information in the steady state. Alternatively, in order to further simplify data processing, the system can also select a single IMU data point closest in time to the video frame timestamp, or a small data sequence before and after the data point, as a representative IMU segment. Through the combination of one or more of the above methods, it can be ensured that the obtained IMU segment can be accurately synchronized in time with the corresponding video frame, and can be optimized according to actual requirements, so as to provide high-quality and high-relevance inertial data input for subsequent feature extraction and fusion, avoiding cross-modal information association errors caused by time misalignment or data redundancy.

[0025] In another optional embodiment, the collection of IMU segments can be triggered by key events. In this mode, the system does not collect complete IMU segments for each video frame, but first uses one or more IMU sensors (for example, sensors located on the core torso of the human body) for real-time monitoring. When a specific indicator (such as the modulus of acceleration or the energy of angular velocity) of the IMU reading exceeds a preset dynamic threshold in a short period of time, the system determines that a "key motion event" occurs. At this time, the system defines a time window centered on the timestamp of the event, and collects data of one or more related IMU sensors in the window to form an IMU segment. This event-triggered collection method can intelligently filter out the most intense moments of motion change, greatly reducing data redundancy in stationary or gentle motion states, improving the efficiency of data collection and processing, and is particularly suitable for analyzing explosive or intermittent actions.

[0026] In yet another optional embodiment, the generation of the IMU segment can be in a manner of weighted fusion. When a human body wears multiple IMU sensors, different sensors have different contribution degrees to a specific action. Therefore, when aggregating the data of multiple sensors, a weight can be assigned to each sensor. The weight can be statically preset, for example, when analyzing the fine operation of the upper limbs, a higher value is set for the weight of the wrist sensor, and a lower value is set for the weight of the ankle sensor. The weight can also be dynamically calculated, for example, the system can dynamically adjust the weight of each sensor data according to the variance or energy size of each sensor data in each time window, and the weight is inclined to the body part that is most active in the current motion. The final IMU segment can be obtained by weighted averaging or weighted splicing of the sensor data, so as to generate an inertial data representation that better reflects the core features of the action and has more semantic information.

[0027] In the above-mentioned human motion posture recognition method based on multi-modal data fusion, the step S2 performs parallel feature extraction on the human motion posture video frame and the IMU segment to obtain the skeletal posture vector and the inertial feature matrix. It should be understood that the parallel extraction can be designed to adapt to the characteristics of different modal data, avoiding the suppression or distortion of single extraction method to heterogeneous data features, so as to retain the unique information of each modal. Specifically, the skeletal posture vector which can accurately represent the spatial position and structural relationship of the human body is extracted from the human motion posture video frame, which accurately describes the spatial distribution and relative position of the limbs through the human key point coordinates, and the inertial feature matrix which can reflect the dynamic change of the motion is extracted from the IMU segment, which captures the acceleration, angular velocity and other inertial information of the motion through encoding the time sequence dynamic change of the IMU data. Through parallel processing, the core features of the two modalities are effectively captured, and the core discriminant information of the two modalities is retained, which improves the effectiveness and depth of subsequent fusion.

[0028] In particular, in one specific embodiment, Figure 4 The flowchart of the sub-step S2 of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application is shown in FIG. 2. As shown in FIG. 2, the step S2 includes: S21, inputting the human motion posture video frame into a pre-trained pose estimation network to obtain K human key point coordinates, wherein the K human key point coordinates constitute the skeletal posture vector; S22, performing one-dimensional convolution coding and nonlinear activation on the IMU segment to obtain the inertial feature matrix. Figure 4

[0029] ​Specifically, the step S21 inputs the human motion posture video frame into a pre-trained posture estimation network to obtain K human key point coordinates, wherein the K human key point coordinates constitute the skeleton posture vector. Specifically, the application extracts discriminative human key point coordinates from the human motion posture video frame, which can structurally represent the spatial posture of the human body, providing a simple and effective visual feature carrier for subsequent cross-modal fusion. In this way, the skeleton posture vector composed of K human key point coordinates is obtained, which accurately depicts the spatial positions and relative layouts of key parts such as the head, torso, and limbs of the human body, effectively removes redundant visual information in the video frame, retains the spatial structure features of the motion posture, and enables the core discriminative information of the visual modality to be structurally presented, thereby laying a clear spatial feature foundation for deep fusion with inertial features.

[0030] In particular, in one possible embodiment, the implementation process of the step S21 is as follows: the preprocessed human motion posture video frame is input into a pre-trained posture estimation network, which gradually extracts and refines image features through a multi-resolution fusion mechanism. The input image is first mapped to features through an initial convolutional layer, then the features are interacted and enhanced through multiple high and low resolution parallel branches, and finally 17 human key point coordinates are output through a prediction head, covering the top of the head, the neck, the left shoulder, the right shoulder, the left elbow, the right elbow, the left wrist, the right wrist, the hip, the left hip, the right hip, the left knee, the right knee, the left ankle, the right ankle, the nose, and the chest. Each key point coordinate is represented by an (x, y) pixel position. The 17 key point coordinates are arranged in a fixed order to form a skeleton posture vector with a dimension of 34, which fully reflects the spatial skeleton structure of the human body in the corresponding video frame.

[0031] Specifically, the step S22 performs one-dimensional convolution encoding and nonlinear activation on the IMU segment to obtain the inertial feature matrix. It should be understood that one-dimensional convolution can automatically learn and filter out noise and redundant information in IMU data. By one-dimensional convolution encoding and nonlinear activation, the application captures complex and nonlinear time series patterns in the IMU segment, such as the acceleration curve shape of a specific action and the instantaneous change of angular velocity. The inertial feature matrix obtained in this way compresses the original data dimension through convolution operation, highlights local time series patterns such as acceleration change rate and angular velocity peak value, and enhances the representation ability of complex motion states after nonlinear activation, so that the core discriminative information of the inertial modality is structured in the form of a matrix, providing an inertial feature foundation rich in dynamic features for deep fusion with the skeleton posture vector.

[0032] In particular, in one possible embodiment, the step S22 is implemented as follows: the IMU segment with a length of 64 and each data point containing 6 inertial parameters, i.e., x, y, z axis acceleration and angular velocity, is input into a one-dimensional convolutional network. First, local feature extraction is performed on the IMU segment by a first convolutional layer with a convolution kernel size of 3, a step size of 1, and 32 output channels, to capture the inertial change pattern between adjacent 3 data points, and the feature nonlinearity expression is enhanced by a ReLU nonlinear activation function. Then, a second convolutional layer with a convolution kernel size of 3, a step size of 2, and 64 output channels is used to further extract higher-level time sequence features and compress the sequence length, and a feature sequence with a length of 32 is output. Finally, an inertial feature matrix with a dimension of 32x64 is obtained, where each row corresponds to a compressed time sequence position, and each column corresponds to an encoded inertial feature, which fully reflects the dynamic inertial change characteristics in the IMU segment.

[0033] In the above human motion posture recognition method based on multi-modal data fusion, the step S3 is to perform mixed Tokenization and embedding on the skeletal posture vector and the inertial feature matrix to obtain a mixed Token sequence. It should be understood that, since the skeletal posture vector and the inertial feature matrix are derived from two heterogeneous modalities of vision and inertia respectively, their data structures and feature attributes are significantly different. The skeletal posture vector embodies spatial structure information in the form of key point coordinates, while the inertial feature matrix carries dynamic inertial characteristics in the form of a time sequence matrix. In order to convert the heterogeneous features of the two into unified sequence features, the skeletal posture vector and the inertial feature matrix are converted into a mixed Token sequence with unified dimensions and semantically associated semantics in the present application. By adding modal embedding, position embedding and other additional information to the Tokens of different modalities, the model can distinguish the modal origin, spatial or time sequence position of the features, and at the same time, the unified feature dimension is obtained by linear projection, obtaining a mixed Token sequence containing single spatial Tokens, single inertial Tokens and classification Tokens. Each Token is processed by embedding to carry modal, position and other meta-information, and has a unified feature dimension, so that the visual spatial features and the inertial dynamic features can be represented in the same sequence framework, eliminating the structural differences between heterogeneous features, providing a structured input basis for subsequent operations, and ensuring effective interaction and complementation of modal information in the subsequent fusion process.

[0034] In particular, in one specific embodiment, Figure 5 The flowchart of the sub-step S3 of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application is shown in FIG. 6. As shown in FIG. 6, Figure 5As shown, the step S3 includes: S31, taking the human body key point coordinates of each position in the skeleton pose vector as a single space Token to obtain a set of single space Tokens; S32, taking each row vector in the inertia feature matrix as a single inertia Token to obtain a set of single inertia Tokens; S33, adding a classification Token; S34, performing linear projection, type embedding and visual modal embedding on each single space Token in the set of single space Tokens to obtain a set of single space Token final representations; S35, performing linear projection, position embedding and inertia modal embedding on each single inertia Token in the set of single inertia Tokens to obtain a set of single inertia Token final representations; S36, aggregating the set of single space Token final representations, the set of single inertia Token final representations and the classification Token to obtain the mixed Token sequence.

[0035] Specifically, the step S31 takes the human body key point coordinates of each position in the skeleton pose vector as a single space Token to obtain a set of single space Tokens. It can be understood that the skeleton pose vector is composed of multiple human body key point coordinates, and these key points correspond to different parts of the human body, such as the head, the torso and the limbs, and the spatial position of each part is the core local feature representing the human body pose. Taking each key point as an independent Token can preserve the independent identity and spatial information of each part, avoid the loss or blurring of local features caused by the overall processing of the skeleton pose vector, and ensure that the subsequent fusion can accurately associate the spatial features and inertia features of each part of the human body.

[0036] In particular, in one possible embodiment, the implementation process of the step S31 is as follows: for a skeleton pose vector composed of 17 human body key point coordinates, the coordinates of each key point are taken as independent single space Tokens, arranged in a fixed order according to the parts of the human body, such as (x1, y1) for the top of the head, (x2, y2) for the neck, etc., to form a set of 17 single space Tokens, each Token taking the coordinate value of its corresponding key point as the initial content, and completely preserving the spatial position information of each part.

[0037] Specifically, the step S32 takes each row vector in the inertial feature matrix as a single inertial Token to obtain a set of single inertial Tokens. It should be understood that the row vectors of the inertial feature matrix correspond to inertial features at different time steps, and each inertial feature at a time step, such as a fused feature of acceleration and angular velocity, is a basic time sequence unit representing dynamic changes of human motion. Taking each row vector as a single inertial Token can preserve the time sequence continuity and local dynamic pattern of the inertial features, avoid confusion of time sequence information caused by processing the matrix as a whole, and ensure that the subsequent fusion can capture the correlation between inertial changes at different time steps and spatial poses.

[0038] In particular, in a possible embodiment, the implementation process of the step S32 is as follows: for an inertial feature matrix with a dimension of 32x64, where 32 rows represent 32 time steps and 64 columns represent multi-dimensional inertial features at the time steps, each row vector in the matrix is taken as an independent single inertial Token, arranged in the order of time steps from 1 to 32, to form a set of 32 single inertial Tokens, each single inertial Token taking the 64-dimensional inertial feature vector of its corresponding row as initial content, and completely preserving dynamic inertial change information at each time step.

[0039] Specifically, the step S33 adds a classification Token. That is, a special classification Token that integrates all feature information is introduced into the mixed Token sequence as an information aggregator of the entire mixed Token sequence, facilitating subsequent extraction of global fusion features for recognition.

[0040] In particular, in a possible embodiment, the implementation process of the step S33 is as follows: a special classification Token with an initial value of a random vector is introduced in addition to the single spatial Token set and the single inertial Token set. The classification Token does not correspond to any specific body part or inertial feature at a time step, but only serves as an information aggregation carrier. It is added to the mixed Token sequence to be formed and participates in the subsequent embedding and fusion process together with other Tokens.

[0041] Specifically, the step S34 performs linear projection, type embedding, and visual modality embedding on each single spatial Token in the set of single spatial Tokens to obtain a set of final representations of single spatial Tokens. Specifically, the dimensions of single spatial Tokens are unified through linear projection, body part identity information is injected through type embedding, and the modality source is specified through visual modality embedding to obtain a set of final representations of single spatial Tokens, each of which can represent the spatial features of the corresponding part and be identified by the model as to the part identity and modality source, providing an interactive spatial feature unit for cross-modal attention fusion.

[0042] In particular, in one possible embodiment, the implementation of step S34 is as follows: first, linear projection is performed, for each single-body Token consisting of 17 human key point coordinates, each single-body Token is initially a 2-dimensional coordinate vector (x, y). In order to adapt these Tokens to the hidden dimension of the subsequent Transformer model, a learnable linear projection layer is introduced, such as nn.Linear(2, 256), which maps each 2-dimensional coordinate vector to a 256-dimensional high-dimensional vector through learning of a weight matrix, realizing the unification and elevation of feature dimension, and providing a basic vector representation for subsequent embedding operations. Then type embedding is performed, since different human key points have different body part identities, such as the top of the head and the left knee, which have different roles in motion gestures. Therefore, a unique type embedding vector of 256 dimensions is set for each of the 17 key points, each type embedding vector is 256-dimensional and learnable, for example, the top of the head corresponds to a unique 256-dimensional vector, and the neck corresponds to another unique 256-dimensional vector. Then the 256-dimensional vector of each single-body Token after linear projection is added element by element with the type embedding vector of the body part corresponding to the Token, so that the Token carries the identity information of the body part it belongs to, helping the model to distinguish the features of different key points. Finally, visual modality embedding is performed, in order to clearly distinguish the modality source of the Token, a 256-dimensional learnable visual modality embedding vector is set, which is specifically used to identify the visual modality. The 256-dimensional vector of each single-body Token after linear projection and type embedding is added element by element with the visual modality embedding vector, so that all single-body Tokens carry the identity of the visual modality, ensuring that the model can clearly identify that it comes from visual data in the subsequent cross-modal fusion process. Through the above sequential operations of linear projection, type embedding and visual modality embedding, each single-body Token finally forms a 256-dimensional vector, and the 17 single-body Tokens together constitute the set of single-body Token final representations.

[0043] Specifically, for each single-inertial Token in the set of single-inertial Tokens, linear projection, position embedding and inertial modality embedding are performed to obtain the set of single-inertial Token final representations. Similarly, the present application unifies the dimensions of single-inertial Tokens through linear projection, injects time step sequence information through position embedding, and makes each inertial Token have unified dimensions and time sequence and modality meta-information through inertial modality embedding, so as to obtain the set of single-inertial Token final representations with unified dimensions and carrying position and modality information, each Token can represent the inertial features of the corresponding time step, and can be identified by the model in terms of time sequence position and modality source, providing interactive time sequence feature units for cross-modal attention fusion.

[0044] In particular, in one possible implementation, the step S35 is implemented as follows: Similar to the step S34 described above, first, for the set of 32 monomer inertial Tokens, the 64-dimensional initial feature vector of each Token is mapped to the model hidden dimension 256-dimensional by an independent linear projection layer. Then a corresponding position embedding is added for each Token, 32 different 256-dimensional learnable vectors corresponding to time steps 1 to 32, respectively, identifying the temporal order. A uniform inertial modal embedding is then added for each Token, a 256-dimensional learnable vector, identifying that it comes from the inertial modality. Finally, the linear projection result is added to the position embedding and the inertial modal embedding to obtain the final representation of each monomer inertial Token, forming a set of 32 256-dimensional vectors of monomer inertial Token final representations.

[0045] In particular, the step S36 aggregates the set of monomer spatial Token final representations, the set of monomer inertial Token final representations and the classification Token to obtain the mixed Token sequence. That is, different types of Tokens are integrated into a continuous mixed Token sequence, so that spatial features, inertial features and classification Tokens coexist in the same sequence framework, obtaining a mixed Token sequence containing all Tokens, the dimensions of each Token in the sequence are uniform and arranged in a certain order, and carry their own meta-information, providing a complete input sequence for the attention mechanism of the subsequent heterogeneous Transformer encoder layer.

[0046] In particular, in one possible implementation, the step S36 is implemented as follows: The set of 17 monomer spatial Token final representations, the set of 32 monomer inertial Token final representations and the 1 classification Token are aggregated in order, with the order being the classification Token first, followed by the monomer spatial Token final representations, and finally the monomer inertial Token final representations, in the order of time steps 1 to 32, forming a mixed Token sequence with a total length of 1+17+32=50, each Token being a 256-dimensional vector, together constituting a mixed Token sequence processable by the Transformer.

[0047] In the human motion posture recognition method based on multi-modal data fusion, the step S4 is to perform spatial-inertial collaborative attention fusion on the mixed Token sequence to obtain a human motion posture multi-modal fusion feature vector. In one specific example of the present application, the step S4 includes: inputting the mixed Token sequence into a heterogeneous Transformer encoder layer for spatial-inertial collaborative attention fusion, the heterogeneous Transformer encoder layer including a parallel multi-head attention layer and a collaborative attention module, a first residual connection layer and a first layer normalization layer, a feedforward neural network, and a second residual connection layer and a second layer normalization layer. It should be understood that, since the mixed Token sequence with simple sequence splicing cannot capture the complex nonlinear dependence relationship between modalities, the parallel multi-head attention layer of the heterogeneous Transformer encoder layer can process the association of Tokens within the same modality, and the collaborative attention module can specifically capture the dynamic association across modalities, the residual connection and normalization layer can stabilize the training process, and the feedforward neural network can enhance the nonlinear expression of features. Through this structure, the deep association between the spatial and inertial modalities can be effectively mined, and the lack of modality information collaboration caused by the lack of targeted fusion mechanism can be avoided. Based on this, the present application utilizes the specific structure of the heterogeneous Transformer encoder layer, strengthens the association of Tokens within the same modality through the multi-head attention layer, such as the spatial relationship between different joints and the timing relationship between different time step inertial features, accurately captures the dynamic association of Tokens across modalities through the collaborative attention module, such as the mapping relationship between joint spatial position and corresponding inertial change, and combines the residual connection, normalization layer and feedforward neural network to convert the mixed Token sequence into a human motion posture multi-modal fusion feature vector containing deep cross-modal association. The spatial structure details and dynamic inertial features of human motion are retained, and the associated information across modalities is deeply integrated, such as the corresponding relationship between limb spatial displacement and acceleration change, the discriminability of the features is significantly enhanced, and a high-quality fusion feature basis is provided for the subsequent multi-layer perceptron classification head to output accurate action labels, effectively improving the recognition ability of subtle action differences in complex scenarios.

[0048] In particular, in one specific embodiment, Figure 6 The flowchart of the step S4 of the human motion posture recognition method based on multi-modal data fusion according to the embodiment of the present application. As Figure 6As shown, the step S4 includes: S41, extracting a subset of core joint Token final representations representing a torso from the set of single-body space Token final representations; S42, performing mean pooling on the subset of core joint Token final representations representing the torso to obtain a subject pose Token representation; S43, inputting the subject pose Token representation into a first small neural network including a linear layer and an activation layer to obtain a temporal weight; S44, performing global mean pooling on the set of single-body inertia Token final representations to obtain a key event Token representation; S45, inputting the key event Token representation into a second small neural network including a linear layer and an activation layer to obtain a spatial weight; and S46, transmitting the temporal weight and the spatial weight to the multi-head attention layer, wherein the temporal weight or the spatial weight acts on a value vector.

[0049] Specifically, the step S41 extracts a subset of core joint Token final representations representing a torso from the set of single-body space Token final representations. That is, the core joint Token final representations representing the overall posture of the human body are screened from the set of single-body space Token final representations to form a subset of core joint Token final representations representing the torso for guiding the inertia mode, which condenses the overall posture information of the human body and provides a high-quality input for subsequent generation of the subject pose Token, so that the visual guiding inertia mechanism has a clear global posture basis.

[0050] In particular, in a possible embodiment, the step S41 is implemented as follows: from the set of 17 single-body space Token final representations, the Token representing the core joint of the torso is screened according to the structural features of the human body, specifically including four Tokens of left shoulder, right shoulder, left hip and right hip. The final representations of these Tokens are all 256-dimensional vectors, which correspond to the spatial features of the shoulder and hip, respectively, and together constitute the subset of core joint Token final representations. This subset retains the spatial information of the connection between the upper body and the lower body of the human body and can effectively reflect the change of the overall posture.

[0051] Specifically, the step S42, the mean pooling is performed on the subset of the final representations of the core joints Token representing the torso to obtain the subject posture Token representation. It should be understood that the mean pooling can aggregate the features of multiple core joints by averaging, avoiding the one-sidedness of a single joint feature. Specifically, the local features of the subset of the final representations of the core joints Token representing the torso are aggregated into a unified global posture representation to obtain the subject posture Token representation which can comprehensively reflect the overall posture of the human body and condenses the spatial features of the core joints, providing a global posture basis for the first small neural network to generate the time weight, ensuring that the time weight matches the overall motion state.

[0052] In particular, in a possible embodiment, the implementation process of the step S42 is as follows: for the subset of the final representations of the four core joints Token representing the left shoulder, the right shoulder, the left hip and the right hip, each being a 256-dimensional vector, the mean pooling operation is adopted to average the corresponding dimensions of the four Tokens, that is, for each dimension d (1≤d≤256), the numerical average of the four Tokens in the d dimension is calculated to obtain a 256-dimensional vector, which is the subject posture Token representation that comprehensively reflects the spatial features of the shoulder and the hip and can effectively reflect the overall torso posture of the human body, such as the overall state of the hip sinking and the shoulder leaning forward in the squat.

[0053] Specifically, the step S43, the subject posture Token representation is input into the first small neural network containing a linear layer and an activation layer to obtain the time weight. It should be understood that the first small neural network can map the subject posture Token representation to a time weight indicating the importance of the inertia Token by learning the correlation between the overall posture and the key time step of the inertia data, so that the focus of the inertia modality matches the overall posture and avoids the interference of irrelevant time steps in the inertia data.

[0054] In particular, in a possible embodiment, the implementation process of the step S43 is as follows: first, the 256-dimensional subject posture Token representation is input into the first small neural network, which contains a linear layer and an activation layer. The input dimension of the linear layer is 256, and the output dimension is 32, which is consistent with the number of inertia Tokens. The subject posture Token is mapped to a 32-dimensional vector through a learnable weight matrix. Then the vector is input into the Sigmoid activation layer to compress the value to between 0 and 1 to obtain 32 time weights, each weight corresponding to an inertia Token, in the order of time steps 1 to 32, for example, in the squat bottom posture, the inertia Token weight corresponding to the moment when the knee joint changes dramatically is adjusted to be above 0.9, while the weight of the moment when the action is smooth is below 0.3, so as to highlight the inertia features of the key time steps.

[0055] Specifically, the step S44 globally average-pools the set of monomer inertia Token final representations to obtain a key event Token representation. It should be understood that the set of monomer inertia Token final representations contains multiple time step local features, and the global average-pooling can integrate all time step features to aggregate the key event Token representation reflecting the overall inertia dynamics, avoid the contingency of a single time step, capture the key inertia event throughout multiple time steps, and provide inertia basis for spatial weight generation.

[0056] In particular, in a possible embodiment, the implementation process of the step S44 is as follows: globally average-pooling the 32 monomer inertia Token final representations each with 256 dimensions, taking the average of the 32 Tokens for each dimension, and obtaining a 256-dimensional key event Token representation. This integrates all time step inertia features, such as periodic acceleration changes during running, to provide overall inertia dynamic basis for inertia-guided vision.

[0057] Specifically, the step S45 inputs the key event Token representation into a second small neural network containing a linear layer and an activation layer to obtain a spatial weight. It should be understood that, since the key event Token representation reflects the overall inertia dynamics but cannot be directly converted into the attention weight of the spatial Token, the second small neural network can learn the association between the inertia event and the key part of the human body and map it to the spatial weight, so that the spatial attention focus matches the inertia event. In particular, in a possible embodiment, the implementation process of the step S45 is similar to the step S43, and the 256-dimensional key event Token is input into the second small neural network, mapped to a 17-dimensional vector through the linear layer, and then activated through the Sigmoid activation to obtain 17 spatial weights.

[0058] Specifically, the step S46 transmits the time weight and the spatial weight to the multi-head attention layer, wherein the time weight or the spatial weight acts on the value vector. It should be understood that the time weight and the spatial weight carry inter-modal guidance information, and acting on the value vector can adjust the contribution degree of different Tokens, amplify important Token features, suppress irrelevant Token influence, and avoid key information being submerged. Through weight adjustment of the value vector, the inertia Token related to the current posture and the spatial Token related to the inertia event have higher weights in attention calculation in the present application, the key inertia time step and the human body part features in the fusion features are enhanced, the cross-modal association is more accurate, and more discriminative fusion representation is provided for subsequent classification.

[0059] In particular, in one possible embodiment, the step S46 is implemented as follows: 32 time weights and 17 space weights are transmitted to the multi-head attention layer, the 256-dimensional inertia Token value vector is multiplied element by element with the corresponding time weight, and the space Token value vector is multiplied element by element with the corresponding space weight. The adjusted value vector participates in attention calculation, so that high-weight Token features are more prominent in fusion. For example, in the knee-inward buckling action, the abnormal inertia time step and the knee joint space Token feature are strengthened, and the fusion pertinence is improved.

[0060] In the above human motion posture recognition method based on multi-modal data fusion, the human motion posture multi-modal fusion feature vector is input into the multi-layer perception machine classification head to obtain the final action label. It can be understood that the multi-layer perception machine classification head maps the high-dimensional human motion posture multi-modal fusion feature vector to the specific action category space step by step through the combination of multi-layer linear transformation and nonlinear activation function, realizing the conversion from fusion feature to action label. Specifically, the multi-layer perception machine classification head captures the complex nonlinear relationship between features and categories by using hierarchical structure: first, the linear layer is used to perform dimension transformation on the input human motion posture multi-modal fusion feature, reducing the feature dimension while extracting more abstract discriminative information. Second, a nonlinear activation function such as ReLU is introduced to enhance the fitting ability of the model to nonlinear features, so that the feature mapping is more consistent with the distribution of actual action categories. Finally, the feature is mapped to a dimension matching the number of action categories through the output layer, and is converted into a probability distribution of each category through the Softmax function. The category with the highest probability is the corresponding action label, which accurately reflects the fine-grained posture of human motion, such as error deep squat-knee-inward buckling, and improves the accuracy of action recognition in complex scenarios.

[0061] In particular, in one possible embodiment, the step S5 is implemented as follows: the 256-dimensional human motion posture multi-modal fusion representation vector obtained by fusing the spatial-inertial collaborative attention is input into a multi-layer perceptron classification head, which includes three linear layers and two ReLU activation functions. The first linear layer maps the 256-dimensional vector to a 128-dimensional vector, which is input into the second linear layer after ReLU activation, and then mapped to a 64-dimensional vector, which is input into the third linear layer after ReLU activation, and finally mapped to a 5-dimensional vector corresponding to 5 fine-grained action categories, including standard deep squat, incorrect deep squat-knee inward buckling, incorrect deep squat-forward center of gravity, standard lunge, and incorrect lunge-torso forward. The Softmax layer is used to output the probability distribution of the 5 categories, for example, the output probability vector is [0.02, 0.91, 0.03, 0.03, 0.01], in which the probability of incorrect deep squat-knee inward buckling is the highest. For single judgment, the category with the highest probability is directly taken as the final action label. For continuous action recognition, moving average smoothing processing is performed on the probability vectors of the continuous 10 frames to eliminate transient jitter, and finally the stable incorrect deep squat-knee inward buckling action label is output.

[0062] Notably, the single spatial Token is the human key point coordinates of each position in the skeletal pose vector, and the single inertial Token is each row vector in the inertial feature matrix, so there is a large difference in the representation dimension between the single spatial Token and the single inertial Token. This difference acts on the value vector via activation, so that the fusion expression of the human motion posture multi-modal fusion representation vector may emphasize the attention significance of space-inertia more, but lacks collaborative consistency. The compensation ability of the linear layer of the multi-layer perceptron classification head is effective, which will affect the accuracy of the final classification result. Based on this, in another preferred embodiment, the step S5 includes: performing spatial pose distribution formalization on the feature values of the human motion posture multi-modal fusion representation vector to obtain a human motion posture multi-modal fusion spatial pose distribution representation vector; performing inertial distribution formalization on the feature values of the human motion posture multi-modal fusion representation vector to obtain a human motion posture multi-modal fusion inertial distribution representation vector; performing consistent joint coding on the human motion posture multi-modal fusion spatial pose distribution representation vector and the human motion posture multi-modal fusion inertial distribution representation vector to obtain a human motion posture multi-modal consistency fusion representation vector; and inputting the human motion posture multi-modal consistency fusion representation vector into a multi-layer perceptron classification head to obtain a final action label.

[0063] Specifically, for the human motion posture multi-modal fusion representation vector, the feature values are denoted as , first, considering the normal distribution of the spatial distribution, the feature value of the human motion posture multi-modal fusion representation vector is formalized in the form of spatial posture distribution to obtain a human motion posture multi-modal fusion spatial posture distribution representation vector, that is, ; wherein, represents the th feature value of the human motion posture multi-modal fusion representation vector, and respectively represent the mean and variance of the set composed of all feature values of the human motion posture multi-modal fusion representation vector, represents the th feature value of the human motion posture multi-modal fusion spatial posture distribution representation vector, , represents the human motion posture multi-modal fusion spatial posture distribution representation vector.

[0064] That is, the spatial posture features contained in the multi-modal fusion vector are converted into formalized representations conforming to the normal distribution probability density, highlighting the distribution rules of key spatial features, suppressing noise features deviating from the normal distribution, and enhancing the discriminability of spatial posture features. In this way, the spatial position distribution of core joints such as the knee joint and the hip joint can be accurately captured, and through the normal distribution model, the model pays more attention to features conforming to the standard deep squat joint angle range, and weakens accidental deviations caused by body shaking, providing more robust spatial feature basis for subsequent.

[0065] and based on the correlation response of the inertia distribution, the feature value of the human motion posture multi-modal fusion representation vector is formalized in the form of inertia distribution to obtain a human motion posture multi-modal fusion inertia distribution representation vector, that is, ; wherein, represents the human motion posture multi-modal fusion representation vector, represents matrix multiplication, represents the transpose symbol, represents the normalized exponential function, represents the human motion posture multi-modal fusion inertia distribution representation vector, wherein the vectors are all in the form of row vectors.

[0066] That is, the spatial posture distribution form (describing the point-based distribution probability density of feature values) and the inertia distribution form (based on the dynamic correlation response probability of inertia) are modeled respectively in units of feature values and feature vectors. Specifically, by matrix multiplication and softmax normalization, the dynamic correlation response relationship between inertia features is captured, and the inertia features are converted from numerical values at a single time point to correlation distribution across time steps, so as to highlight the time sequence dependence of inertia features, such as the acceleration peak at the start of the action and the inertia continuation of the subsequent action.

[0067] ​​Then, the human motion posture multi-modal fusion spatial posture distribution representation vector and the human motion posture multi-modal fusion inertial distribution representation vector are uniformly and jointly coded to obtain a human motion posture multi-modal uniformity fusion representation vector, that is, ; wherein, represents the human motion posture multi-modal fusion spatial posture distribution representation vector, represents the human motion posture multi-modal fusion inertial distribution representation vector, represents the position point multiplication, represents the vector subtraction, represents the vector addition, represents the F-norm of the matrix, represents the two-norm of the matrix, represents the predetermined weight hyperparameter, represents the human motion posture multi-modal uniformity fusion representation vector.

[0068] That is, first, a low-dimensional manifold mapping is performed for the spatial-inertial relationship, and then each distribution form is compensated and weighted as a weighted coefficient, so that the spatial posture-inertial distribution is uniformly and jointly learned, so that a collaborative robust norm of the spatial posture-inertial distribution representation is established, the spatial-inertial collaborative uniformity of the fusion expression of the human motion posture multi-modal fusion representation vector is improved, and the accuracy of the classification result of the multi-layer perception classification head is improved. Specifically, by weighting fusion of the F-norm capable of measuring the overall matching degree of the spatial-inertial distribution and the two-norm capable of measuring the local feature deviation, the modal conflict between the spatial posture and the inertial feature is eliminated, such as normal joint position in vision but motion pause in inertial data, the consistency association of the two is strengthened, and the collaboration of the cross-modal features is improved. Finally, the human motion posture multi-modal uniformity fusion representation vector is input into the multi-layer perception classification head to obtain the final action label.

[0069] In summary, the human motion posture recognition method based on multi-modal data fusion based on the embodiments of the present application is illustrated, which first synchronously acquires human motion posture video frames and IMU segments, then extracts features from the two kinds of heterogeneous data in a split parallel manner, respectively capturing spatial posture information and dynamic inertial features of IMU. Subsequently, these modal-specific features are uniformly represented through a hybrid Tokenization and embedding mechanism, and a spatial-inertial collaborative attention fusion mechanism is further introduced to realize dynamic association and deep fusion of cross-modal information, and finally the multi-modal fusion representation vector obtained by fusion is classified to realize accurate recognition of human motion posture. In this way, the defect of insufficient capture of subtle action differences in traditional fusion can be overcome, and the accuracy and stability of action recognition in complex scenes can be improved.

[0070] Figure 7 A block diagram of a human motion gesture recognition system based on multi-modal data fusion according to an embodiment of the present application. As shown in Figure 7 The human motion gesture recognition system 100 based on multi-modal data fusion according to an embodiment of the present application comprises: a multi-modal data acquisition module 110, configured to acquire human motion gesture video frames and IMU segments aligned with the human motion gesture video frames; a split-parallel feature extraction module 120, configured to perform split-parallel feature extraction on the human motion gesture video frames and the IMU segments to obtain a skeletal pose vector and an inertial feature matrix; a mixed Tokenization and embedding module 130, configured to perform mixed Tokenization and embedding on the skeletal pose vector and the inertial feature matrix to obtain a mixed Token sequence; a multi-modal fusion module 140, configured to perform spatial-inertial collaborative attention fusion on the mixed Token sequence to obtain a human motion gesture multi-modal fusion representation vector; and a motion classification module 150, configured to input the human motion gesture multi-modal fusion representation vector into a multi-layer perception machine classification head to obtain a final motion label.

[0071] As described above, the human motion gesture recognition system 100 based on multi-modal data fusion according to an embodiment of the present application can be implemented in various wireless terminals, such as a server with a human motion gesture recognition algorithm based on multi-modal data fusion, and the like. In one possible implementation, the human motion gesture recognition system 100 based on multi-modal data fusion according to an embodiment of the present application can be integrated into a wireless terminal as a software module and / or a hardware module. For example, the human motion gesture recognition system 100 based on multi-modal data fusion can be a software module in the operating system of the wireless terminal, or can be an application program developed for the wireless terminal; of course, the human motion gesture recognition system 100 based on multi-modal data fusion can also be one of the many hardware modules of the wireless terminal.

[0072] Alternatively, in another example, the human motion gesture recognition system 100 based on multi-modal data fusion and the wireless terminal can also be separate devices, and the human motion gesture recognition system 100 based on multi-modal data fusion can be connected to the wireless terminal through a wired and / or wireless network, and transmit interactive information in an agreed data format.

[0073] Here, those skilled in the art can understand that the specific operations of each step in the above human motion gesture recognition system based on multi-modal data fusion have been described in detail above with reference to the human motion gesture recognition method based on multi-modal data fusion of Figures 1 to 6 , and therefore repeated descriptions thereof will be omitted.

Claims

1. A human motion gesture recognition method based on multi-modal data fusion, characterized in that, The method comprises the following steps: Obtaining human motion posture video frames and IMU segments aligned with the human motion posture video frames; Splitting and parallel feature extraction is performed on the human motion posture video frames and the IMU segments to obtain a skeletal posture vector and an inertial feature matrix; Mixed Tokenization and embedding are performed on the skeletal posture vector and the inertial feature matrix to obtain a mixed Token sequence, including: taking the human key point coordinates of each position in the skeletal posture vector as a single spatial Token to obtain a set of single spatial Tokens; taking each row vector in the inertial feature matrix as a single inertial Token to obtain a set of single inertial Tokens; adding a classification Token; performing linear projection, type embedding and visual modal embedding on each single spatial Token in the set of single spatial Tokens to obtain a set of single spatial Token final representations; performing linear projection, position embedding and inertial modal embedding on each single inertial Token in the set of single inertial Tokens to obtain a set of single inertial Token final representations; Spatial-inertial collaborative attention fusion is performed on the mixed Token sequence to obtain a human motion posture multi-modal fusion representation vector, including: inputting the mixed Token sequence into a heterogeneous Transformer encoder layer for spatial-inertial collaborative attention fusion, specifically including: extracting a subset of core joint Token final representations representing the torso from the set of single spatial Token final representations; performing mean pooling on the subset of core joint Token final representations representing the torso to obtain a subject posture Token representation; inputting the subject posture Token representation into a first small neural network containing a linear layer and an activation layer to obtain a time weight; performing global mean pooling on the set of single inertial Token final representations to obtain a key event Token representation; inputting the key event Token representation into a second small neural network containing a linear layer and an activation layer to obtain a spatial weight; transmitting the time weight and the spatial weight to a multi-head attention layer, wherein the time weight or the spatial weight acts on a value vector; Inputting the human motion posture multi-modal fusion representation vector into a multi-layer perception classification head to obtain a final action label. 2.The human motion gesture recognition method based on multi-modal data fusion according to claim 1, characterized in that, Obtaining human motion posture video frames and IMU segments aligned with the human motion posture video frames, including: obtaining the human motion posture video frames; taking the timestamps of the human motion posture video frames as a reference, aggregating all IMU data points falling within this frame time interval to obtain the IMU segments. 3.The human motion gesture recognition method based on multi-modal data fusion according to claim 2, characterized in that, This frame time interval is ; wherein, is a timestamp of a human motion posture video frame, is a video frame interval. 4.The human motion gesture recognition method based on multi-modal data fusion according to claim 1, characterized in that, The human motion posture video frame and the IMU segment are branched and parallel feature extraction to obtain a skeletal posture vector and an inertial feature matrix, including: inputting the human motion posture video frame into a pre-trained posture estimation network to obtain K human key point coordinates, wherein the K human key point coordinates constitute the skeletal posture vector; one-dimensional convolutional coding and nonlinear activation are performed on the IMU segment to obtain the inertial feature matrix. 5.The human motion gesture recognition method based on multi-modal data fusion according to claim 4, characterized in that, The skeletal posture vector and the inertial feature matrix are mixed Tokenization and embedding to obtain a mixed Token sequence, also including: aggregating the set of single-body space Token final representations, the set of single-body inertial Token final representations and the classification Token to obtain the mixed Token sequence. 6.The human motion gesture recognition method based on multi-modal data fusion according to claim 5, characterized in that, The mixed Token sequence is subjected to spatial-inertial collaborative attention fusion to obtain a human motion posture multi-modal fusion representation vector, including: inputting the mixed Token sequence into a heterogeneous Transformer encoder layer for spatial-inertial collaborative attention fusion, the heterogeneous Transformer encoder layer including parallel multi-head attention layers and collaborative attention modules, a first residual connection layer and a first layer normalization layer, a feedforward neural network and a second residual connection layer and a second layer normalization layer. 7.The human motion gesture recognition method based on multi-modal data fusion according to claim 1, characterized in that, The human motion posture multi-modal fusion representation vector is input into a multi-layer perceptron classification head to obtain a final action label, including: spatial posture distribution formalization of the feature values of the human motion posture multi-modal fusion representation vector to obtain a human motion posture multi-modal fusion spatial posture distribution representation vector; inertial distribution formalization of the feature values of the human motion posture multi-modal fusion representation vector to obtain a human motion posture multi-modal fusion inertial distribution representation vector; consistent joint coding of the human motion posture multi-modal fusion spatial posture distribution representation vector and the human motion posture multi-modal fusion inertial distribution representation vector to obtain a human motion posture multi-modal consistency fusion representation vector; inputting the human motion posture multi-modal consistency fusion representation vector into a multi-layer perceptron classification head to obtain a final action label.

8. A human motion gesture recognition system based on multi-modal data fusion, characterized in that, It includes: A multi-modal data acquisition module is configured to acquire human motion posture video frames and IMU segments aligned with the human motion posture video frames; A branched and parallel feature extraction module is configured to perform branched and parallel feature extraction on the human motion posture video frames and the IMU segments to obtain a skeletal posture vector and an inertial feature matrix; The mixed Tokenization and embedding module is configured to perform mixed Tokenization and embedding on the skeleton pose vector and the inertial feature matrix to obtain a mixed Token sequence, including: taking human body key point coordinates of each position in the skeleton pose vector as a single spatial Token to obtain a set of single spatial Tokens; taking each row vector in the inertial feature matrix as a single inertial Token to obtain a set of single inertial Tokens; adding a classification Token; performing linear projection, type embedding and visual modal embedding on each single spatial Token in the set of single spatial Tokens to obtain a set of single spatial Token final representations; performing linear projection, position embedding and inertial modal embedding on each single inertial Token in the set of single inertial Tokens to obtain a set of single inertial Token final representations; The multi-modal fusion module is configured to perform spatial-inertial collaborative attention fusion on the mixed Token sequence to obtain a human motion posture multi-modal fusion representation vector, including: inputting the mixed Token sequence into a heterogeneous Transformer encoder layer to perform spatial-inertial collaborative attention fusion, specifically including: extracting a subset of core joint Token final representations representing a torso from the set of single spatial Token final representations; performing mean pooling on the subset of core joint Token final representations representing the torso to obtain a subject posture Token representation; inputting the subject posture Token representation into a first small neural network containing a linear layer and an activation layer to obtain a time weight; performing global mean pooling on the set of single inertial Token final representations to obtain a key event Token representation; inputting the key event Token representation into a second small neural network containing a linear layer and an activation layer to obtain a spatial weight; transmitting the time weight and the spatial weight to a multi-head attention layer, wherein the time weight or the spatial weight acts on a value vector; The action classification module is configured to input the human motion posture multi-modal fusion representation vector into a multi-layer perception classification head to obtain a final action label.

Images