A human activity recognition method and system for a multi-modal mobile sensing scenario

Abstract

The application discloses a human activity recognition method and system for a multi-modal mobile sensing scene, and belongs to the technical field of human activity recognition. The application proposes a two-stage pre-training technical solution for gradually reducing motion interference in a mobile scene: the first stage is cross-modal feature reconstruction, after feature extraction is performed on multiple modal data, the features of static data are used to constrain the features of dynamic data, and motion interference within the modal is eliminated; the second stage is reliable modal guided multi-modal alignment, inertial sensing data which is strong in motion interference robustness is used as a reference, features of other modal reconstruction are aligned in a feature space based on a Gram matrix, and motion interference between the modal is eliminated; when a downstream task is applied, the subject parameter is retained and the classification head is fine-tuned, so that the new task can be quickly adapted. Experimental results show that the application has more accurate, stable and effective multi-modal human activity recognition ability in a mobile scene.

Description

Technical Field

[0001] This invention relates to the field of Human Activity Recognition (HAR) technology, and in particular to a method and system for human activity recognition in multimodal motion sensing scenarios. Background Technology

[0002] With the rapid development of IoT technology, human activity recognition technology has become an important research and application topic in the field of human-computer interaction. It not only provides us with a way to deeply understand and recognize human behavior patterns from a sensor perspective, but also assists users in their daily lives and shares tasks, playing a vital role and demonstrating enormous potential in many scenarios such as abnormal behavior monitoring, healthcare, smart homes, and work collaboration.

[0003] However, most existing human activity recognition technologies rely on fixed-deployment sensors, which require significant effort to install and debug, and are often difficult to relocate. Furthermore, in indoor environments with multiple obstructions, using fixed-deployment sensors for human activity recognition often results in blind spots due to distance and installation angle issues, failing to meet the need for comprehensive and continuous monitoring.

[0004] Mobile robots are equipped with depth cameras and millimeter-wave radar for path navigation and obstacle avoidance, lidar for target detection, microphones for user interaction, and currently the most advanced color cameras for recognition. These sensors endow robots with powerful environmental awareness and interaction capabilities, and also make it possible to implement high-precision human activity recognition systems on robots. Therefore, human activity recognition on mobile platforms like robots, which can freely follow people and are equipped with multiple sensors, has become an emerging research direction.

[0005] However, when using mobile robots to collect data, it is unavoidable to eliminate the interference of movement on the data acquisition process. This interference can be divided into several categories, including viewpoint shifts caused by changes in relative position or angle, increased noise due to changes in the radio frequency signal reflection path, and changes in environmental background information caused by changes in sensor position.

[0006] Currently, many motion interference correction methods attempt to mitigate these effects. For example, preprocessing video data by combining human detection, motion compensation, and frame sampling can improve the model's resistance to motion interference by selecting key frames with greater information content and reducing interference in irrelevant areas. Another approach to processing spectral data acquired by millimeter-wave radar is to take a signal processing perspective, selecting a reference signal in the environment that contains only motion interference noise, and then removing the same type of motion interference noise from the target signal through cancellation.

[0007] However, the motion interference correction methods mentioned above for human activity recognition tasks in mobile scenarios only focus on individual sensor modalities. When the task to be recognized becomes more complex, the recognition performance of a single modality is often unsatisfactory. These methods ignore the complementary information and synchronization relationship between multiple modalities, and the problem of intermodal interference still exists.

[0008] Therefore, how to systematically eliminate motion interference at both intramodal and intermodal levels, so that multimodal human activity recognition systems can maintain high accuracy and robustness in mobile scenarios, has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0009] This invention addresses the motion interference problems faced by existing technologies in multimodal human activity recognition during robot movement, such as viewpoint shift, increased noise, and changes in background environment. It proposes a human activity recognition method and system for multimodal mobile perception scenarios that can eliminate motion interference during robot movement. The method utilizes cross-modal feature reconstruction to eliminate intramodal motion interference and, based on the concept of modal alignment, aligns vulnerable modalities to highly reliable modalities, thereby eliminating intermodal motion interference. This achieves accurate human activity recognition under motion interference conditions. It is easy to deploy, can achieve long-term continuous monitoring through robot active following, and can expand the range of recognized actions by leveraging the advantages of multimodal fusion.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] A method for human activity recognition in multimodal motion sensing scenarios includes the following stages:

[0012] Phase 1: Cross-modal feature reconstruction. After extracting features from data from multiple modalities, the feature reconstruction task uses the features of static data to reconstruct the features of dynamic data, thereby eliminating intramodal motion interference.

[0013] The second stage is reliable mode-oriented multimodal alignment. Based on the reliable mode with strong robustness to motion disturbance, the features reconstructed from other modes are aligned in feature space to eliminate motion disturbance between modes.

[0014] Finally, the multimodal features processed in the two stages were used for the human activity recognition task.

[0015] Furthermore, the first stage specifically includes:

[0016] Feature encoding is performed on the data of each modality to obtain the initial feature representation;

[0017] Attention weights between modalities are calculated using a cross-attention module, and multimodal information is used for weighted feature fusion to obtain a weighted feature representation.

[0018] A training batch containing dynamic and static data of the same category is constructed. The feature embedding of the dynamic data is aligned with the feature embedding of the corresponding static data through the feature reconstruction module. The reconstruction loss is calculated and the network parameters are updated with the constraint that the feature embedding of the static data remains unchanged before and after reconstruction.

[0019] Furthermore, the formula for calculating the reconstruction loss is as follows:

[0020]

[0021] in, Represents the i-th mode. This represents the data set corresponding to the k-th label. Characteristics representing the original static data, Features representing motion data after cross-attention For modality Feature reconstruction module, These are the balancing weighting coefficients.

[0022] Furthermore, the second stage specifically includes:

[0023] Inertial sensing data that is robust to motion disturbances is selected as the reliable mode for alignment. A pre-trained feature extractor is used to extract features from the inertial sensing data, and the parameters of the feature extractor are frozen.

[0024] Keeping the parameters of the feature reconstruction module unchanged after the first stage of training, the alignment loss based on the Gram matrix is ​​calculated together with the features of the reconstructed features of other modalities and the features of the inertial sensing data. The feature encoder parameters of other modalities are then updated so that the feature space distribution of each modality tends to be consistent with the reliable modality.

[0025] Furthermore, the formula for calculating the alignment loss based on the Gram matrix is ​​as follows:

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] in, A function to calculate the volume of a polygonal pyramid composed of multiple eigenvectors. Characteristics representing inertial sensing data, Representing the features of the other N modes, For a training batch of data, For temperature coefficient, The loss is the mapping from inertial modes to other modes. The loss for mapping other modes to inertial modes, This represents the final alignment loss.

[0032] Furthermore, the multimodal features processed in the two stages are used for human activity recognition tasks, specifically including:

[0033] The main parameters of the model after two-stage training are retained, while the feature extractors related to the inertial sensing data are removed.

[0034] Add a classification head adapted for downstream tasks after the reconstructed features of the desired modality;

[0035] The classification head is fine-tuned using partial data from downstream tasks to adapt the model to new human activity recognition tasks.

[0036] Furthermore, the method also includes a data preprocessing step: before inputting each modal data into the model, modality-specific motion interference cancellation methods are used to initially eliminate intra-modal interference; the data preprocessing step includes:

[0037] For depth video data captured by a depth camera, human region detection methods are used to identify and crop human regions, and then key video frames are selected based on the cumulative index of inter-frame difference distribution.

[0038] For point cloud data obtained by millimeter-wave radar, noise points are removed based on density clustering, and the point cloud clusters closest to the center of the viewpoint are selected and retained by combining distance and orientation information.

[0039] Secondly, the present invention provides a human activity recognition system for multimodal motion sensing scenarios, comprising the following modules to implement the method described in any of the above:

[0040] The cross-modal feature reconstruction module is used to extract features from data of multiple modalities and then use the features of static data to reconstruct the features of dynamic data through the feature reconstruction task, thereby eliminating motion interference within the modality.

[0041] A reliable modality-oriented multimodal alignment module is connected to the cross-modal feature reconstruction module. It is used to perform feature space alignment on features reconstructed from other modalities based on a reliable modality that is robust to motion disturbances, thereby eliminating motion disturbances between modalities.

[0042] The downstream task application module is connected to the multimodal alignment module and is used to apply the multimodal features after two-stage processing to the human activity recognition task.

[0043] Furthermore, the cross-modal feature reconstruction module includes:

[0044] The feature encoding unit is used to encode the features of data from each modality to obtain an initial feature representation;

[0045] The cross-attention unit is used to calculate the attention weights between different modalities through cross-attention, and to perform weighted feature fusion using multimodal information to obtain a weighted feature representation.

[0046] The feature reconstruction unit aligns the feature embeddings of dynamic data with the corresponding feature embeddings of static data using the feature reconstruction module. It calculates the reconstruction loss and updates the network parameters, assuming that the feature embeddings of the static data remain unchanged before and after reconstruction.

[0047] Furthermore, the reliable mode-guided multimodal alignment module includes:

[0048] The reliable modality processing unit selects inertial sensing data that is robust to motion disturbances as the aligned reliable modes, uses a pre-trained feature extractor to extract features from the inertial sensing data, and freezes the parameters of the feature extractor.

[0049] The alignment loss calculation unit is used to calculate the alignment loss based on the Gram matrix by combining the reconstructed features of other modes with the features of the inertial sensing data.

[0050] The parameter update unit is used to update the feature encoder parameters of other modes according to the alignment loss, so that the feature space distribution of each mode tends to be consistent with the reliable mode.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] Existing multimodal human activity recognition technologies suffer from severe performance degradation due to motion interference when sensing in mobile scenarios. In contrast, this invention addresses the motion interference problem in mobile scenarios by proposing a human activity recognition method for multimodal mobile sensing scenarios. It employs a two-stage pre-training process to progressively mitigate motion interference. First, cross-modal feature reconstruction reduces the difference in feature representations between dynamic and static data. Then, highly reliable modalities are introduced for modal alignment to enhance intermodal synchronization. This effectively reduces the impact of interference on feature representations under robot motion conditions, improving the model's robustness and generalization ability in dynamic environments. It combines the advantages of mobile scenarios and multimodal operation, offering greater transfer flexibility while also leveraging the strengths of multiple modalities to improve recognition accuracy.

[0053] Specifically, the method of the present invention reduces the difference between static data features and motion data features and eliminates intramodal interference by using a cross-modal feature reconstruction module. The reconstructed data features are then input into a reliable modality-oriented multimodal alignment module, which introduces an inertial sensing modality with high reliability in mobile scenarios. Through multimodal alignment training, the feature space of the susceptible modality is aligned to the high-reliability modality, eliminating intermodal motion interference and achieving accurate multimodal human activity recognition in mobile scenarios.

[0054] Furthermore, experimental results demonstrate that, under the interference conditions introduced by the two basic motion modes of robot forward and backward translation and left and right rotation, the recognition performance of this invention is close to the best recognition accuracy (80.6%) of existing methods in static scenes under the same settings. The recognition accuracy reaches 80% and 74.9% respectively under the two basic motion modes, which is an average improvement of 9.1% and 10.7% compared with existing multimodal human activity recognition methods, demonstrating a more accurate, stable and effective multimodal human activity recognition capability in mobile scenes. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0056] Figure 1 This is a schematic diagram of the overall architecture of the human activity recognition method for multimodal mobile sensing scenarios provided in an embodiment of the present invention. Detailed Implementation

[0057] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] The proposed method for human activity recognition in multimodal motion sensing scenarios has the following architecture: Figure 1 As shown, the key idea is to address motion interference from two perspectives: intramodal motion interference elimination and intermodal motion interference elimination. The feature reconstruction task eliminates the differences between dynamic and static data at the feature level within a modality. Then, the high-quality feature classification space of the reliable modality is used to align the feature space of the motion-disturbed modality, providing a more accurate and robust feature foundation for subsequent activity recognition.

[0059] The overall architecture can be divided into two core phases:

[0060] (1) Cross-modal feature reconstruction: After preprocessing, the data from different modalities are processed by their respective encoders to extract initial feature representations. Then, a cross-attention module is used to establish associations between different modalities to guide information complementarity. After that, the reconstruction module is responsible for extracting key motion information from the data and filtering out irrelevant noise motion information. In this process, the features of dynamic data and the corresponding static data features are used to calculate the learning process of the reconstruction loss constraint model, eliminating the motion differences between motion data features and static data features, thereby achieving the suppression of motion interference within the modality.

[0061] (2) Reliable modality-guided multimodal alignment: After feature reconstruction, the feature representation of a single modality has been significantly improved. However, the reconstruction correction of single-modal data inevitably impairs the synchronization between simultaneously acquired multimodal data. Therefore, to improve multimodal fusion performance, highly reliable modalities are needed to repair the correspondence between different modalities. Specifically, this stage utilizes inertial sensing devices that are easy to acquire and more robust to motion disturbances. A pre-trained feature encoder is used to extract the data features, and the Gram matrix-based alignment loss is calculated together with the features reconstructed from other modalities. This loss is then used to update the network parameters. Throughout the training process, the feature encoder of the inertial sensing data is frozen, and the feature space of other susceptible modalities is optimized. This promotes a more consistent structural distribution of different modalities in the feature space, effectively mitigating the distribution shift caused by motion disturbances and intramodal corrections, thereby eliminating intermodal motion disturbances.

[0062] The specific implementation details of the present invention are further described below.

[0063] This embodiment provides a method for human activity recognition in multimodal motion perception scenarios, including: data preprocessing, two-stage training (cross-modal feature reconstruction and reliable modality-guided multimodal alignment), and downstream task application.

[0064] (1) Data preprocessing

[0065] For each mode, the present invention first uses a mode-specific motion interference cancellation method in the data preprocessing stage to initially eliminate intra-mode interference.

[0066] Taking depth camera and millimeter-wave radar modes in robotic platform scenarios as an example:

[0067] For depth video data captured by a depth camera, this invention uses a human body region detection method to identify the human body and crop relevant regions. Specifically, a human detection algorithm based on the Mask_RCNN model of the detection2 package in Python is employed to detect human bounding boxes in each frame and crop out the region containing the human body, effectively removing background interference and reducing changes in the position of the human body region caused by robot movement. Then, a frame sampling method is used to select key video frames corresponding to quantiles based on the cumulative index of inter-frame difference distribution. Specifically, the differences in human body region images between adjacent frames are calculated to construct an inter-frame difference sequence; the cumulative distribution function of this sequence is calculated, and the 16 frames corresponding to the 16th quantile in the difference sequence are selected as key change frames, discarding redundant frames. This effectively reduces the differences in the position of the identified region in the depth video caused by sensor movement and the differences caused by different activity durations.

[0068] For point cloud data obtained from millimeter-wave radar, this invention first removes noise points with low distribution density based on the density clustering method (DBSCAN). Specifically, by setting the neighborhood radius ε and the minimum number of points MinPts, the point cloud is clustered into multiple clusters, and isolated noise points that do not belong to any cluster are discarded. Then, each point cloud cluster is filtered by combining distance and azimuth information, retaining only the point cloud cluster closest to the viewpoint center. Specifically, the coordinates of the center point of each point cloud cluster are calculated, and its distance and azimuth deviation from the viewpoint center (selected based on prior knowledge) are calculated. The point cloud cluster with the closest distance and the smallest azimuth deviation is selected as the effective target, and the remaining point cloud clusters are discarded as interference.

[0069] Through the above data preprocessing, some obvious motion interference was initially eliminated, laying the foundation for subsequent feature extraction and reconstruction.

[0070] (2) Cross-modal feature reconstruction

[0071] There are a total of Each mode, after data preprocessing, is... The data is The feature encoder can be used to extract features from each modality, which can be represented as:

[0072]

[0073] in Representing modes Feature encoder, For encoded modal The feature representation vector.

[0074] For example, N=3, including depth camera mode (RGB-D), millimeter-wave radar mode (Radar) and inertial sensing mode (IMU). Depth video data uses 3D CNN as feature encoder, millimeter-wave point cloud data uses 2D CNN as feature encoder, and inertial sensing data uses 1D CNN as feature encoder.

[0075] While data preprocessing can initially eliminate intramodal interference, it cannot eliminate the influence of motion interference on the Doppler velocity and intensity information carried by millimeter-wave point cloud data. It also introduces the problem of depth video image size stretching, resulting in significant differences between static and dynamic data at the feature level. Therefore, this invention allocates a separate feature reconstruction module for each modality, using feature reconstruction to reconstruct the corresponding static scene features from the data features in the motion scene, reducing the differences between the two types of data at the feature level.

[0076] Specifically, this invention first inputs the data into the cross-attention module, using multimodal information to assist the reconstruction task:

[0077]

[0078]

[0079] in This represents the cross-modal attention calculation process. For the calculated attention weights, For modality The weighted feature representation vector.

[0080] Then, a data reconstruction module based on a variational autoencoder (VAE) is used to reconstruct the feature representation of the dynamic data. Specifically, the training batches are constructed to ensure that they contain dynamic and static data of the same category. A reconstruction loss is used to align the feature embeddings of the dynamic data after passing through the data reconstruction module with the corresponding static data feature embeddings, while keeping the static data feature embeddings unchanged before and after passing through the feature reconstruction module. In this way, the present invention focuses the feature reconstruction module on reconstructing feature information destroyed by motion disturbances, ignoring the interference of changes in the static features themselves.

[0081] Assume the first The data set corresponding to each label is Then the loss function of the reconstruction process can be expressed as:

[0082]

[0083] Among them, symbols The symbols representing the characteristics of the original static data This represents the features of motion data after cross-attention. It is modal The feature reconstruction module ultimately yields the modal characteristics. No. The reconstruction loss for each label is λ is the balancing weight coefficient, which is set to 1 in the example.

[0084] Through such feature reconstruction tasks, intramodal motion interference can be further eliminated, and the differences in the representation of static and dynamic data at the feature level can be reduced, which can then be used for classification tasks in subsequent processes.

[0085] (3) Reliable mode-guided multimodal alignment

[0086] In multimodal scenarios, in addition to intramodal motion interference, intermodal motion interference also exists, which still negatively impacts multimodal fusion performance. This is because the synchronicity between modes is weakened, and the asynchrony problem is further exacerbated during the elimination of motion interference within a single mode.

[0087] To address this issue, this invention takes a modal alignment approach, introducing highly reliable modes to achieve feature space alignment of susceptible modes. Specifically, this invention selects easily collectable and readily available inertial sensing data (IMU) as the alignment target mode because it is unaffected by noise from robot motion and can be acquired from widely used devices such as smartwatches, smart rings, and mobile phones.

[0088] During training, the data preprocessing, feature encoder, and feature reconstruction module from the previous feature reconstruction stage are retained, while the parameters of the data reconstruction module are frozen. However, the parameters of the feature encoder remain trainable. This approach preserves the model's ability to eliminate intramodal motion disturbances while aligning the feature representation space with the inertial sensing data from a multimodal perspective. Furthermore, the inertial sensing data used is processed by a pre-trained and frozen feature extractor to extract features, preventing other susceptible modes from contaminating the high reliability of the inertial sensing data under motion disturbances.

[0089] Considering that at least three modalities need to be aligned, this invention uses a Gram matrix-based contrastive learning alignment loss, changing the distance calculation method between feature vectors from cosine distance to the volume of a pyramid composed of vectors, thus completing the feature space alignment of more modalities in one step and avoiding the knowledge forgetting problem caused by the multi-step alignment required by conventional contrastive learning loss.

[0090] Let the eigenvectors of each mode after reconstruction be represented as follows: The formula for calculating the loss function can be expressed as:

[0091]

[0092]

[0093]

[0094]

[0095]

[0096] in The formula for calculating the volume of a polygonal pyramid composed of multiple eigenvectors is as follows: This represents the features of inertial sensing data after passing through the feature extractor. This represents a dataset for a training batch. Calculate the loss of mapping inertial sensing modes to other modes. Calculate the loss for mapping other modes to the inertial sensing mode, and finally use the total alignment loss. express.

[0097] By minimizing The model aligns the feature spaces of other modes to the reliable inertial mode feature space, effectively eliminating motion interference between modes.

[0098] (4) Downstream task application

[0099] When applying the human activity model of this invention to a specific task, the main parameters are retained, the feature extractor related to the inertial sensing data is removed, and a classification head adapted to the task is added only after the reconstructed feature output of the required modality. Subsequently, the classification head is fine-tuned using partial data from downstream tasks, enabling the model to quickly adapt to new tasks and thus exhibit good multimodal human activity recognition performance under motion interference.

[0100] Specifically, this embodiment takes 10 basic human activity recognition activities (pushing, drawing a circle (counter-clockwise), raising a hand, knocking, walking, sitting down, standing up, picking up objects, throwing objects, and small jumps) as an example. After completing two-stage training, the parameters of the feature encoder, cross-attention module, and feature reconstruction module corresponding to the depth camera and millimeter-wave radar are retained, while the IMU feature extractor is removed. After reconstructing the feature output, a classification head consisting of two fully connected layers is added, with an output dimension of 10, corresponding to the 10 activity categories.

[0101] The classifier head is fine-tuned using a small amount of labeled data from downstream tasks (e.g., 15 samples per class), keeping the main parameters unchanged and only updating the parameters of the classifier head. After a few iterations of training, the model can achieve good recognition performance on the new task.

[0102] On the other hand, the present invention also provides a human activity recognition system for multimodal motion sensing scenarios, comprising:

[0103] The cross-modal feature reconstruction module is used to extract features from data of multiple modalities and then use the features of static data to reconstruct the features of dynamic data through the feature reconstruction task, thereby eliminating motion interference within the modality.

[0104] A reliable modality-oriented multimodal alignment module is connected to the cross-modal feature reconstruction module. It is used to perform feature space alignment on features reconstructed from other modalities based on a reliable modality that is robust to motion disturbances, thereby eliminating motion disturbances between modalities.

[0105] The downstream task application module is connected to the multimodal alignment module and is used to apply the multimodal features after two-stage processing to the human activity recognition task.

[0106] Furthermore, the cross-modal feature reconstruction module includes:

[0107] The feature encoding unit is used to encode the features of data from each modality to obtain an initial feature representation;

[0108] The cross-attention unit is used to calculate the attention weights between different modalities through cross-attention, and to perform weighted feature fusion using multimodal information to obtain a weighted feature representation.

[0109] The feature reconstruction unit is used to align the feature embeddings of dynamic data with the feature embeddings of corresponding static data through the feature reconstruction module, and calculate the reconstruction loss and update the network parameters with the constraint that the feature embeddings of static data remain unchanged before and after reconstruction.

[0110] Furthermore, the reliable mode-guided multimodal alignment module includes:

[0111] The reliable modality processing unit selects inertial sensing data that is robust to motion disturbances as the aligned reliable modes, uses a pre-trained feature extractor to extract features from the inertial sensing data, and freezes the parameters of the feature extractor.

[0112] The alignment loss calculation unit is used to calculate the alignment loss based on the Gram matrix by combining the reconstructed features of other modes with the features of the inertial sensing data.

[0113] The parameter update unit is used to update the feature encoder parameters of other modes according to the alignment loss, so that the feature space distribution of each mode tends to be consistent with the reliable mode.

[0114] Downstream task application modules include:

[0115] The model pruning unit is used to retain the main parameters of the model after two-stage training and remove the feature extractors related to the inertial sensing data.

[0116] The classification head adaptation unit is used to add a classification head adapted to downstream tasks after the reconstructed feature output of the required modality;

[0117] The fine-tuning training unit is used to fine-tune the classification head using partial data from downstream tasks, adapting the model to new human activity recognition tasks.

[0118] Compared with the prior art, the present invention has the following beneficial technical effects:

[0119] (1) Systematically solve motion interference problem. This invention addresses motion interference from both intramodal and intermodal perspectives. It eliminates the feature differences between dynamic and static data within a modality through cross-modal feature reconstruction, and repairs the synchronization between modalities through reliable modality-guided multimodal alignment, thus forming a complete motion interference elimination scheme and overcoming the problem that existing technologies only focus on one aspect.

[0120] (2) Innovative feature reconstruction mechanism. This invention designs a feature reconstruction module based on cross-attention and variational autoencoder, which uses multimodal information to assist reconstruction and aligns dynamic data features with static data features through reconstruction loss constraints, effectively reducing the difference in feature representation between motion state and static state and improving the robustness of single-modal features.

[0121] (3) Efficient multimodal alignment method. This invention uses inertial sensing data as a highly reliable modality and adopts a contrastive learning loss based on the Gram matrix. By calculating the volume of a pyramid composed of multiple feature vectors, the feature space alignment of multiple modalities is completed in one step, avoiding the knowledge forgetting problem caused by the multi-step alignment required by conventional contrastive learning and improving the synchronization between modalities.

[0122] (4) Excellent experimental results. Experimental results show that, under the interference conditions introduced by the two basic motion modes of robot forward and backward translation and left and right rotation, the recognition performance of the present invention is close to the best recognition accuracy of the existing method in static scenes (80.6%) under the same settings. The recognition accuracy reaches 80% and 74.9% respectively under the two basic motion modes, which is an average improvement of 9.1% and 10.7% compared with the existing multimodal human activity recognition methods, demonstrating a more accurate, stable and effective multimodal human activity recognition capability in mobile scenes.

[0123] (5) Good scalability. The present invention adopts a modular design, which can flexibly increase or decrease the number of modes according to the actual application scenario. When downstream tasks are applied, only the classification head needs to be finely adjusted to quickly adapt to new tasks, which has good practical value and promotion prospects.

[0124] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for human activity recognition in multimodal motion sensing scenarios, characterized in that, Includes the following stages: Phase 1: Cross-modal feature reconstruction. After extracting features from data from multiple modalities, the feature reconstruction task uses the features of static data to reconstruct the features of dynamic data, thereby eliminating intramodal motion interference. The second stage is reliable mode-oriented multimodal alignment. Based on the reliable mode with strong robustness to motion disturbance, the features reconstructed from other modes are aligned in feature space to eliminate motion disturbance between modes. Finally, the multimodal features processed in the two stages were used for the human activity recognition task.

2. The human activity recognition method for multimodal mobile sensing scenarios according to claim 1, characterized in that, The first stage specifically includes: Feature encoding is performed on the data of each modality to obtain the initial feature representation; Attention weights between modalities are calculated using a cross-attention module, and multimodal information is used for weighted feature fusion to obtain a weighted feature representation. A training batch containing dynamic and static data of the same category is constructed. The feature embedding of the dynamic data is aligned with the feature embedding of the corresponding static data through the feature reconstruction module. The reconstruction loss is calculated and the network parameters are updated with the constraint that the feature embedding of the static data remains unchanged before and after reconstruction.

3. The human activity recognition method for multimodal mobile sensing scenarios according to claim 2, characterized in that, The formula for calculating the reconstruction loss is: in, Represents the i-th mode. This represents the data set corresponding to the k-th label. Characteristics representing the original static data, Features representing motion data after cross-attention For modality Feature reconstruction module, These are the balancing weighting coefficients.

4. The human activity recognition method for multimodal mobile sensing scenarios according to claim 1, characterized in that, The second stage specifically includes: Inertial sensing data that is robust to motion disturbances is selected as the reliable mode for alignment. A pre-trained feature extractor is used to extract features from the inertial sensing data, and the parameters of the feature extractor are frozen. Keeping the parameters of the feature reconstruction module unchanged after the first stage of training, the alignment loss based on the Gram matrix is ​​calculated together with the features of the reconstructed features of other modalities and the features of the inertial sensing data. The feature encoder parameters of other modalities are then updated so that the feature space distribution of each modality tends to be consistent with the reliable modality.

5. The human activity recognition method for multimodal mobile sensing scenarios according to claim 4, characterized in that, The formula for calculating the alignment loss based on the Gram matrix is ​​as follows: in, A function to calculate the volume of a polygonal pyramid composed of multiple eigenvectors. Characteristics representing inertial sensing data, Representing the features of the other N modes, For a training batch of data, For temperature coefficient, The loss is the mapping from inertial modes to other modes. The loss for mapping other modes to inertial modes, This represents the final alignment loss.

6. The human activity recognition method for multimodal mobile sensing scenarios according to claim 1, characterized in that, The multimodal features processed in two stages are then used for human activity recognition tasks, specifically including: The main parameters of the model after two-stage training are retained, while the feature extractors related to the inertial sensing data are removed. Add a classification head adapted for downstream tasks after the reconstructed features of the desired modality; The classification head is fine-tuned using partial data from downstream tasks to adapt the model to new human activity recognition tasks.

7. The human activity recognition method for multimodal mobile sensing scenarios according to claim 1, characterized in that, The method also includes a data preprocessing step: before each modal data is input into the model, modality-specific motion interference cancellation methods are used to initially eliminate intramodal interference; Data preprocessing steps include: For depth video data captured by a depth camera, human region detection methods are used to identify and crop human regions, and then key video frames are selected based on the cumulative index of inter-frame difference distribution. For point cloud data obtained by millimeter-wave radar, noise points are removed based on density clustering, and the point cloud clusters closest to the center of the viewpoint are selected and retained by combining distance and orientation information.

8. A human activity recognition system for multimodal motion sensing scenarios, characterized in that, Includes the following modules to implement the method as described in any one of claims 1 to 7: The cross-modal feature reconstruction module is used to extract features from data of multiple modalities and then use the features of static data to reconstruct the features of dynamic data through the feature reconstruction task, thereby eliminating motion interference within the modality. A reliable modality-oriented multimodal alignment module is connected to the cross-modal feature reconstruction module. It is used to perform feature space alignment on features reconstructed from other modalities based on a reliable modality that is robust to motion disturbances, thereby eliminating motion disturbances between modalities. The downstream task application module is connected to the multimodal alignment module and is used to apply the multimodal features after two-stage processing to the human activity recognition task.

9. The human activity recognition system for multimodal mobile sensing scenarios according to claim 8, characterized in that, The cross-modal feature reconstruction module includes: The feature encoding unit is used to encode the features of data from each modality to obtain an initial feature representation; The cross-attention unit is used to calculate the attention weights between different modalities through cross-attention, and to perform weighted feature fusion using multimodal information to obtain a weighted feature representation. The feature reconstruction unit is used to align the feature embeddings of dynamic data with the feature embeddings of corresponding static data through the feature reconstruction module, and calculate the reconstruction loss and update the network parameters with the constraint that the feature embeddings of static data remain unchanged before and after reconstruction.

10. The human activity recognition system for multimodal mobile sensing scenarios according to claim 8, characterized in that, The reliable mode-guided multimodal alignment module includes: The reliable modality processing unit selects inertial sensing data that is robust to motion disturbances as the aligned reliable modes, uses a pre-trained feature extractor to extract features from the inertial sensing data, and freezes the parameters of the feature extractor. The alignment loss calculation unit is used to calculate the alignment loss based on the Gram matrix by combining the reconstructed features of other modes with the features of the inertial sensing data. The parameter update unit is used to update the feature encoder parameters of other modes according to the alignment loss, so that the feature space distribution of each mode tends to be consistent with the reliable mode.

Images