Edge-cloud collaborative human activity recognition modeling method based on heterogeneous multi-modal data

Abstract

The application discloses a kind of edge cloud cooperation human activity recognition modeling methods based on heterogeneous multi-modal data, and is divided into two stages of centralized pre-training and multi-modal semi-supervised fine-tuning.In the first stage, by dynamic mask contrast learning, pre-train the basic model based on converter using the joint dataset with super-class information, so that it has the ability to extract robust features from any modal combination.In the second stage, the local unlabeled data is used by the end-side client to generate pseudo-labels through weak mask view and calculate losses based on strong mask view for local update;Cloud side aggregates heterogeneous model parameters from different clients, and fine-tunes the global model using a small amount of labeled data.The application effectively alleviates the data heterogeneity problem by separating single-modal feature encoding and cross-modal information fusion, and can efficiently train a high-performance multi-modal human activity recognition model using a small amount of labeled data and unlabeled data while protecting data privacy.

Description

Technical Field

[0001] This invention relates to the field of Human Activity Recognition (HAR) technology, and in particular to a human activity recognition modeling method based on edge-cloud collaboration of heterogeneous multimodal data. Background Technology

[0002] Human activity recognition plays a vital role in daily life. Monitoring specific movements through wearable devices and wireless sensing devices helps identify dangerous behaviors such as falls in the elderly and allows for appropriate interventions. It can also detect early biometric features of chronic diseases like Alzheimer's and Parkinson's. Furthermore, human activity recognition is applied in motion analysis, helping athletes analyze training movements and optimize performance. In addition, human activity recognition technology can be used in human-computer interaction, enabling touchless control and improving user experience by recognizing user actions or gestures. In real-world scenarios, human activity is complex and dynamic. Therefore, achieving accurate human activity recognition by integrating sensing data from different modalities (accelerometers, gyroscopes, cameras, millimeter-wave radar, and WiFi, etc.) has significant application value.

[0003] With the development of deep learning technology and the expansion of pre-training datasets, the "pre-training + fine-tuning" model framework has been widely used in the field of human activity recognition. This means that a multimodal human activity recognition model is first pre-trained on a large dataset in the cloud, and then fine-tuned on downstream data from the target user group, thereby reducing migration costs and improving accuracy on downstream tasks.

[0004] However, due to the distribution of downstream data across the target user group, data heterogeneity still exists within it. For example, some households may be unwilling to install RGB cameras or depth cameras due to privacy or price concerns, and different users may have different behavioral patterns and habits. This requires the model to maintain high accuracy under inputs with arbitrary modal combinations. Furthermore, the scarcity of labeled data remains a persistent problem in real-world scenarios. Acquiring labeled multimodal data is costly, and most data from target users is completely unlabeled and stored locally. Only a small amount of labeled data from test users is stored on the cloud side.

[0005] Some studies use federated learning methods, allowing different downstream users to use perceptual data of different modalities. However, these methods require that the local data of downstream users must be labeled. Some studies utilize self-supervised learning to enable models to be effectively pre-trained using unlabeled data; however, these methods can only handle fixed and limited perceptual modalities and cannot adapt to variations in different modal combinations, thus severely limiting their availability of pre-training data. Summary of the Invention

[0006] The purpose of this invention is to propose a human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data. This method pre-trains the human activity recognition model on a joint dataset using dynamic mask contrastive learning, enhancing the model's generalization ability under arbitrary modality combinations and enabling it to handle heterogeneous multimodal data. By employing edge-cloud collaboration for multimodal semi-supervised collaborative fine-tuning, the model can autonomously adapt using local heterogeneous and unlabeled data. This effectively solves the problems of sensor modality mismatch and high manual annotation costs in real-world scenarios while protecting privacy.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data includes a centralized pre-training stage and a multimodal semi-supervised collaborative fine-tuning stage.

[0009] The centralized pre-training phase is executed on the cloud side and includes:

[0010] The transformer-based base model is pre-trained using a joint dataset containing superclass information;

[0011] The pre-training employs a dynamic mask contrastive learning method, which constructs different views of multimodal inputs by applying different masking strategies, and uses the superclass information for supervised contrastive learning, enabling the base model to learn to extract robust and consistent feature vectors from arbitrary modal combinations.

[0012] The multimodal semi-supervised collaborative fine-tuning stage is executed in an edge-cloud collaborative environment and is used to adapt the pre-trained base model to the target user group of downstream tasks. This stage includes:

[0013] On the client side, the client uses local unlabeled multimodal data and adopts the same masking strategy as in the pre-training stage to generate weak masked views and strong masked views; it uses the weak masked views to generate pseudo-labels and calculates the loss based on the prediction results of the pseudo-labels and the strong masked views to update the local model parameters.

[0014] On the cloud side, heterogeneous model parameters from different clients are aggregated to update the global model, and a small amount of labeled data is used to fine-tune the updated global model.

[0015] Furthermore, the centralized pre-training phase specifically includes:

[0016] Constructing a multimodal pre-trained dataset: By reorganizing human activity categories, multiple publicly available multimodal human activity datasets are integrated to create a joint dataset with the superclass information;

[0017] Building the basic model architecture: The basic model includes a modal encoder, a position encoder, a modal token, and a converter block;

[0018] The modal encoder encodes raw data from different modalities into feature vectors of a unified dimension; the positional encoder injects temporal information into the model; the modal token is a learnable vector used to distinguish different modalities and enhance the model's ability to learn relationships between modalities; and the converter block captures temporal correlations and cross-modal relationships.

[0019] Furthermore, the dynamic mask contrast learning method includes:

[0020] Different masking strategies are applied to the feature vector F input to the converter block to generate different contrast feature vectors;

[0021] The masking strategy includes:

[0022] Strong masking: Randomly mask the feature vectors in each modality, repeating the process twice to generate two strong masked feature vectors for comparison. and ;

[0023] Weak masking: Randomly select a subset of modalities, mask all feature vectors within the selected modalities, while ensuring that at least one modality remains unmasked, to generate a weakly masked contrast feature vector. ;

[0024] The comparison feature vector , and The input is fed into the converter block and processed by the mapping head to obtain the corresponding output vector. , , ;

[0025] Using the superclass information, a supervised contrastive loss function is employed to bring feature vectors belonging to the same superclass closer together in the representation space, and to push feature vectors belonging to different superclasses apart, in order to train the base model.

[0026] Furthermore, in the strong masking strategy, for the selected feature vector, it is replaced with a learnable mask token with a first probability, replaced with a random feature vector of the same modality with a second probability, or kept unchanged with a third probability; the strong masking strategy adopts a mask ratio greater than a preset threshold to ensure that the contrast feature vectors of different views have sufficient distinctiveness.

[0027] Furthermore, in the strong masking strategy, for the selected feature vector, it is replaced with a learnable mask token with a first probability, replaced with a random feature vector of the same modality with a second probability, or kept unchanged with a third probability; the strong masking strategy adopts a mask ratio greater than a preset threshold to ensure that the contrast feature vectors of different views have sufficient distinctiveness.

[0028] Furthermore, the multimodal semi-supervised collaborative fine-tuning stage specifically includes:

[0029] In each round of communication, the cloud side randomly selects a portion of clients as active clients and distributes the current global model parameters to the active clients;

[0030] On the client side, based on local unlabeled data, a first view is generated using the same weak masking strategy as in the pre-training phase. The model's prediction results are then obtained based on this first view. When the maximum confidence of the prediction results exceeds a preset threshold... When this prediction result is used as a pseudo-label ;

[0031] Using the same strong masking strategy as in the pre-training phase, a strong mask is applied to the local unlabeled data to generate a second view. The model prediction results based on the second view are then compared with the pseudo-labels. Calculate cross-entropy loss To update local model parameters;

[0032] After local training is complete, the active client sends the updated local model parameters to the cloud.

[0033] Furthermore, the cloud side aggregates the received local model parameters, specifically including:

[0034] For each modality in the global model, its corresponding modality encoder and modality token are aggregated respectively;

[0035] For non-heterogeneous parts with the same architecture in the global model, aggregation is performed directly;

[0036] If a certain modality does not appear in any of the active clients in the current communication round, then the model parameters corresponding to that modality remain unchanged.

[0037] Furthermore, the cloud side utilizes a small amount of labeled dataset. Fine-tuning the aggregated global model includes:

[0038] Apply a weak mask to the cloud-side labeled data to obtain the third view;

[0039] Model prediction results based on the third view and the true labels Calculate the monitoring loss To update global model parameters;

[0040] The fine-tuned and updated global model is distributed to active clients in the next round of communication, and the above process is repeated until the model converges or the preset number of communication rounds is reached.

[0041] Furthermore, in the multimodal semi-supervised collaborative fine-tuning stage, single-modal feature encoding and cross-modal information fusion are processed separately; wherein, single-modal feature encoding is completed independently by the modal encoder, and the parameters of the modal encoder are aggregated separately according to the modality; cross-modal information fusion is completed by the converter block of the shared architecture, and the parameters of the converter block are aggregated among all clients.

[0042] Secondly, the present invention provides a multimodal semi-supervised human activity recognition system based on edge-cloud collaboration, including a cloud server and multiple edge clients, the system being used to execute the method described in any of the above-mentioned embodiments.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] While some existing studies can effectively use unlabeled data for pre-training, they can only handle fixed modal inputs and are not suitable for situations where modal input combinations change dynamically in real-world scenarios. Some studies use federated learning methods, allowing downstream users to use perceptual data from different modalities. However, these methods require that the local data of downstream users must be labeled.

[0045] In comparison, the technical solution of this application is more closely related to real-world scenarios, solving the difficulties in implementation caused by sensor modality mismatch and the high cost of manual annotation. When facing highly heterogeneous multimodal human activity recognition data, the dynamic mask contrastive learning method can enhance the model's generalization ability under different modality combinations. The edge-cloud collaborative multimodal semi-supervised collaborative fine-tuning algorithm overcomes the traditional limitation of requiring a large amount of labeled data, supporting efficient fine-tuning on unlabeled heterogeneous data, reducing weight bias caused by data modality heterogeneity, and improving the model fine-tuning effect. The heterogeneous multimodal model aggregation algorithm can effectively integrate client model weights from different modalities, especially for efficient information fusion between edge devices and the cloud. This technology solves the challenges posed by modality mismatch and data heterogeneity in the real world and improves the model's application capability in multimodal data environments. Experiments demonstrate that the model constructed by the method of this invention achieves significant performance improvements on different datasets (as shown in Table 1). For example, with only 1, 3, and 5 training samples per category, the model improved accuracy by 39.4%, 35.5%, and 33.0%, respectively.

[0046] Table 1

[0047] Dataset Name Modal Number of participants Number of activity categories MRI Accelerometer, gyroscope, skeletal points 20 11 UTD Accelerometer, gyroscope, millimeter-wave radar, skeletal points 8 27 PFHAR Accelerometer, gyroscope, millimeter-wave radar, skeletal points, depth camera 30 15 Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0049] Figure 1 This is a schematic diagram of the overall architecture of the edge-cloud collaborative human activity recognition and modeling method based on heterogeneous multimodal data provided in an embodiment of the present invention.

[0050] Figure 2 The overall workflow of the pre-training stage provided in the embodiments of the present invention. Detailed Implementation

[0051] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0052] This invention first proposes a pre-training method for dynamic mask contrastive learning. Unlike existing pre-training methods that use fixed modalities and tasks on a single dataset, dynamic mask contrastive learning supports highly heterogeneous multimodal human activity recognition data with arbitrary modalities, positions, and other variations. It constructs different views for samples using two different masking strategies. By combining superclass information from multiple datasets to compute supervised contrastive loss, the model can accept any combination of modalities as input and learn complementary and consistent information between modalities.

[0053] In the fine-tuning stage for downstream tasks, this invention proposes a multimodal semi-supervised fine-tuning algorithm based on edge-cloud collaboration and a heterogeneous multimodal model aggregation algorithm. This allows the target user's local data to be modally heterogeneous and unlabeled, with only a small amount of labeled data from test users stored on the cloud side. At the same time, it separates feature extraction within each modality from cross-modal feature fusion, thereby reducing the weight bias introduced by different modal combinations on the client and improving the performance of the edge-cloud collaborative fine-tuning stage.

[0054] This invention aims to improve the practical application capabilities of multimodal human activity recognition systems, while protecting user privacy and promoting the development of applications in fields such as smart health and smart homes.

[0055] The proposed method for human activity recognition and modeling based on edge-cloud collaboration using heterogeneous multimodal data has the following overall architecture: Figure 1As shown, it is divided into two stages: centralized pre-training and multimodal semi-supervised collaborative fine-tuning.

[0056] (1) Centralized Pre-training Stage: This stage is conducted on the cloud side. The goal of this invention is to construct a base model with generalization capabilities, capable of extracting feature vectors with representational value from any data modality. First, this invention integrates multiple publicly available multimodal human activity datasets by reorganizing human activity categories, thereby creating a multimodal pre-training dataset (joint dataset) with superclass information. Based on this joint dataset, this invention proposes a converter-based base model architecture, including modality encoders, positional encoders, modality tokens, and converter blocks, to capture the temporal relationships between different modalities. Subsequently, a novel multimodal pre-training method—dynamic mask contrastive learning—is used to pre-train the base model on the joint dataset. Unlike traditional contrastive learning methods or mask reconstruction methods limited to a single modality, the method of this invention utilizes the inherent physical characteristics of different modalities and constructs different views of multimodal inputs by applying two different masking strategies. Combining the superclass information in the joint dataset, the base model has the ability to extract robust and consistent feature vectors from any combination of modalities.

[0057] (2) Multimodal Semi-Supervised Collaborative Fine-Tuning Stage Based on Edge-Cloud Collaboration: The purpose of this stage is to adapt the pre-trained base model to the target user group in the downstream task. This stage allows the local data of the client to be modally heterogeneous and unlabeled, requiring only a small amount of labeled data provided by the cloud-side test users. During edge-side training, a masking strategy similar to that of the pre-training stage is used to generate weak and strong masked views. Subsequently, the weak masked view is used to generate pseudo-labels for the downstream human activity recognition task, and cross-entropy loss is calculated between the pseudo-labels and the prediction results of the strong masked view. In addition, on the cloud side, this invention proposes an innovative model aggregation method to aggregate heterogeneous model weights from different modal clients. The small amount of labeled data on the cloud side is used to train the global base model, thereby enhancing its ability to handle various data modality combinations. By separating single-modal feature extraction from cross-modal information fusion, the model weight bias caused by data heterogeneity can be significantly reduced.

[0058] The following section further elaborates on the specific implementation details of the key modules:

[0059] (1) Intensive pre-training phase

[0060] 1) Modal encoder

[0061] In order to effectively capture the features of different sensing modes, such as Figure 2As shown, this invention employs different feature encoders to encode data from different modalities into a unified dimension. These modality encoders are time-independent, allowing each data modality to encode multiple consecutive time frames separately. The modality encoder can be any available time-independent deep learning model. For example, this invention uses 1-DCNN layers for accelerometer and gyroscope data, PointNet++ for LiDAR data, and MobileNetV2 for depth image data, etc.

[0062] 2) Location coding

[0063] For human activity recognition tasks, the sequence of actions is crucial (for example, the signal features of "standing up" and "sitting down" are similar, but the order is reversed). Position encoding tells the model where in the sequence the signal occurs. Figure 2 As shown, position encoding is added This invention uses standard non-learnable sine and cosine one-dimensional position coding. Instead of adding the same position information to the sensor feature vector for each modality, this invention uses a global set of position codes. To better capture the unique information of each modality.

[0064] 3) Modal tokens

[0065] like Figure 2 As shown, in the case of Add to Furthermore, this invention introduces a learnable modality token for each modality. .For example, The shape is Then it was expanded and added to In the middle. In the future and Add to Subsequently, the present invention obtained These are used in subsequent converter blocks. The purpose of adding modality tokens is to distinguish different modalities and enhance the model's ability to learn the relationships between modalities.

[0066] 4) Converter block

[0067] The purpose of using a converter block in this invention is to fully utilize temporal correlation and cross-modal relationships. Assume the input to the converter block is... ,but It will pass through the converter block in sequence. For converter blocks , The layer-normalized version (LN) is passed sequentially through a multi-head self-attention (MSA) block and a multilayer perceptron (MLP). A residual connection is applied after each MSA block or MLP block. It is a converter block The output, and used as a converter block. The input can be calculated as follows:

[0068]

[0069]

[0070] in This represents the number of converter blocks.

[0071] For the output of the last converter block, this invention takes the average value along the time frame dimension to obtain the final output, which has the following shape: This invention avoids using the widely adopted CLS token output as the final output, as it may lead to instability during the training process.

[0072] 5) Dynamic mask comparison learning

[0073] Unlike existing contrastive learning-based methods, which rely on the raw input data... The present invention applies powerful data augmentation techniques to generate different training views. The method of this invention applies the input feature vector of the converter block. Two different masking strategies are applied to create different training views.

[0074] The two masking strategies are: (i) Strong mask: a random mask for each modality, which randomly masks a fixed proportion of the feature vector in each modality. This process is performed twice. This invention considers this type of mask to be a strong mask (because information from all modalities is partially lost), generating two contrasting feature vectors of strong masks. and (ii) Weak masking: This is a random masking of complete modalities. A random number of modalities are selected, and all feature vectors in these modalities are masked, while the remaining modalities remain unmasked (at least one modality remains unmasked). This type of masking is weak (because information from some modalities is still complete), generating a contrast feature vector for the weak mask. .

[0075] For strong masks, this invention employs a similar method. If a feature vector is chosen as the mask, this invention will use a learnable mask token with an 80% probability. Replace it with a random feature vector of the same modality, or keep the feature vector unchanged, with a 10% probability. Unlike word feature vectors, due to the temporal continuity of sensor data... Adjacent feature vectors in a map can be very similar. To ensure sufficient difference between contrasting feature vectors from different views, this invention increases the masking ratio (e.g., to 80%) in a strong masking strategy. In any masking strategy, Zero-padded feature vectors will be ignored and not considered as available modes.

[0076] Unlike existing mask-based reconstruction methods, this invention employs a supervised contrastive loss with superclass information from a joint dataset to train the base model, enabling it to extract general features from any combination of modalities. This is used in generating contrastive feature vectors. , and Then, they are fed into the converter block. and mapping header ,in , , Represents the mapping header Regularize the output vector. For example... Figure 2 As shown, the goal of dynamic masked contrastive learning is to bring positive features (within the same superclass) closer together while separating negative features (within different superclasses). During inference, no masking strategy is applied. It will be directly fed into the converter block. .

[0077] In dynamic masked contrastive learning, data in the same mini-batch comes from different datasets with different modalities. By applying a weak masking strategy, the modalities of samples in each dataset change dynamically in each mini-batch. This allows all possible modality combinations to appear in the same mini-batch, thus bringing them closer together or separating them through supervised contrastive loss. To enhance the robustness of feature extraction, a strong masking strategy creates two additional views for joint loss computation. Models trained in this way are better able to understand temporal relationships and the interconnections between modalities.

[0078] (2) Multimodal semi-supervised collaborative fine-tuning stage based on edge-cloud collaboration

[0079] In obtaining a pre-trained multimodal base model Subsequently, this invention utilizes multimodal semi-supervised collaborative fine-tuning to perform edge-cloud collaborative fine-tuning on target user groups (clients) that possess heterogeneous and unlabeled multimodal data. This invention first defines the collaborative fine-tuning problem and introduces the multimodal semi-supervised collaborative fine-tuning process on both the edge and cloud sides.

[0080] 1) Problem Definition

[0081] Assuming that the present invention has during the collaborative fine-tuning process One client is available, and makes Indicates the client The available unlabeled datasets. Similarly, let This represents a small dataset of labeled data collected from a few beta users, which can be obtained on the cloud side. This invention assumes that the modalities available to each client may differ. Let... This represents the parameters of the cloud-side global model, which are initialized to... The parameters are then added to the output of the converter block, along with a linear header. The parameters of the client model are expressed as follows: This invention assumes that the client model may have different modal encoders and modal tokens. The goal of multimodal semi-supervised collaborative fine-tuning is to fine-tune... In order to use in the edge-cloud collaborative mode and Perform classification tasks.

[0082] 2) Multimodal semi-supervised collaborative fine-tuning at the edge

[0083] In each round of communication, the cloud side first randomly selects a portion of clients as active clients and distributes global parameters. This includes the corresponding modality encoder for each available modality on each client. During local training on each client, unlabeled multimodal data... First, a modal encoder is used, then position encoding and modal tokens are added to generate... Then, a pseudo-label is generated for each input sample. This pseudo-label is subsequently used to calculate the standard cross-entropy loss. The pseudo-label can be obtained as follows:

[0084]

[0085] in yes The weak masked version, It is the output of the last linear layer of the model, after applying softmax. This is a predefined hyperparameter representing the threshold for retaining pseudo-labels. Then, this invention uses pseudo-labels to calculate the cross-entropy loss and applies it... The strong masked version, denoted as :

[0086]

[0087] in This represents the cross-entropy loss. The total loss is the average of all input data. The masking strategy used in local training is the same as in pre-training, generating different views. However, the strong mask is only applied once.

[0088] Unlike centralized semi-supervised learning methods, which include both labeled and unlabeled data in mini-batches, the client only has unlabeled data. Therefore, only... It is used to update local parameters. After each round of local training, the local parameters of each active client are sent to the cloud for aggregation.

[0089] 3) Cloud-based model aggregation

[0090] After collecting local parameters on the cloud side, this invention first aggregates them to update... Then, this invention uses a small amount of labeled dataset. This is used to fine-tune the global model. When aggregating local parameters, the heterogeneous components are the modal encoder and modal token. For each modality in the global model, this invention aggregates the modal encoder and modal token separately using FedAVG. If a modality is not included in all active clients in the current communication round, the corresponding modal encoder and modal token will not be updated. Other local parameters (parts with the same architecture, such as mask tokens, converter blocks, and linear headers) are directly aggregated using FedAVG.

[0091] 4) Cloud-based multimodal semi-supervised collaborative fine-tuning

[0092] After model aggregation, Will be used on the cloud side Fine-tuning was performed using supervised loss. Unlike local training, this time only... Apply a weak mask to calculate the supervised loss:

[0093]

[0094] Once the cloud-side fine-tuning is complete, the updated It will be sent to the active client for the next round of communication. This process will repeat until the global model converges on the target user group data, or reaches the predefined number of communication rounds. After multimodal semi-supervised collaborative fine-tuning, all masking strategies will be removed, and the global model can be directly used for the target user group.

[0095] In multimodal semi-supervised collaborative fine-tuning, the encoding of modal features and the fusion of information between modalities are explicitly separated. The feature encoding of each modality is minimally affected by the encodings of other modalities. This is because the modality encoder performs only time-independent feature mapping and is trained separately from the dynamic combination of modalities. The modality encoder is aggregated for each individual modality, while transformer blocks sharing the same architecture are aggregated across all clients to perform modality fusion. In this way, the data heterogeneity problem among target user groups can be significantly mitigated.

[0096] In summary, this invention first designs a dynamic masked contrastive learning pre-training method, which supports pre-training of highly heterogeneous multimodal human activity recognition data under various modalities, locations, and other variations. By employing two different masking strategies—strong and weak masking—different views are constructed for the samples, and contrastive loss is calculated using superclass information from multiple datasets, enabling the model to accept any combination of modalities as input and possessing strong generalization ability. Secondly, this invention also designs a multimodal semi-supervised collaborative fine-tuning algorithm based on edge-cloud collaboration. Through the collaborative work of edge devices and the cloud, the model can be effectively fine-tuned on heterogeneous unlabeled data of the target user group. This algorithm first generates pseudo-labels on the client side using a weak masked view and calculates cross-entropy loss, then performs local training using unlabeled data. On the cloud side, a small amount of labeled data is used to further fine-tune the global model, and the performance of the global model is optimized by aggregating model parameters from different clients. In this way, the algorithm can reduce weight bias caused by different modality combinations, improve the model's adaptability to various modality combinations, and thus achieve higher accuracy and robustness in the case of unlabeled data, making it particularly suitable for multimodal data and heterogeneous user groups in real-world scenarios. Furthermore, this invention also designs a heterogeneous multimodal model aggregation algorithm to address the problem of how to effectively integrate model parameters from different modal clients between edge devices and the cloud. In practical applications, different clients may have different data modalities; for example, some clients may only use accelerometers and gyroscopes, while others may also include depth cameras or millimeter-wave radar. This algorithm uses techniques such as FedAVG to separately aggregate client models from different modalities, ensuring that the encoder and modality token for each modality can be updated independently, thereby reducing weight bias caused by differences in data modalities. This aggregation algorithm effectively improves the global model's adaptability to multimodal data combinations and enhances the model's robustness and performance when processing heterogeneous data.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for human activity recognition and modeling based on edge-cloud collaboration using heterogeneous multimodal data, characterized in that, This includes a centralized pre-training phase and a multimodal semi-supervised collaborative fine-tuning phase; The centralized pre-training phase is executed on the cloud side and includes: The transformer-based base model is pre-trained using a joint dataset containing superclass information; The pre-training employs a dynamic mask contrastive learning method, which constructs different views of multimodal inputs by applying different masking strategies, and uses the superclass information for supervised contrastive learning, enabling the base model to learn to extract robust and consistent feature vectors from arbitrary modal combinations. The multimodal semi-supervised collaborative fine-tuning stage is executed in an edge-cloud collaborative environment and is used to adapt the pre-trained base model to the target user group of downstream tasks. This stage includes: On the client side, the client uses local unlabeled multimodal data and adopts the same masking strategy as in the pre-training stage to generate weak masked views and strong masked views; it uses the weak masked views to generate pseudo-labels and calculates the loss based on the prediction results of the pseudo-labels and the strong masked views to update the local model parameters. On the cloud side, heterogeneous model parameters from different clients are aggregated to update the global model, and a small amount of labeled data is used to fine-tune the updated global model.

2. The human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data according to claim 1, characterized in that, The centralized pre-training phase specifically includes: Constructing a multimodal pre-trained dataset: By reorganizing human activity categories, multiple publicly available multimodal human activity datasets are integrated to create a joint dataset with the superclass information; Building the basic model architecture: The basic model includes a modal encoder, a position encoder, a modal token, and a converter block; The modal encoder encodes raw data from different modalities into feature vectors of a unified dimension; the positional encoder injects temporal information into the model; the modal token is a learnable vector used to distinguish different modalities and enhance the model's ability to learn relationships between modalities; and the converter block captures temporal correlations and cross-modal relationships.

3. The human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data according to claim 2, characterized in that, The dynamic mask contrast learning method includes: Different masking strategies are applied to the feature vector F input to the converter block to generate different contrast feature vectors; The masking strategy includes: Strong masking: Randomly mask the feature vectors in each modality, repeating the process twice to generate two strong masked feature vectors for comparison. and ; Weak masking: Randomly select a subset of modalities, mask all feature vectors within the selected modalities, while ensuring that at least one modality remains unmasked, to generate a weakly masked contrast feature vector. ; The comparison feature vector , and The input is fed into the converter block and processed by the mapping head to obtain the corresponding output vector. , , ; Using the superclass information, a supervised contrastive loss function is employed to bring feature vectors belonging to the same superclass closer together in the representation space, and to push feature vectors belonging to different superclasses apart, in order to train the base model.

4. The human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data according to claim 3, characterized in that, In the strong masking strategy, for the selected feature vector, it is replaced with a learnable mask token with a first probability, replaced with a random feature vector of the same modality with a second probability, or kept unchanged with a third probability. The strong masking strategy uses a masking ratio greater than a preset threshold to ensure that the contrast feature vectors of different views have sufficient distinctiveness.

5. The human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data according to claim 2, characterized in that, In the strong masking strategy, for the selected feature vector, it is replaced with a learnable mask token with a first probability, replaced with a random feature vector of the same modality with a second probability, or kept unchanged with a third probability. The strong masking strategy uses a masking ratio greater than a preset threshold to ensure that the contrast feature vectors of different views have sufficient distinctiveness.

6. The human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data according to claim 1, characterized in that, The multimodal semi-supervised collaborative fine-tuning stage specifically includes: In each round of communication, the cloud side randomly selects a portion of clients as active clients and distributes the current global model parameters to the active clients; On the client side, based on local unlabeled data, a first view is generated using the same weak masking strategy as in the pre-training phase. The model's prediction results are then obtained based on this first view. When the maximum confidence of the prediction results exceeds a preset threshold... When this prediction result is used as a pseudo-label ; Using the same strong masking strategy as in the pre-training phase, a strong mask is applied to the local unlabeled data to generate a second view. The model prediction results based on the second view are then compared with the pseudo-labels. Calculate cross-entropy loss To update local model parameters; After local training is complete, the active client sends the updated local model parameters to the cloud.

7. The human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data according to claim 1, characterized in that, The cloud side aggregates the received local model parameters, specifically including: For each modality in the global model, its corresponding modality encoder and modality token are aggregated respectively; For non-heterogeneous parts with the same architecture in the global model, aggregation is performed directly; If a certain modality does not appear in any of the active clients in the current communication round, then the model parameters corresponding to that modality remain unchanged.

8. The human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data according to claim 1, characterized in that, The cloud side utilizes a small amount of labeled datasets. Fine-tuning the aggregated global model includes: Apply a weak mask to the cloud-side labeled data to obtain the third view; Model prediction results based on the third view and the true labels Calculate the monitoring loss To update global model parameters; The fine-tuned and updated global model is distributed to active clients in the next round of communication, and the above process is repeated until the model converges or the preset number of communication rounds is reached.

9. The human activity recognition modeling method based on edge-cloud collaboration using heterogeneous multimodal data according to claim 1, characterized in that, In the multimodal semi-supervised collaborative fine-tuning stage, single-modal feature encoding and cross-modal information fusion are processed separately. Specifically, single-modal feature encoding is completed independently by the modal encoder, and the parameters of the modal encoder are aggregated separately according to the modality. Cross-modal information fusion is completed by the converter block of the shared architecture, and the parameters of the converter block are aggregated among all clients.

10. A multimodal semi-supervised human activity recognition system based on edge-cloud collaboration, comprising a cloud server and multiple edge clients, characterized in that, The system is used to perform the method according to any one of claims 1 to 9.

Images