Human body activity identification method based on multi-modal perception data

Abstract

A human body activity identification method based on multi-modal perception data belongs to the field of human body activity identification, and comprises the following steps: aligning each modal data to the same representation space through consistency and complementarity learning of each modal data so as to realize the universality of modal combination; then, a small number of labeled samples are regarded as anchor points in different distributions, heterogeneous representation vectors crossing scenes, wearing positions and devices are aligned through the anchor points, and therefore the model can adapt to various scenes, wearing positions and devices. According to the method, unified, generalized and efficient human body activity identification can be realized, input of any modal combination can be adapted to meet diversified perception task requirements, and relatively high robustness is shown under the conditions of scene, wearing position, equipment change and the like.

Description

Technical Field

[0001] This invention belongs to the field of human activity recognition technology, specifically relating to a human activity recognition method based on multimodal perception data. Background Technology

[0002] Human activity recognition plays a vital role in people's daily lives, especially in the healthcare field. Monitoring specific movements through wearable devices and wireless sensing devices helps assess cardiac function levels for early warning of heart failure, identify dangerous behaviors such as falls and take appropriate measures, and promptly detect biometrics in the early stages of chronic diseases such as Alzheimer's or Parkinson's. Furthermore, human activity recognition is also valuable in social safety governance. For example, by continuously recording and analyzing individual movement logs, it helps to identify abnormal behavior, trace incidents, and provide risk warnings, thereby improving public safety capabilities. In real-world scenarios, human activity is complex and dynamic. Therefore, multimodal perception has become an important paradigm for motion monitoring, achieving accurate perception by fusing data from different sources, such as accelerometers (ACC), gyroscopes, cameras, millimeter-wave radar, and WiFi.

[0003] However, existing activity recognition methods are primarily designed for fixed modal data from specific types of sensors at fixed locations. This makes them ill-suited to diverse real-world scenarios, resulting in model fragmentation. For example, monitoring a person's activities throughout the day may involve switching between indoor and outdoor environments, leading to a switch in available sensors—that is, a switch in perception modality. Furthermore, training activity recognition models in existing methods relies heavily on large amounts of labeled data. Even though some research has begun to focus on few-shot learning, it still requires a relatively high proportion of labeled data (approximately a hundred data points per action category) to achieve accuracy comparable to supervised learning methods. This severely limits the practical application of activity recognition technology.

[0004] Training activity recognition models using a pre-trained, jointly fine-tuned architecture based on a base model can effectively enhance the model's adaptability to scene changes and reduce the amount of labeled data required for training. However, existing base model research is mostly focused on text-based and image-based models, which cannot be directly applied to the perceptual data domain. This is partly because most existing base models rely on text modalities for semantic matching between modalities, an approach that performs poorly on non-intuitive perceptual data; and partly because the inherent heterogeneity among perceptual datasets hinders the acquisition of large amounts of high-quality data. Summary of the Invention

[0005] The purpose of this invention is to provide a human activity recognition method based on multimodal perception data, which addresses the model fragmentation problem caused by the insufficient adaptability of existing activity recognition methods to dynamic changes in the perception scene, thereby achieving unified, generalized, and efficient human activity recognition. This invention employs a mask self-supervised pre-training strategy, which can adapt to inputs with arbitrary modal combinations to meet diverse perception task requirements. Simultaneously, it introduces a few-sample alignment strategy, exhibiting strong robustness under conditions of scene, wearing position, and device variations.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] This invention provides a method for human activity recognition based on multimodal sensing data, which specifically includes the following steps:

[0008] In the self-supervised pre-training stage, the feature encoders of each modality are used to extract the high-dimensional representation vector sequence corresponding to each modality; by randomly masking some representation vector segments in the high-dimensional representation vector sequence, the masked representation vector segments are restored using the Transformer model; the consistency and complementarity information between the data of each modality are learned through the recovery loss function based on the idea of ​​contrastive learning, and the difference between the representation vector segments before and after restoration is minimized while maximizing the discriminability between different representation vector segments during the learning process;

[0009] In the few-shot alignment and fine-tuning stage, the feature encoders and Transformer models of each modality are initialized with the parameters trained in the self-supervised pre-training stage, and these parameters are frozen. An adaptation layer is introduced and the samples belonging to the same behavior category are aligned through the alignment loss function. By maximizing the similarity between samples within a class while minimizing the similarity between samples between classes, samples from the same behavior category share the same classification boundary. An iterative fine-tuning mechanism is used to fine-tune the pre-trained feature encoder, Transformer model, adaptation layer, and newly added classification layer to obtain a well-trained human behavior recognition model.

[0010] Furthermore, in the self-supervised pre-training phase, an unlabeled multimodal pre-training dataset needs to be created by merging publicly available multimodal datasets.

[0011] Furthermore, before randomly masking some fragments of the high-dimensional representation vector sequence, the high-dimensional representation vector sequence is projected to the same dimension through a projection layer, and the outputs of each modality are concatenated into a representation vector sequence. The sequence is then filled with zero representation vectors to the maximum length, ensuring that the length of the representation vector sequence remains consistent.

[0012] Furthermore, when extracting the high-dimensional representation vector sequence corresponding to each modality, the input sequence of each modality is first divided into small input segments according to time, and then these small input segments are sent into the feature encoder to extract the representation vector segments; then, positional information and modal information are added to the representation vector segments through positional encoding and modal encoding, and at the same time, a CLS token is added to the sequence header of the entire representation vector segment.

[0013] Furthermore, the mathematical expression for the Transformer module is:

[0014]

[0015] In this system, MSA stands for Multi-head Attention Layer, LN for Normalization Layer, and MLP for Multilayer Perceptron. The intermediate results of the l-th layer of the Transformer model are used for subsequent MLP computation, H l H is the output of the l-th layer of the Transformer model. l-1 This is the output of layer l-1 of the Transformer model.

[0016] Furthermore, in the self-supervised pre-training phase, 50% of the representation vector segments in the original representation vector sequence are selected first, and these representation vector segments are replaced with learnable mask representations, representation vector segments at other positions are randomly copied, or these representation vector segments are kept unchanged.

[0017] Furthermore, the mathematical expression for the recovery loss function is:

[0018]

[0019] in, The set of indices of the representation vector fragments that are masked. Let be the set of indices of all representation vector segments within a training batch, where i is the index of the currently masked representation vector segment, and j is the index of the j-th representation vector segment among all representation vector segments excluding the masked representation vector segment at index i. For two representation vector fragments and Cosine similarity between them The representation vector fragment before restoration. for The mask is the representation vector fragment recovered by the Transformer model, K is the set of the masked representation vector fragments, and τ is the temperature coefficient.

[0020] Furthermore, the few-sample alignment stage includes the following steps:

[0021] S2.2.1: By aligning the loss function Minimize the distance between CLS tokens of the same category to align all input modal sequences from different scenarios, wearing locations, and devices to the same classification boundary;

[0022] S2.2.2: Minimize the distance between CLS tokens of the same behavior category across different modal input combinations while maintaining the distance between CLS tokens of all modalities.

[0023] Furthermore, in step S2.2.1, the alignment loss function The mathematical expression is:

[0024]

[0025] in, Let s be a set of CLS tokens of the same behavior category in the same batch, where i is the index of the current CLS token, q is the index of the q-th CLS token in the set of CLS tokens excluding CLS token i, and s(CLS j ,CLS i Let s(CLS token i) be the cosine similarity between CLS token i and CLS token j. q ,CLS i Let q be the cosine similarity between CLS token q and CLS token j.

[0026] Furthermore, in step S2.2.2, the CLS that has been aligned in the previous stage is utilized. t The token serves as an anchor to ensure that all input modal sequences in the current phase correspond to the CLS. a The token remains consistent and acts as the CLS corresponding to the dynamic modal input combination sequence in the modal input combination set C. i The alignment target of the tokens is determined using the alignment loss function. To learn:

[0027]

[0028] in, Let MSE be the set of all modal combinations, β be the root mean square error, and β be the scaling parameter that emphasizes preserving the performance of the full-modal CLS token classification.

[0029] The beneficial effects of this invention are:

[0030] Existing activity recognition models can only handle fixed modal inputs and are not suitable for situations where modal input combinations change dynamically in real-world scenarios. Furthermore, the performance of existing activity recognition models is severely limited by the amount of labeled data. In contrast, the human activity recognition method based on multimodal perception data provided in this invention has application scenarios closer to reality. Through a mask learning module, the model adapts to dynamic modal input combinations, allowing for joint training with multiple activity recognition datasets. Relying on few-sample alignment, the model quickly adjusts its adaptation to different scenarios, wearing positions, and devices. It achieves an average accuracy of 92.2% across seven test datasets using approximately 10 labeled data points per behavior category and a large amount of unlabeled data for training, thus realizing more accurate and effective human behavior recognition.

[0031] This invention provides a human activity recognition method based on multimodal perception data, applicable to, but not limited to, various mainstream perception modalities and their arbitrary combinations, such as accelerometers, gyroscopes, millimeter-wave radar, skeletal points, WiFi, RGB cameras, infrared cameras, and LiDAR. Faced with changes in new scenes, devices, or locations, only a very small number of fine-tuning samples (e.g., 1–5 samples per class) are needed to achieve the recognition performance of existing small models that rely on tens of thousands of supervised samples for fine-tuning. Attached Figure Description

[0032] Figure 1 The flowchart of a human activity recognition method based on multimodal perception data provided by the present invention is shown.

[0033] Figure 2 This is the encoding process for high-dimensional representation vector sequences. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to the accompanying drawings.

[0035] This invention provides a human activity recognition method based on multimodal perception data, which can handle arbitrary modal combination inputs and maintain high recognition accuracy even with few samples. First, drawing inspiration from the knowledge completion principle of the human brain, a mask learning alignment framework is designed. This involves learning the consistency and complementarity of various modal data to align them to the same representation space, achieving the universality of modal combinations. Then, a small number of labeled samples are treated as anchor points with different distributions. These anchor points are used to align heterogeneous representation vectors across scenes, wearing positions, and devices, enabling the model to adapt to various scenes, wearing positions, and devices.

[0036] See Figure 1 As shown, the human activity recognition method based on multimodal perception data provided by this invention can be divided into two stages:

[0037] (1) Self-supervised pre-training phase;

[0038] This invention proposes a mask-based self-supervised pre-training strategy for perceptual data, enabling the model to adapt to dynamically combined multimodal inputs during the self-supervised pre-training stage and possessing the ability to train on multiple human activity recognition datasets. The specific implementation process is as follows:

[0039] S1.1: First, an unlabeled multimodal pre-training dataset is created by merging publicly available multimodal datasets. These publicly available multimodal datasets, such as dataset 1, dataset 2, ..., dataset n, can originate from accelerometers (ACC), gyroscopes (Gyro), cameras, millimeter-wave radar, and WiFi, etc.

[0040] S1.2: Then, each modality's independent feature encoder is used to extract a high-dimensional representation vector sequence for each modality. After passing through an internal projection layer, the sequence is projected to the same dimension. The outputs of each modality are then concatenated into a representation vector sequence and filled with zero representation vectors to the maximum length, ensuring that the length of the representation vector sequence remains consistent regardless of the number of input modalities.

[0041] The overall process of transforming raw data input into a high-dimensional representation vector sequence is as follows: Figure 2 As shown, the input sequence for each modality is segmented into smaller input segments over time, and these segments are then fed into a feature encoder to extract representation vector segments. Positional and modal information encodings are then added to these representation vector segments to provide temporal and modal information to the model, respectively. To learn the representation of the entire high-dimensional representation vector sequence, an additional learnable classification token (CLS) is added to the sequence header of each representation vector segment. The CLS token plays a crucial role in the subsequent few-shot alignment and fine-tuning stages.

[0042] Specifically, it is assumed that the present invention may have a total of m modal inputs. Each time a dataset is used for training, the model receives a subset {M} of all modalities. a ,…,M k To address the dynamic changes in modal input combinations, this invention uses the input sequence x∈R for each modality. D×L Divide into n segments based on time. (Modal M) a (corresponding input sequence) and (Modal M) k The corresponding input sequence), where D is the dimension of the input mode and L is the length of the input mode. The n segments are fed into mode M respectively. a The corresponding feature encoder and mode M k The corresponding feature encoder extracts representation vector fragments. as well as The mathematical expression representing the vector segment extraction process is as follows:

[0043]

[0044] in, For the extracted mode M i The corresponding representation vector fragment, For mode M i The corresponding feature extractor, For mode M i The input sequence corresponds to n segments.

[0045] Based on this, add sine and cosine position embedding. and modal embedding The input sequence for the Transformer model is as follows:

[0046]

[0047] Wherein, CLS stands for learnable classification token. For the extracted mode M a The corresponding representation vector fragment, For the extracted mode M k The corresponding representation vector fragment.

[0048] S1.3: Finally, a mask learning module was designed to randomly mask high-dimensional representation vector sequences. This invention utilizes a Transformer model to recover masked representation vector fragments. It employs a recovery loss function based on contrastive learning to learn consistency and complementarity information between different modal data, minimizing the difference between the recovered and unrecovered representation vector fragments while maximizing their discriminability. This multimodal mask learning method performs mask learning on the high-dimensional vectors obtained after feature extraction from each modality, avoiding the dimensionality differences of the original data for each modality in multimodal scenarios. Furthermore, the contrastive learning-based recovery loss function proposed in this invention, compared to traditional mask recovery loss functions, allows the model to simultaneously focus on intermodal consistency and complementarity information.

[0049] To make the model more stable during training on multiple datasets, this invention adopts a post-normalization and deep normalization architecture because the pre-normalization architecture encounters convergence problems during training as the number of Transformer model layers increases.

[0050] Specifically, the mathematical expression for the Transformer module is as follows:

[0051]

[0052] In this system, MSA stands for Multi-head Attention Layer, LN for Normalization Layer, and MLP for Multilayer Perceptron. The intermediate results of the l-th layer of the Transformer model are used for subsequent MLP computation, H l H is the output of the l-th layer of the Transformer model. l-1 This is the output of layer l-1 of the Transformer model.

[0053] This invention employs a mask learning module for self-supervised pre-training. It selects 50% of the representation vector segments from the original representation vector sequence and either replaces these segments with learnable mask representations (80%), randomly copies representation vector segments from other positions (10%), or leaves them unchanged (10%). Then, a recovery loss function based on contrastive learning is designed to simultaneously learn the consistency and complementarity information between multimodal sensing data. The main idea of ​​recovery is to utilize information from other modalities to reconstruct the masked representation vector segments, while simultaneously, based on contrastive learning, minimizing the consistency of representation vector segments within the same class while maximizing the difference from other representation vector segments in the same batch.

[0054] Specifically, the mathematical expression for the recovery loss function is as follows:

[0055]

[0056] in, The set of indices of the representation vector fragments that are masked. Let be the set of indices of all representation vector segments within a training batch, where i is the index of the currently masked representation vector segment, and j is the index of the j-th representation vector segment among all representation vector segments excluding the masked representation vector segment at index i. For two representation vector fragments and Cosine similarity between them The representation vector fragment before restoration. for The masked representation vector fragments are recovered using the Transformer model, where K is the set of masked representation vector fragments, and τ is the temperature coefficient. The recovered mode M... a The corresponding representation vector segment is represented as The recovered mode M k The corresponding representation vector segment is represented as

[0057] (2) Few-sample alignment and fine-tuning stage;

[0058] To enable models to quickly adapt to new scenarios, wearing positions, and devices using only a small amount of labeled multimodal data, this invention proposes a few-shot alignment method to enhance the pre-trained model's ability to adapt across modal combinations, scenarios, device positions, and device types, thereby improving the accuracy of human activity recognition models in few-shot scenarios. This proposed few-shot alignment method not only effectively preserves the general knowledge gained during the self-supervised pre-training phase but also significantly improves the model's adaptability to changes in different scenarios, positions, and devices, thus achieving higher accuracy and stronger generalization in human activity recognition. The few-shot alignment method consists of two parts: first, it aligns the differences in scenarios, device positions, and device types through supervised contrastive learning; then, it improves the model performance under various modal combinations by aligning the token CLS of missing modalities while maintaining the CLS distance of the full-modal classification tokens.

[0059] The specific implementation process of the few-shot alignment method is as follows:

[0060] S2.1: First, initialize the feature encoder and Transformer model for each modality using the parameters trained in the self-supervised pre-training phase, and freeze these parameters.

[0061] S2.2: Next, an adaptation layer is introduced, and samples belonging to the same behavior category are aligned using an alignment loss function. By maximizing the similarity between samples within a class while minimizing the similarity between samples between classes, samples from the same behavior category share the same classification boundary. The purpose of few-sample alignment is to enable the model to quickly adapt to new scenarios, wearing positions, and devices with limited labeled data. In this way, the aligned model not only reduces fine-tuning overhead but also enhances knowledge transfer capabilities.

[0062] The fundamental reason for poor performance when directly applying models to new scenarios, wearing positions, and devices is that changes in data distribution cause the original classification boundary to deviate from the target domain data. To address this, this invention introduces a few-shot adaptation layer to align representation vector sequences of the same behavioral category from different scenarios, wearing positions, and devices to the same classification boundary. Specifically, the feature encoder and Transformer model are initialized using model weights from the self-supervised pre-training phase, extracting the raw input data into a representation vector sequence. However, since the modal input combination may change dynamically, the length of the representation vector sequence also varies. To overcome this, this invention uses CLS tokens for alignment and fine-tuning. Because CLS tokens are initialized without specific semantic information, they can integrate the semantic information of other representation vector fragments in the multimodal input sequence, thereby better representing the semantics of the entire modal input combination.

[0063] The few-shot alignment stage specifically includes the following steps:

[0064] S2.2.1: By aligning the loss function Minimize the distance between CLS tokens of the same category to align all input modal sequences from different scenarios, wearing locations, and devices to the same classification boundary.

[0065] Specifically, the mathematical expression for the alignment loss function is as follows:

[0066]

[0067] in, Let s be a set of CLS tokens of the same behavior category in the same batch, where i is the index of the current CLS token, q is the index of the q-th CLS token in the set of CLS tokens excluding CLS token i, and s(CLS j ,CLS i Let s(CLS token i) be the cosine similarity between CLS token i and CLS token j. q ,CLS i Let q be the cosine similarity between CLS token q and CLS token j.

[0068] S2.2.2: While maintaining the distance between CLS tokens across all modalities, minimize the distance between CLS tokens of the same behavior category across different modal input combinations. This helps improve the model's ability to transfer knowledge between modalities and enhances its ability to recover from missing modalities. To achieve this, the invention utilizes CLS tokens aligned in the previous stage. t The token serves as an anchor to ensure that all input modal sequences in the current phase correspond to the CLS. a The token remains consistent and acts as the CLS corresponding to the dynamic modal input combination sequence in the modal input combination set C. i The token alignment target is specifically achieved using the following alignment loss function. To learn:

[0069]

[0070] in, Let MSE be the set of all modal combinations, β be the root mean square error, and β be the scaling parameter that emphasizes preserving the performance of the full-modal CLS token classification.

[0071] By aligning a small number of labeled samples, CLS tokens from different modalities, scenarios, wearing locations, and devices can share a more consistent data distribution. Even with only 10 labeled data points per behavior category, the few-sample alignment method can effectively align samples within a behavior category.

[0072] S2.3: Finally, an iterative fine-tuning mechanism is used to fine-tune the pre-trained feature encoder, Transformer model, adaptation layer, and newly added classification layer, adapting the model to the specific task through the iterative fine-tuning process.

[0073] After completing the above training, similar to the large text and image models, the CLS tokens can be classified using a linear classification layer to output the classification results for human behavior recognition. However, since the classification layer is randomly initialized, significant gradient changes occur during the fine-tuning phase, leading to knowledge forgetting in the already trained model components. Therefore, this invention employs an iterative fine-tuning method. Initially, the feature encoder, Transformer layer, and adaptation layer are frozen, and the classification layer is trained separately. After N rounds of iterative fine-tuning, the feature encoder, Transformer layer, and adaptation layer are trained in reverse N rounds, and the classification layer is frozen again. This process is repeated until the preset upper limit of the number of training rounds is reached, ultimately obtaining a trained human behavior recognition model.

[0074] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for human activity recognition based on multimodal sensing data, characterized in that, Includes the following steps: In the self-supervised pre-training stage, the feature encoders of each modality are used to extract the high-dimensional representation vector sequence corresponding to each modality; by randomly masking some representation vector segments in the high-dimensional representation vector sequence, the masked representation vector segments are restored using the Transformer model; the consistency and complementarity information between the data of each modality are learned through the recovery loss function based on the idea of ​​contrastive learning, and the difference between the representation vector segments before and after restoration is minimized while maximizing the discriminability between different representation vector segments during the learning process; In the few-shot alignment and fine-tuning stage, the feature encoders and Transformer models of each modality are initialized with the parameters trained in the self-supervised pre-training stage, and these parameters are frozen; an adaptation layer is introduced and the samples belonging to the same behavior category are aligned through the alignment loss function. By maximizing the similarity between samples within the class while minimizing the similarity between samples between classes, samples from the same behavior category share the same classification boundary. An iterative fine-tuning mechanism is used to fine-tune the pre-trained feature encoder, Transformer model, adaptation layer, and newly added classification layer to obtain a well-trained human behavior recognition model.

2. The method for human activity recognition based on multimodal perception data according to claim 1, characterized in that, In the self-supervised pre-training phase, an unlabeled multimodal pre-training dataset needs to be created by merging publicly available multimodal datasets.

3. The method for human activity recognition based on multimodal perception data according to claim 1, characterized in that, Before randomly masking some fragments of the high-dimensional representation vector sequence, the high-dimensional representation vector sequence is projected to the same dimension through a projection layer. The outputs of each mode are then concatenated into a representation vector sequence, which is then padded with zero representation vectors to the maximum length to ensure that the length of the representation vector sequence remains consistent.

4. The method for human activity recognition based on multimodal perception data according to claim 1, characterized in that, When extracting the high-dimensional representation vector sequence corresponding to each modality, the input sequence of each modality is first divided into small input segments according to time, and then these small input segments are sent into the feature encoder to extract the representation vector segments; then, positional information and modal information are added to the representation vector segments through positional encoding and modal encoding, and a CLS token is added to the sequence header of the entire representation vector segment.

5. The method for human activity recognition based on multimodal perception data according to claim 1, characterized in that, The mathematical expression for the Transformer module is: In this system, MSA stands for Multi-head Attention Layer, LN for Normalization Layer, and MLP for Multilayer Perceptron. The intermediate results of the l-th layer of the Transformer model are used for subsequent MLP computation, H l H is the output of the l-th layer of the Transformer model. l-1 This is the output of layer l-1 of the Transformer model.

6. The method for human activity recognition based on multimodal perception data according to claim 1, characterized in that, In the self-supervised pre-training phase, 50% of the representation vector segments in the original representation vector sequence are selected first. These representation vector segments are then replaced with learnable mask representations, representation vector segments at other positions are randomly copied, or these representation vector segments are left unchanged.

7. The method for human activity recognition based on multimodal perception data according to claim 1, characterized in that, The mathematical expression for the recovery loss function is: in, The set of indices of the representation vector fragments that are masked. Let be the set of indices of all representation vector segments within a training batch, where i is the index of the currently masked representation vector segment, and j is the index of the j-th representation vector segment among all representation vector segments excluding the masked representation vector segment at index i. For two representation vector fragments and Cosine similarity between them The representation vector fragment before restoration. for The mask is the representation vector fragment recovered by the Transformer model, K is the set of the masked representation vector fragments, and τ is the temperature coefficient.

8. The method for human activity recognition based on multimodal perception data according to claim 1, characterized in that, The few-sample alignment stage includes the following steps: S2.2.1: By aligning the loss function Minimize the distance between CLS tokens of the same category to align all input modal sequences from different scenarios, wearing locations, and devices to the same classification boundary; S2.2.2: Minimize the distance between CLS tokens of the same behavior category across different modal input combinations while maintaining the distance between CLS tokens of all modalities.

9. A method for human activity recognition based on multimodal perception data according to claim 8, characterized in that, The alignment loss function The mathematical expression is: in, Let s be a set of CLS tokens of the same behavior category in the same batch, where i is the index of the current CLS token, q is the index of the q-th CLS token in the set of CLS tokens excluding CLS token i, and s(CLS j ,CLS i Let s(CLS token i) be the cosine similarity between CLS token i and CLS token j. q ,CLS i Let q be the cosine similarity between CLS token q and CLS token j.

10. A method for human activity recognition based on multimodal perception data according to claim 8, characterized in that, In step S2.2.2, the CLS that has been aligned in the previous stage is utilized. t The token serves as an anchor to ensure that all input modal sequences in the current phase correspond to the CLS. a The token remains consistent and acts as the CLS corresponding to the dynamic modal input combination sequence in the modal input combination set C. i The alignment target of the tokens is determined using the alignment loss function. To learn: in, Let MSE be the set of all modal combinations, β be the root mean square error, and β be the scaling parameter that emphasizes preserving the performance of the full-modal CLS token classification.

Images