Imu signal cross-modal semantic understanding and generation method for llm

Abstract

A method for cross-modal semantic understanding and generation of IMU signals for LLM (Limited Language Model) is proposed. In the offline multimodal alignment stage, a pre-trained visual language model is used to generate motion-related text descriptions from videos synchronized with the IMU signals. Multi-granular features are extracted through temporal fusion, and an IMU encoder is trained using coarse-grained inter-sample alignment and fine-grained intra-sample alignment strategies to achieve initial alignment of IMU signals and text in a shared semantic space. In the online retrieval and enhancement generation stage, features are extracted from newly input IMU signals using the pre-trained IMU encoder, and similar signal-text pairs are retrieved from a pre-constructed IMU-text pairing RAG database. A structural similarity graph between IMU signals is constructed, and a central node and its neighbors are selected to form an anchor point-cluster structure. This structure, along with the retrieved text descriptions, is input into a large language model to generate high-quality text descriptions consistent with the semantics of the IMU signals. This invention enables a large language model to generate high-quality text descriptions consistent with motion semantics, even under the constraint of imprecise alignment between the IMU and text, thus improving its perceptual performance in tasks such as activity recognition, behavior analysis, and human-computer interaction.

Description

Technical Field

[0001] This invention relates to a technology in the field of Large Language Models (LLMs), specifically a method for cross-modal semantic understanding and generation of inertial measurement unit (IMU) signals for large language models. Background Technology

[0002] Existing Large Language Models (LLMs) face the following challenges in applying natural language descriptions to IMU signals: First, IMU-text pairing data is extremely scarce because manually annotating IMU signals with high-quality, diverse text descriptions is costly. Second, there is significant modal heterogeneity and semantic granularity mismatch between IMU signals and text. IMU signals represent the dynamic characteristics of motion execution (such as vibration and acceleration), belonging to low-level physical modes; while textual language carries semantic concepts of motion tasks (such as "chopping vegetables" and "running"), belonging to high-level abstract descriptions. Directly applying traditional cross-modal alignment methods (such as CLIP-based contrastive learning) often yields limited results and struggles to establish accurate semantic mappings. Furthermore, in real-world deployments, due to sensor noise, individual differences, and motion complexity, the mapping relationship between IMU and text is inherently fuzzy and uncertain, leading to "imprecise alignment." Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies, such as being limited to video and text modalities and being unable to understand temporal sensing signals. It proposes a cross-modal semantic understanding and generation method for IMU signals oriented towards LLM, which enables large language models to generate high-quality text descriptions consistent with motion semantics under the constraint of imprecise alignment between IMU and text, thereby improving their perception performance in tasks such as activity recognition, behavior analysis, and human-computer interaction.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a method for cross-modal semantic understanding and generation of IMU signals for LLM, comprising:

[0006] 1) Offline multimodal alignment stage: Using a pre-trained visual language model, motion-related text descriptions are generated from videos synchronized with IMU signals; multi-granular features are extracted through temporal fusion, and IMU encoders are trained by combining coarse-grained inter-sample alignment and fine-grained intra-sample alignment strategies to achieve preliminary alignment of IMU signals and text in a shared semantic space.

[0007] The training of the IMU encoder adopts a hierarchical contrastive learning strategy anchored by video and text: at the coarse-grained level, semantic alignment between samples is achieved through global contrastive loss; at the fine-grained level, local contrastive loss is calculated based on temporal segment division to achieve temporal alignment within samples.

[0008] 2) Online retrieval and enhancement generation stage: For newly input IMU signals, features are extracted using a pre-trained IMU encoder, and similar signal-text pairs are retrieved from the pre-built IMU-text pairing RAG database; a structural similarity graph between IMU signals is constructed, and the central node and its neighbors are selected to form an anchor point-clustering structure, which is input into the large language model together with the retrieved text description to generate a high-quality text description that is consistent with the semantics of the IMU signals.

[0009] The construction of the structural similarity graph between IMU signals is specifically as follows: for the retrieved IMU signal samples, the similarity between nodes is calculated based on cosine similarity and dynamic time warping distance weighting. Nodes with similarity exceeding a preset threshold are connected, and the node with the highest connectivity is selected as the center node, which together with its one-hop neighbors forms an anchor point-clustering structure.

[0010] This invention relates to an LLM-oriented IMU signal cross-modal semantic understanding and generation system for implementing the above-mentioned method, comprising: a text description unit, a training unit, a retrieval unit, and a description generation unit, wherein: the text description unit generates motion-related text descriptions based on video information synchronized with the IMU signals using a pre-trained visual language model, thus obtaining text descriptions; the training unit extracts multi-granularity features through temporal fusion based on the IMU signals and the text description information output by the text description unit, and trains the IMU encoder by combining coarse-grained inter-sample alignment and fine-grained intra-sample alignment strategies, thus obtaining a trained IMU encoder and an IMU-text pairing RAG database; the retrieval unit extracts features based on newly input IMU signals and the IMU encoder and database information output by the training unit, and retrieves similar signal-text pairs from the database, thus obtaining retrieval results; the description generation unit constructs a structural similarity graph between IMU signals based on the retrieval results output by the retrieval unit, selects the central node and its neighbors to form an anchor point-clustering structure, and inputs it together with the retrieved text descriptions into a large language model to generate text descriptions consistent with the semantics of the IMU signals, thus obtaining the final result.

[0011] Technical effect

[0012] This invention utilizes video as a bridge to automatically generate motion-related text through a pre-trained visual language model, achieving large-scale, low-cost data construction and avoiding tedious manual annotation. Through a retrieval enhancement generation mechanism based on IMU structural relationships, it dynamically utilizes the semantic information of similar samples to compensate for and correct initial alignment deficiencies, significantly improving the semantic accuracy and contextual relevance of the generated text. Compared with existing technologies, this invention can simultaneously model global semantics and local temporal correspondences, effectively alleviating alignment difficulties caused by modal heterogeneity and granularity mismatch. It performs excellently in zero-shot and few-shot activity recognition and cross-scene tasks. Only IMU signals are required during the inference stage, eliminating the need for video input, thus protecting user privacy and making it suitable for resource-constrained wearable devices and other practical scenarios. Beyond IMU signals, its "offline alignment + online retrieval enhancement" approach can be extended to other low-interpretability sensor signal and language fusion tasks. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the invention;

[0014] Figure 2 A schematic diagram illustrating the construction and text generation of prompts using video as a bridge;

[0015] Figure 3 This is a schematic diagram of the timing fusion module.

[0016] Figure 4 A schematic diagram illustrating the construction of the IMU structural similarity graph and text generation during the enhanced generation stage for online retrieval. Detailed Implementation

[0017] like Figure 1 As shown in this embodiment, a method for cross-modal semantic understanding and generation of IMU signals for LLM is provided, including:

[0018] Step 1: Generate motion text based on video bridging and cueing engineering. Specifically, input the visual context consisting of video frame sequences, high-level action semantic labels, and metadata representing the shooting perspective into a pre-trained visual language big model. Use text cueing to constrain and guide the model generation process so that the text description output by the visual language big model will effectively remove visual semantics unrelated to the IMU signal.

[0019] In this embodiment, the pre-trained visual language large-scale model is the Qwen2.5-VL-32B-Instruct-AWQ model. This model is a visual language multimodal large-scale model open-sourced by Alibaba's Tongyi Qianwen team in March 2025. It adopts 4-bit AWQ quantization technology and can run smoothly on a single consumer-grade GPU. Its core features include: support for dynamic resolution and frame rate processing, capable of processing long videos exceeding 1 hour and accurately locating key events; employing multimodal rotational position coding (MRoPE) to achieve unified spatiotemporal position modeling of text, images, and videos; and possessing fine-grained visual analysis capabilities, capable of efficiently generating motion-related text descriptions synchronized with IMU signals in video.

[0020] The text prompts explicitly require the model to ignore static backgrounds, irrelevant objects, and complex scene semantics, and instead focus on human motion dynamics that can be captured by inertial sensors.

[0021] For example: "Analyze the video captured by the head-mounted device, and describe the movement of the person's head and torso based solely on changes in camera height and orientation." The visual language model outputs text generation such as "The person is continuously jumping up and down" or "The person is walking by alternating arm swings in a stable rhythm." These sentences emphasize the physical execution of the movement, thus forming a potential semantic correspondence with the acceleration and angular velocity patterns contained in the IMU signal, providing high-quality "pseudo-label" text data for subsequent cross-modal alignment.

[0022] Step 2, Multimodal temporal feature fusion and structured representation, specifically includes:

[0023] 2.1 Single-modal feature encoding: The IMU signal sequence, video frame sequence, and generated text sequence in the training data are set as follows: , , And features are extracted using the corresponding encoder, specifically: For the IMU encoder to be trained, its parameters Random initialization; and These are pre-trained video encoders and text encoders, respectively, with parameters... Frozen during training. All output features. ,in For batch size, For time step, For feature dimensions.

[0024] 2.2 Time Segmentation Compression and Fusion: Combining the temporal features of each modality Divide evenly along the time axis into The segments are designed to capture motion patterns at different time scales, specifically: ,in: A temporal feature tensor for a specific modality (e.g., IMU signal, text, or video features), with an asterisk "*" used to distinguish each segment from different modalities. For batch size, For time step, For feature dimensions. The number of segments into which the timeline is evenly divided, and assuming... Can be Divisible by integers, each fragment contains A continuous time step. Through compression and nonlinear transformation using a separate lightweight multilayer perceptron, a more compact fragment level is obtained. .

[0025] 2.3 Long-term dependency modeling: Average pooling is performed on each compressed segment to obtain its summary vector. Then, sinusoidal positional encoding is added to these fragment vectors to inject temporal sequence information, and they are fed into a multi-head self-attention layer for processing, resulting in... This refers to a multimodal temporal feature representation that combines local details with global context after fusion, specifically: a multi-head self-attention layer. ,in: . This refers to the batch size, which is the number of data samples processed at one time. The number of time segments, that is, the number of segments into which the temporal characteristics are evenly divided; For feature dimensions. It is the first A summary vector of each time segment. It is all A sequence of summary vectors of fragments concatenated along the time axis. It is a sinusoidal positional encoding function used to inject temporal order information into each segment vector in a sequence. Multi-head attention layers perform global dependency modeling on the input sequence. It is a fused feature representation of a certain modality after long-term dependency modeling. The superscript "*" corresponds to the modality identifier. The final temporal feature representations for the three modalities—IMU, video, and text—are used as inputs for subsequent multimodal fusion.

[0026] Step 3: Conduct multi-granularity contrastive semantic alignment training using a dual-granularity contrastive learning strategy to narrow the semantic gap between the IMU modality and the video and text modalities. Specifically, this includes:

[0027] 3.1 Coarse-grained inter-sample alignment: Calculate the global average vector of all fragment features for each sample. For each modality pair in the batch, a contrastive learning objective based on InfoNCE loss is used to encourage matching IMU-text sample pairs to be globally closer to each other, while distancing unmatched sample pairs from each other. Specifically: It has two modes. For cosine similarity calculation, Temperature over-parameter, It refers to the batch size. This is the index of the samples in the batch. For the first Each sample in modality The global variable is on top. For the first Each sample in modality The global variable is on top. For the first Each sample in modality The global variable is on top. It is a natural exponential function. It is the natural logarithm function.

[0028] 3.2 Fine-grained intra-sample alignment: directly aligning fragment sequences and A strict segment-by-segment alignment loss is applied, forcing the IMU signal and the text description to have the maximum semantic relevance in each corresponding local time segment, thereby finely modeling the cross-modal temporal correspondence, specifically: ,in: It has two modes. For cosine similarity calculation, Temperature over-parameter, It refers to the batch size. This represents the number of time segments. This is the index of the samples in the batch. For the first Each sample in modality The first A time segment variable, For the first Each sample in modality The first A time segment variable, It is a natural exponential function. It is the natural logarithm function.

[0029] 3.3 Joint Optimization Objective: Calculate the total training loss function based on the coarse-grained and fine-grained losses across all modality pairs (IMU-video, IMU-text, video-text). Optimize the IMU encoder by minimizing this joint loss. The parameters are configured so that the output features can maintain consistency with the visual semantics of the video and the linguistic semantics of the text at multiple levels, thus completing the initial embedding of the IMU signal in the shared semantic space, specifically: This is the set of all modal pairs that need to be aligned.

[0030] Step 4: In the online phase, query feature extraction and multimodal similarity retrieval are performed, specifically including:

[0031] 4.1 When a new, unlabeled query IMU signal is received, the fully trained IMU encoder is first utilized in the offline phase. Extract its high-dimensional semantic feature representation The encoder has the ability to map raw inertial signals into a semantic space shared with the text.

[0032] 4.2 Using a pre-built IMU-text pairing RAG database, the reference sample most semantically similar to the current query is retrieved. The retrieval process calculates query features. With all text features in the database The cosine similarity is implemented as follows: , ,in: , For the training set IMU features of each sample Provide its corresponding text description (which can be manually annotated or generated during the offline phase).

[0033] 4.3 Based on similarity score Sort in descending order and select the highest score. These samples constitute the initial candidate reference set. .

[0034] Step 5: Based on the retrieval pair information obtained in Step 4, construct an IMU signal relationship graph based on multi-dimensional similarity, representing nodes with candidate IMU features. , where: node set ,when At the node and Establish an edge between them Its weight is Edge weight It comprehensively considers two dimensions: feature space alignment and original temporal alignment. ,in: For the first Candidate IMU features, For similarity threshold, The cosine similarity function measures the directional consistency of feature vectors in the semantic space. To normalize the dynamic time warping distance, the two original IMU timing signals are directly calculated. The optimal alignment path cumulative distance can effectively handle local deformations in motion velocity and phase. Its normalized form ensures that the output value is within a reasonable range. This is an adjustable hyperparameter used to balance the contributions of semantic similarity and morphological similarity in the composition.

[0035] The edge set of the IMU signal relationship graph This reflects a stable and significant multivariate similarity relationship among candidate signals.

[0036] Step 6: Extract anchor-cluster semantic substructure. Specifically, in the IMU signal relationship graph obtained in Step 5, the node with the highest connectivity is taken as the semantic center anchor. The IMU signal corresponding to this point represents the most core and structurally influential motion pattern in the candidate set. The most frequently selected motion pattern is randomly chosen from the one-hop neighbor nodes of the semantic center anchor point. Nodes (if the number of neighbors is insufficient) Then select all (to form an anchor point-cluster subgraph). This is to capture the diverse semantic context surrounding the core pattern for text generation in step 7.

[0037] Step 7: Generate text based on structured multi-source prompts: Integrate multi-source information into a structured prompt template and input it into the large language model in Step 1 to guide its generation process. This process enables the model not only to "see" similar examples, but also to "understand" the intrinsic connections between these examples and their possible association with the query signal, thereby producing a final description that is more semantically accurate, richer in content, and more coherent in context.

[0038] The structured prompt template includes: a) the retrieved Top-K text description; b) the IMU signal description corresponding to the central anchor point; and c) natural language text describing the anchor point-clustering structure relationship.

[0039] Through practical application experiments, the method for cross-modal semantic understanding and generation of IMU signals for LLM, based on this invention, was run using the PyTorch framework on an NVIDIA A6000 / Ascend 910B GPU hardware environment and evaluated on three publicly available multimodal datasets: Ego4D, MMAct, and Ego-Exo4D. Experimental results show that in the cross-modal retrieval task from IMU to text, the recall rate of this invention significantly outperforms existing state-of-the-art methods, with a maximum improvement of 57.45%. Regarding text generation quality, the similarity between the generated description and the real text is improved by up to 31.90%, verifying the effectiveness of this invention in improving the IMU signal perception capability of LLM, and demonstrating that the technical effects are real and reproducible. Taking the publicly available multimodal dataset Ego4D, which contains a large amount of first-person perspective video and synchronized IMU data, as an example, the implementation process of this invention is illustrated, including:

[0040] Step 1: Offline multimodal alignment training, specifically including:

[0041] 2.1 Data Preparation: Sample segments (e.g., 5-second segments) containing synchronized video and IMU signals were selected from the Ego4D dataset. A pre-trained Qwen2.5-VL visual language model was used to generate text descriptions for each video segment. The prompt was designed as: "You have a first-person video taken from a head-mounted camera. Please describe the head and body movements in one sentence, based on the camera's height and orientation." The generated description, such as "A person is bending over carefully tending to plants in a garden," focuses on changes in body posture and is related to IMU signals.

[0042] 2.2 Model Building and Training:

[0043] IMU encoder: adopts SWIN Transformer architecture, and the input is preprocessed IMU timing signal.

[0044] Video / Text Encoder: Use the corresponding encoder from the pre-trained Qwen2.5-VL-7B model and freeze its parameters.

[0045] Temporal fusion module: Divides the feature sequences of each modality into... Each time period is compressed independently using an MLP, then its location is encoded, and finally fused across time periods using two layers of multi-head self-attention.

[0046] Loss function: Combined global contrastive loss and local segment alignment loss The total loss is the sum of the two losses over all modal pairs (video-text, video-IMU, text-IMU).

[0047] Training: The AdamW optimizer is trained on the training set of the selected dataset until convergence.

[0048] Step 2: Online retrieval enhances the reasoning process during the generation stage, specifically including:

[0049] 2.1 Database Construction: Store the IMU features of all samples in the training set and their corresponding generated text (or manually annotated text) to form an IMU-text pairing RAG database.

[0050] 2.2 For a new query IMU signal:

[0051] Feature extraction and retrieval: Extract query features using a trained IMU encoder. .calculate Retrieve the Top-10 (cosine similarity) text features from all text features in the database. The most similar text description.

[0052] Mapping and Selection: Using these 10 corresponding IMU signals as nodes, a similarity graph is constructed according to the method described in claims 5 and 6 (setting...). Select the node with the highest degree in the graph as the center anchor point, and randomly select 5 of them. )Neighbor.

[0053] Hint Construction and Generation: The hint used for text generation is as follows: "Refer to the following similar motion descriptions: [Top-10 texts are listed here]. Among them, the core motion pattern corresponds to: [Insert center anchor text here]. Relevant motion context includes: [Describe the relationship between the anchor and its neighbors here, such as 'These motions all involve repetitive movements of the upper body']. Based on the above information, describe the motion that the current sensor signal may represent." Input this hint into a large language model (such as Qwen3-32B) to generate the final description.

[0054] Performance Validation: Evaluation was conducted on the Ego4D test set. In the IMU-to-text retrieval task, the proposed method achieved a Recall@10 of 61.43%, significantly outperforming the baseline method IMU2CLIP (48.20%). Furthermore, the semantic similarity between the generated text and the actual action description reached 0.5546, indicating high generation quality. Experimental results are shown in Table 1.

[0055] Table 1

[0056] Compared to existing technologies, this method, in the offline training phase, effectively addresses the issues of semantic granularity mismatch between IMU signals and text, as well as the scarcity of training data, by introducing video as a bridging modality and employing a hierarchical cross-modal contrastive learning strategy that combines coarse and fine granularity. This enables the IMU encoder to learn more robust alignment values. In the online inference phase, this method designs a retrieval enhancement generation mechanism based on the IMU's intramodal semantic graph structure. By constructing a relationship graph between signals, it filters text descriptions with similar structures to the query signals, thereby bridging the semantic gap left by offline alignment and significantly improving the relevance and accuracy of text generation. These technological innovations work together to achieve significant improvements surpassing existing state-of-the-art (SOTA) performance in both cross-modal retrieval and text generation tasks.

[0057] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for cross-modal semantic understanding and generation of IMU signals for LLM, characterized in that, include: 1) Offline multimodal alignment stage: Using a pre-trained visual language model, motion-related text descriptions are generated from videos synchronized with IMU signals; multi-granular features are extracted through temporal fusion, and IMU encoders are trained by combining coarse-grained inter-sample alignment and fine-grained intra-sample alignment strategies to achieve preliminary alignment of IMU signals and text in a shared semantic space. 2) Online retrieval and enhancement generation stage: For newly input IMU signals, features are extracted using a pre-trained IMU encoder, and similar signal-text pairs are retrieved from the pre-built IMU-text pairing RAG database; a structural similarity graph between IMU signals is constructed, and the central node and its neighbors are selected to form an anchor point-clustering structure, which is input into the large language model together with the retrieved text description to generate a high-quality text description that is consistent with the semantics of the IMU signals.

2. The method for cross-modal semantic understanding and generation of IMU signals for LLM as described in claim 1, characterized in that, The training of the IMU encoder adopts a hierarchical contrastive learning strategy anchored by video and text: at the coarse-grained level, semantic alignment between samples is achieved through global contrastive loss; at the fine-grained level, local contrastive loss is calculated based on temporal segment division to achieve temporal alignment within samples.

3. The method for cross-modal semantic understanding and generation of IMU signals for LLM as described in claim 1, characterized in that, The construction of the structural similarity graph between IMU signals is specifically as follows: for the retrieved IMU signal samples, the similarity between nodes is calculated based on cosine similarity and dynamic time warping distance weighting. Nodes with similarity exceeding a preset threshold are connected, and the node with the highest connectivity is selected as the center node, which together with its one-hop neighbors forms an anchor point-clustering structure.

4. The method for cross-modal semantic understanding and generation of IMU signals for LLM as described in claim 1, characterized in that, The pre-trained IMU encoder uses a hierarchical Transformer network to extract local and global features of the IMU signal through block processing and shift window mechanism; the pre-built IMU-text pairing RAG database consists of IMU signal features obtained from the training set through offline processing and their corresponding text descriptions.

5. The method for cross-modal semantic understanding and generation of IMU signals for LLM according to any one of claims 1-4, characterized in that, The offline multimodal alignment stage specifically includes: Step 1: Generate motion text based on video bridging and prompting engineering. Specifically, input the visual context composed of video frame sequences, high-level action semantic labels, and metadata representing the shooting perspective into the pre-trained visual language big model. Use text prompts to constrain and guide the model generation process so that the text description output by the visual language big model will effectively remove visual semantics that are unrelated to the IMU signal. Step 2, Multimodal temporal feature fusion and structured representation, specifically includes: 2.1 Single-modal feature encoding: The IMU signal sequence, video frame sequence, and generated text sequence in the training data are set as follows: , , And features are extracted using the corresponding encoder; 2.2 Time Segmentation Compression and Fusion: Combining the temporal features of each modality Divide evenly along the time axis into The segments are designed to capture motion patterns at different time scales, specifically: ,in: For a specific modality (e.g., IMU signal, text, or video features), the superscript "*" is used to distinguish each segment from different modality types. For batch size, For time step, For feature dimension, The number of segments into which the timeline is evenly divided, and assuming... Can be Divisible by integers, each fragment contains A continuous time step Through compression and nonlinear transformation using a separate lightweight multilayer perceptron, a more compact fragment level is obtained. ; 2.3 Long-term dependency modeling: Average pooling is performed on each compressed segment to obtain its summary vector. Then, sinusoidal positional encoding is added to these fragment vectors to inject temporal sequence information, and they are fed into a multi-head self-attention layer for processing, resulting in... This refers to a multimodal temporal feature representation that combines local details with global context after fusion; Step 3: Conduct multi-granularity contrastive semantic alignment training using a dual-granularity contrastive learning strategy to narrow the semantic gap between the IMU modality and the video and text modalities. Specifically, this includes: 3.1 Coarse-grained inter-sample alignment: Calculate the global average vector of all fragment features for each sample. For each modality pair in the batch, a contrastive learning objective based on InfoNCE loss is used to encourage matching IMU-text sample pairs to be globally closer to each other, while distancing unmatched sample pairs from each other. Specifically: It has two modes. For cosine similarity calculation, Temperature over-parameter, It refers to the batch size. This is the index of the samples in the batch. For the first Each sample in modality The global variable is on top. For the first Each sample in modality The global variable is on top. For the first Each sample in modality The global variable is on top. It is a natural exponential function. It is the natural logarithm function; 3.2 Fine-grained intra-sample alignment: directly aligning fragment sequences and A strict segment-by-segment alignment loss is applied, forcing the IMU signal and the text description to have the maximum semantic relevance in each corresponding local time segment, thereby finely modeling the cross-modal temporal correspondence, specifically: ,in: It has two modes. For cosine similarity calculation, Temperature over-parameter, It refers to the batch size. The number of time segments, This is the index of the samples in the batch. For the first Each sample in modality The first A time segment variable, For the first Each sample in modality The first A time segment variable, It is a natural exponential function. It is the natural logarithm function; 3.3 Joint Optimization Objective: Calculate the total training loss function based on the coarse-grained and fine-grained losses across all modality pairs (IMU-video, IMU-text, video-text). Optimize the IMU encoder by minimizing this joint loss. The parameters are configured so that the output features can maintain consistency with the visual semantics of the video and the linguistic semantics of the text at multiple levels, thus completing the initial embedding of the IMU signal in the shared semantic space, specifically: This is the set of all modal pairs that need to be aligned.

6. The method for cross-modal semantic understanding and generation of IMU signals for LLM as described in claim 5, characterized in that, The single-modal feature encoding mentioned above is specifically as follows: For the IMU encoder to be trained, its parameters Random initialization; and These are pre-trained video encoders and text encoders, respectively, with parameters... All output features are frozen during training. ,in For batch size, For time step, For feature dimensions.

7. The method for cross-modal semantic understanding and generation of IMU signals for LLM as described in claim 5, characterized in that, The aforementioned long-term dependency modeling specifically refers to a multi-head self-attention layer. ,in: This refers to the batch size, which is the number of data samples processed at one time. The number of time segments, that is, the number of segments into which the temporal characteristics are evenly divided; For feature dimension, It is the first A summary vector of each time segment. It is all A sequence of summary vectors of fragments concatenated along the time axis. It is a sinusoidal positional encoding function used to inject temporal order information into each segment vector in a sequence. Multi-head attention layers perform global dependency modeling on the input sequence. It is a fused feature representation of a certain modality after long-term dependency modeling. The superscript "*" corresponds to the modality identifier. The final temporal feature representations for the three modalities—IMU, video, and text—are used as inputs for subsequent multimodal fusion.

8. The method for cross-modal semantic understanding and generation of IMU signals for LLM according to any one of claims 1-4, characterized in that, The online search enhancement generation stage specifically includes: Step 4: In the online phase, query feature extraction and multimodal similarity retrieval are performed, specifically including: 4.1 When a new, unlabeled query IMU signal is received, the fully trained IMU encoder is first utilized in the offline phase. Extract its high-dimensional semantic feature representation ; 4.2 Using a pre-built IMU-text pairing RAG database, the reference sample most semantically similar to the current query is retrieved. The retrieval process calculates query features. With all text features in the database The cosine similarity is implemented as follows: , ,in: , For the training set IMU features of each sample Its corresponding text description; 4.3 Based on similarity score Sort in descending order and select the highest score. These samples constitute the initial candidate reference set. ; Step 5: Based on the retrieval pair information obtained in Step 4, construct an IMU signal relationship graph based on multi-dimensional similarity, representing nodes with candidate IMU features. , where: node set ,when At the node and Establish an edge between them Its weight is Edge weight It comprehensively considers two dimensions: feature space alignment and original temporal alignment. ,in: For the first Candidate IMU features, For similarity threshold, The cosine similarity function measures the directional consistency of feature vectors in the semantic space. To normalize the dynamic time warping distance, the two original IMU timing signals are directly calculated. The cumulative distance of the optimal alignment path between them It is an adjustable hyperparameter used to balance the contributions of semantic similarity and morphological similarity in the composition; Step 6: Extract anchor-cluster semantic substructure. Specifically, in the IMU signal relationship graph obtained in Step 5, the node with the highest connectivity is taken as the semantic center anchor. The IMU signal corresponding to this point represents the most core and structurally influential motion pattern in the candidate set. The most frequently selected motion pattern is randomly chosen from the one-hop neighbor nodes of the semantic center anchor point. Each node forms an anchor-cluster subgraph. To capture the diverse semantic context surrounding this core pattern for text generation in step 7; Step 7: Generate text based on structured multi-source prompts: Integrate multi-source information into a structured prompt template and input it into the large language model in Step 1 to guide its generation process.

9. The method for cross-modal semantic understanding and generation of IMU signals for LLM as described in claim 8, characterized in that, The structured prompt template includes: a) the retrieved Top-K text description; b) the IMU signal description corresponding to the central anchor point; and c) natural language text describing the anchor point-clustering structure relationship.

10. A system for LLM-oriented cross-modal semantic understanding and generation of IMU signals, implementing the method of any one of claims 1-9, characterized in that, include: The system comprises a text description unit, a training unit, a retrieval unit, and a description generation unit. Specifically: the text description unit generates motion-related text descriptions using a pre-trained visual language model based on video information synchronized with the IMU signal; the training unit extracts multi-granular features through temporal fusion based on the IMU signal and the text descriptions output by the text description unit, and trains the IMU encoder using coarse-grained inter-sample alignment and fine-grained intra-sample alignment strategies, resulting in a trained IMU encoder and an IMU-text pairing RAG database; the retrieval unit extracts features using the IMU encoder based on newly input IMU signals and the IMU encoder and database information output by the training unit, and retrieves similar signal-text pairs from the database, obtaining retrieval results; and the description generation unit constructs a structural similarity graph between IMU signals based on the retrieval results output by the retrieval unit, selects a central node and its neighbors to form an anchor-cluster structure, and inputs it along with the retrieved text descriptions into a large language model to generate text descriptions semantically consistent with the IMU signals, thus obtaining the final result.

Images