Millimeter wave semantic information extraction method based on large language model and data synthesis

Abstract

The application discloses a millimeter wave semantic information extraction method based on a large language model and data synthesis, which comprises the following steps: generating a sports scene description by using a large language model, and generating a corresponding three-dimensional human body motion model; simulating the interaction behavior of a millimeter wave radar signal and the generated three-dimensional human body motion model through ray tracing, collecting all the returned radar signal path information, superimposing and calibrating to form a complete millimeter wave simulation signal, and performing field randomization; constructing a semantic alignment millimeter wave interpreter, and using the millimeter wave interpreter to extract semantic information of the millimeter wave. The application alleviates the challenge of data acquisition, solves the problem that the real data is too dependent on the generation of millimeter wave data guided by the LLM, enhances the data production efficiency, and reduces the demand for manual debugging and intervention.

Description

Technical Field

[0001] The present invention relates to the field of millimeter wave sensing technology, and in particular to a millimeter wave semantic information extraction method based on a large language model and data synthesis. Background Art

[0002] With the rapid development of millimeter-wave sensing technology, it has shown great potential in human-centric applications such as health monitoring, security monitoring, and smart homes, mainly due to its non-invasive and privacy-preserving characteristics. However, a major challenge lies in training deep neural network models to accurately interpret the human activity information contained in millimeter-wave signals, which requires a large amount of high-quality labeled data. Currently, the high cost of data collection and annotation causes existing models to perform poorly in new environments or for unseen users. To alleviate the problem of data scarcity, early methods attempted to generate synthetic data through channel models, simulators, or deep learning techniques to reduce dependence on real-world data. However, the synthetic data generated by these methods still lacks guidance for downstream task scenarios and global inference, and often cannot fully capture the complex characteristics of real-world scenarios, making the performance of the model unstable in practical applications.

[0003] Existing traditional millimeter wave data processing technologies, such as those relying on specific classification criteria or reconstruction algorithms, have inherent limitations in the deep semantic understanding and complex scene interpretation required. They often struggle to effectively correlate raw millimeter wave data with the complex semantic information behind it, and are even less able to provide in-depth, human-friendly interpretation of this information using natural language.

[0004] Large language models (LLMs) demonstrate significant advantages in natural language understanding and generation, enabling deep semantic analysis and reasoning. Despite their enormous potential, LLMs still face significant technical challenges in effectively applying their semantic understanding capabilities to millimeter-wave data processing, particularly in guiding the generation of millimeter-wave data with specific semantic meanings and accurately interpreting the semantic information of these data and their corresponding scenarios. Specifically, there is currently a lack of a technical solution that can integrate the semantic understanding advantages of large language models to overcome the shortcomings of traditional millimeter-wave data processing in semantic information generation and interpretation. This is particularly evident in the use of LLMs to generate high-quality, diverse millimeter-wave data and simultaneously produce corresponding semantic interpretations. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the existing technology and propose a method for extracting millimeter wave semantic information based on a large language model and data synthesis.

[0006] The object of the present invention is achieved through the following technical solution: a method for extracting millimeter wave semantic information based on a large language model and data synthesis, the method comprising:

[0007] S1. Use a large language model to generate motion scene descriptions, use a pre-trained text-driven human motion generation model to process the description text, and generate the corresponding 3D human motion model;

[0008] S2. Use ray tracing to simulate the interaction between the millimeter-wave radar signal and the generated 3D human motion model. Collect all signal path information returning to the radar, superimpose and calibrate it to form a complete millimeter-wave simulation signal, and perform domain randomization.

[0009] S3. Build a mmWave interpreter, which includes a pre-trained mmWave feature extractor and sentence feature extractor, a multimodal projection layer, and a pre-trained large language model backbone that supports mmWave modal input.

[0010] During the pre-training process of the feature extractor, the millimeter wave signal features are aligned with the text features;

[0011] S4. Use the millimeter wave interpreter to extract semantic information from millimeter waves.

[0012] Furthermore, generating a motion scene description specifically includes:

[0013] Input the scenario description, apply syntax and synonym replacement strategies, enhance the diversity of actions with multiple syntactic structures, and output explicit action descriptions, implicit behaviors and result predictions.

[0014] Furthermore, the text-driven human motion generation model processes text including: first, decomposing the text into tokens and expressing them as vector embeddings, wherein each vector in the vector embeddings is obtained by searching the embedding matrix, and inputting the vector embeddings into the encoder to generate output features, wherein the output features include the representation of the entire input context at each position, and using a decoder to convert the features output by the encoder into an action sequence to represent each action key point; the action sequence is converted into a three-dimensional human skeleton key point representation through coordinate transformation and scaling, and smoothed using Gaussian filtering.

[0015] Furthermore, the process of acquiring the millimeter wave simulation signal specifically includes:

[0016] During ray tracing, multiple rays are sent along the line of sight starting from the radar emission position. Each ray represents a possible signal path. These rays will interact with the facets of the human body model in three-dimensional space, and all interaction points, including reflection, scattering, and absorption points, are recorded.

[0017] For each interaction point, the time delay of the signal path and its energy intensity are calculated, taking into account the effects of material properties, incident and reflection angles on energy. Subsequently, the signal path information of all returning radars is collected, and the superposition effect of the signals on each path is simulated. The returned RF signals are superimposed on the time axis, and their energy intensities are accumulated to form a complete millimeter wave simulation signal.

[0018] Furthermore, the field randomization includes:

[0019] By adjusting the radar viewing angle, random weight distribution in the segmented human mesh, and random antenna patterns, we ensure that the synthetic data remains valid in diverse and changing environments.

[0020] Furthermore, the millimeter wave feature extractor includes: using a Transformer-based encoder, first dividing the input spectrum into a series of flattened two-dimensional patches, each patch is projected into a latent feature space, and retaining the contextual information of the time and Doppler axes by using learnable position embeddings. Next, the global features of the input are extracted through a multi-layer Transformer encoder, and the information is aggregated through learnable classification tokens to output a global feature embedding and a series of sequence embeddings, wherein the global feature embedding is used for semantic alignment pre-training, and the sequence embedding is used for the millimeter wave modal input of the millimeter wave interpreter.

[0021] Furthermore, the millimeter wave interpreter specifically includes: the millimeter wave features output by the pre-trained millimeter wave feature extractor are converted into a format that is easy to integrate through a multi-layer perceptron projection layer, aligned with the language features, and then merged and output to the pre-trained Phi3 large-scale language model backbone to understand multimodal content in an autoregressive manner; the large-scale language model backbone uses low-rank adaptation technology during the pre-training process and introduces a trainable low-rank matrix in the linear layer of the Transformer.

[0022] Furthermore, the pre-training of the millimeter wave feature extractor specifically includes: by pairing millimeter wave features with sentence embeddings, the model learns to maximize the similarity of corresponding pairs while minimizing the similarity between non-corresponding pairs, and by using cosine similarity to evaluate the distance of multimodal features in the latent space, it supports seamless comparison between millimeter wave and text representations, and uses the InfoNCE loss function in model training to achieve robust semantic alignment.

[0023] Furthermore, the sentence embedding feature extractor embeds the input token into a high-dimensional feature space to capture important grammatical and semantic information, and adds positional encoding. The processed token passes through the Transformer encoder to aggregate global sentence-level features; finally, a compact sentence-level embedding is generated through a convergence mechanism, which is used as a global feature aligned with the millimeter wave motion features.

[0024] On the other hand, the specification of the present invention also provides a millimeter wave semantic information extraction device based on a large language model and data synthesis, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the millimeter wave semantic information extraction method based on a large language model and data synthesis.

[0025] Beneficial effects of the present invention:

[0026] (1) This paper leverages the powerful reasoning and generation capabilities of large language models (LLMs) to achieve automated millimeter-wave data synthesis. This technique significantly reduces the need for and associated costs of collecting large amounts of real data. This is particularly important for scenarios where extensive experimental data is typically required for training. It effectively alleviates data acquisition challenges and addresses the overreliance on real data when using LLMs to guide millimeter-wave data generation.

[0027] (2) This invention improves the diversity and quality of synthesized millimeter wave data by designing a module that can generate diverse scene descriptions. This enables the synthesized data to cover a wider range of application scenarios and operating conditions, and ensures that the generated data can truly reflect the complex situations in actual applications in terms of content and structure, such as various human activity patterns and changes in background environment. This improves the practicality and fidelity of the synthesized data guided by LLM, enabling it to better serve subsequent semantic understanding tasks.

[0028] (3) This invention significantly enhances the semantic understanding and generalization capabilities of the overall model by providing an advanced signal feature extraction method and combining it with a pre-trained language model (such as BERT) to semantically enhance the representation of natural language. By developing an extraction technology that efficiently captures key features from millimeter wave signals, it can accurately convey complex human behavior and movement information. At the same time, the designed semantic embedding optimizes the natural language description capability, laying a solid foundation for subsequent LLM to perform deep semantic analysis and accurately interpret millimeter wave data and text information.

[0029] (4) This invention uses an advanced semantic alignment strategy to achieve precise matching and deep semantic docking between millimeter-wave signal features and natural language text features, thereby effectively improving the integration of multimodal data. On this basis, this invention constructs an end-to-end automated synthesis process from text descriptions to millimeter-wave signals. Through the cross-modal conversion capability from text to millimeter-wave signals, the entire process is automated, which not only significantly enhances data production efficiency but also reduces the need for manual debugging and intervention, successfully overcoming the difficulty of efficiently integrating LLM and millimeter-wave data processing processes.

[0030] (5) This invention significantly improves the accuracy and richness of text descriptions generated from millimeter-wave signals by deeply integrating the powerful understanding capabilities of large-scale language models (LLMs). This integration ensures that the generated semantic information is not only accurate in content but also rich and natural in expression, thereby ensuring the overall quality of the synthesized data and its semantic interpretation and its practical value in practical applications, effectively addressing the shortcomings of existing technologies in providing high-quality semantic information interpretation of millimeter-wave data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0032] Figure 2 An electromagnetic simulation diagram of a radio frequency signal provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of domain randomization for simulation-to-real conversion provided by an embodiment of the present invention;

[0034] Figure 4 A schematic diagram of the structure of the millimeter wave interpreter WaveLLM provided in an embodiment of the present invention;

[0035] Figure 5 Schematic diagram of a millimeter-wave semantic information extraction device based on a large language model and data synthesis provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0037] This paper proposes a method for extracting millimeter-wave semantic information based on a large language model and data synthesis. This method involves both millimeter-wave data synthesis and millimeter-wave interpretation. Its design goal is to significantly improve the performance of millimeter-wave sensing systems through effective model training and data generation. Simulated millimeter-wave data synthesized using this method is then used for data augmentation to train a large language model-based model for interpreting millimeter-wave signal semantic information. The following details the specific components of this solution and its operating principles.

[0038] like Figure 1 As shown, an embodiment of the present invention provides a method for extracting millimeter wave semantic information based on a large language model and data synthesis, including:

[0039] S1. Use a large language model to generate motion scene descriptions, use a pre-trained text-driven human motion generation model to process the description text, and generate the corresponding 3D human motion model;

[0040] This step specifically includes:

[0041] Scenario Description Generator:

[0042] The scenario description generator analyzes the context of the user input through a large language model, generating diverse motion cues. These cues not only include explicit action descriptions but also infer implicit behaviors and potential consequences. To ensure that the generated synthetic data is adaptable to various application scenarios, special attention is paid to the accuracy of action definitions and the diversity of descriptions.

[0043] When generating motion descriptions, syntactic and synonym replacement strategies are applied to enhance the diversity of actions in various syntactic structures, such as different tenses and inverted patterns, thereby generating rich and diverse synthetic data.

[0044] Motion Synthesizer:

[0045] The pre-trained text-driven human motion generation model MoMask is used to process the generated descriptive text cues, generating corresponding three-dimensional human motion sequences. These sequences are represented by three-dimensional human skeleton key points, which are then converted into detailed three-dimensional meshes by fitting the SMPL human surface model.

[0046] The text-driven human motion generation model converts input text into three-dimensional human motion sequences through several key steps. First, the input text is tokenized, breaking the text TT into a series of tokens represented as {t1, t2, …, tn}. These tokens are represented as vector embeddings {v1, v2, …, vn}, where each vector vi is obtained by looking up the embedding matrix EE: vi = E(ti). Next, these embedding sequences are input into a Transformer encoder to generate a series of output features {h1, h2, …, hn}. The encoding process of the Transformer can be represented as: hi = TransformerEncoder(v1, v2, …, vn), where hi is the model's representation of the entire input context at position i. Subsequently, through the Transformer's output features {h1, h2, …, hn}, a decoder is applied to convert these features into a sequence of action tokens {m1, m2, …, mk}, where each mi is a representation of an action key point. This process can be represented by a decoding function f: mi = f(h1, h2, …, hn). Finally, the sequence of action tokens {m1, m2, …, mk} is further converted into a three-dimensional human skeleton key point representation {p1, p2, …, pk} through coordinate transformation and scaling, where pi is a key point represented by three-dimensional coordinates.

[0047] Since 3D human skeleton motion data generated in the time domain may have jitter and inter-frame discontinuity issues, a Gaussian filter is applied to smooth the time series to improve the smoothness and naturalness of the motion sequence. To smooth jitter and inter-frame discontinuity in motion data, a Gaussian filter can be used to process the time series. Assuming the original motion sequence is x(t), where t represents time, the smoothed sequence after Gaussian filtering can be expressed as:

[0048]

[0049] S2. Use ray tracing to simulate the interaction between the millimeter-wave radar signal and the generated 3D human motion model. Collect all signal path information returning to the radar, superimpose and calibrate it to form a complete millimeter-wave simulation signal, and perform domain randomization.

[0050] The specific steps are:

[0051] RF Signal Electromagnetic Simulation:

[0052] Applying physical electromagnetic principles, the interaction between millimeter-wave radar signals and the generated three-dimensional human body model is simulated to generate realistic radar data. Specifically, in the ray tracing process, starting from the radar launch position, multiple rays are sent along the line of sight, and each ray represents a possible signal path. These rays will interact with the vertices of the human body model in three-dimensional space, and all interaction points are recorded, including reflection, scattering, and absorption points. For each interaction point, the time delay of the signal path and its energy intensity are calculated, considering the influence of material properties, incident and reflection angles on energy. Subsequently, the signal path information of all returning radars is collected, and the superposition effect of the signals of each path is simulated. The returned RF signals are superimposed on the time axis, and their energy intensities are accumulated to form a complete millimeter-wave simulation signal. Such as Figure 2 As shown, the signal path loss (determined by the transmission distance), antenna loss (determined by the antenna's incident and outgoing directions), and scattering loss (determined by the scattering direction and the normal vector direction at the scattering point) are specifically considered. Accurate calculations are used to improve the authenticity and reliability of the synthesized signal. Simulations are performed using the following path coefficient calculation equation:

[0053]

[0054] Where Ctx and Crx are the gains of the transmitting antenna and the receiving antenna after considering the radar pattern, λ is the wavelength of the millimeter wave signal, R is the distance between the radar and the target, dA′ is the projected area of ​​the scattering surface, Γ is the scattering coefficient, is the scattering pattern function, and the energy coefficient of each simulation path is finally calculated .

[0055] Domain Randomization for Sim-to-RealTransfer:

[0056] like Figure 3 As shown in the figure, to address the discrepancies between synthetic and real data, domain randomization technology is used. By varying the radar viewing angle, the weights of the body segments, the antenna pattern, and adding random background noise, the model is trained to recognize features that remain constant under different conditions, improving its generalization ability. This includes adjusting the radar viewing angle, randomizing the weight distribution within the segmented body mesh, and randomizing the antenna pattern to ensure that the synthetic data remains valid in diverse and changing environments.

[0057] S3. Build a mmWave interpreter, which includes a pre-trained mmWave feature extractor, a multimodal projection layer, and a pre-trained large language model backbone that supports mmWave modality input.

[0058] The pre-training process of the millimeter wave feature extractor includes: semantically aligning the global features of the millimeter wave signal generated by the millimeter wave feature extractor with the corresponding text features through the text representation output by the sentence embedding feature extractor, and training using a loss function;

[0059] This step specifically includes:

[0060] The proposed mmWave-to-text semantic information understanding framework transforms complex mmWave signals into semantically rich natural language descriptions through the collaborative work of four key modules, thereby improving the performance of downstream perception tasks. These modules include: (1) a mmWave signal feature extractor, which uses a Transformer-based encoder to extract signal features from mmWave spectral data, providing a comprehensive feature foundation for semantic understanding; (2) a sentence feature extractor, which uses the BERT model to generate rich semantic embeddings from the input text, ensuring the effective presentation of natural language descriptions; (3) a semantic alignment pre-training, which uses a CLIP-inspired contrastive learning method to semantically align mmWave signal features with corresponding text features, creating a unified semantic framework to bridge the gap between radar data and language; and (4) a mmWave interpreter, WaveLLM, which integrates the aligned features into a large language model (LLM) framework, enabling the generation of contextually accurate and semantically rich text descriptions based on mmWave signal input. This framework obtains the embedded information of millimeter waves and text through a feature extractor, establishes the relationship between signals and language through semantic alignment pre-training, and finally WaveLLM delivers the aligned features to LLM to produce semantically accurate text, forming a complete conversion process from radar data to natural language.

[0061] Millimeter wave signal feature extractor:

[0062] This module uses a Transformer-based encoder to capture the temporal information in millimeter wave signals, especially the details in micro-Doppler spectra. Specifically, the input spectral data is processed into a two-dimensional matrix I ∈ , where R indicates that the data is in the real number domain, H represents the time axis, and W represents the Doppler frequency axis, which represents the radial velocity of the target. The encoder first divides the input spectrum into a series of flattened two-dimensional patches, and each patch is projected into a latent feature space with a dimension of D. By using learnable position embedding, the contextual information of the time and Doppler axes is retained. Next, the global features of the input are extracted through a multi-layer Transformer encoder, and the information is aggregated through learnable classification tokens to output a global feature embedding and a series of sequence embeddings. The global feature embedding is used for subsequent semantic alignment pre-training, and the sequence embedding is used for the millimeter wave modal input of the millimeter wave interpreter.

[0063] Sentence feature extractor:

[0064] The sentence feature extractor is used to extract features from text, which are used to align with the millimeter wave global feature embedding in the feature extractor semantic alignment pre-training, and are also used as language feature input in WaveLLM.

[0065] The sentence feature extractor, based on the existing BERT model, is specifically designed to generate rich semantic embeddings for text input. Built on the Transformer architecture, this model effectively captures the complex contextual dependencies in text. Initially, input tokens are embedded into a high-dimensional feature space to capture important grammatical and semantic information. To preserve the sequential structure of the text, positional encoding is incorporated. The processed tokens are then passed through the Transformer encoder to aggregate global sentence-level features. Finally, a convergence mechanism generates a compact sentence-level embedding, which serves as a global feature aligned with the millimeter wave motion features.

[0066] Feature extractor semantic alignment pre-training:

[0067] This model uses a contrastive learning approach, inspired by the CLIP framework, to align mmWave signal features with text features. By pairing mmWave features with sentence embeddings, the model learns to maximize the similarity between corresponding pairs while minimizing the similarity between non-corresponding pairs. By using cosine similarity to assess the distance between multimodal features in latent space, it enables seamless comparison between mmWave and text representations. The InfoNCE loss function is used during model training to achieve robust semantic alignment.

[0068] Millimeter wave interpreter WaveLLM:

[0069] like Figure 4 As shown, WaveLLM aims to unlock the potential of mmWave semantic understanding and integrate mmWave signal input with LLM's multimodal context parsing. It is mainly composed of the mmWave feature extractor, multimodal projection layer, sentence feature extractor pre-trained in the above process, and the pre-trained Phi3 large-scale language model (LLM) backbone. By enhancing the ability of pre-trained LLM to process mmWave signal features and language features. The mmWave features are converted into an easy-to-integrate format through a shallow multi-layer perceptron (MLP) projection layer, and then aligned with the language features. Finally, the combined input is fed into the LLM to understand the multimodal content in an autoregressive manner. In order to maintain the wide applicability of the pre-trained LLM while enhancing its understanding and reasoning capabilities of mmWave signals, the existing mature low-rank adaptation (LoRA) technology is applied. By introducing a trainable low-rank matrix in the linear layer of the Transformer, the multimodal LLM can be effectively trained, providing powerful understanding capabilities, and proficient in parsing and reasoning in the mmWave and text data spaces. As Figure 4 As shown, the millimeter wave signal is converted into millimeter wave feature words by the millimeter wave feature extractor, and the text is converted into text feature words by the sentence feature extractor. After merging, they are input into the LLM, and the ignored words and the expected output words are output.

[0070] S4, using the millimeter wave interpreter to extract semantic information of the millimeter wave.

[0071] Corresponding to the foregoing embodiment of the method for extracting semantic information of millimeter waves based on a large language model and data synthesis, the present application also provides an embodiment of a device for extracting semantic information of millimeter waves based on a large language model and data synthesis.

[0072] Referring to Figure 5 The device for extracting semantic information of millimeter waves based on a large language model and data synthesis provided by the embodiment of the present application comprises a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to implement the method for extracting semantic information of millimeter waves based on a large language model and data synthesis in the foregoing embodiment.

[0073] The embodiment of the device for extracting semantic information of millimeter waves based on a large language model and data synthesis provided by the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software realization as an example, as a logically meaningful device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for execution by the processor of the device with data processing capability where it is located. From the hardware level, as shown in Figure 5 As shown in the figure, in addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, the device with data processing capability where the device embodiment is located usually also comprises other hardware according to the actual functions of the device with data processing capability, and details are not described here. Figure 5

[0074] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and is not described here.

[0075] ​For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0076] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a millimeter wave semantic information extraction method based on a large language model and data synthesis in the above embodiment is implemented.

[0077] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0078] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the millimeter wave semantic information extraction method based on a large language model and data synthesis.

[0079] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0080] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A method for extracting millimeter wave semantic information based on a large language model and data synthesis, characterized in that: The method includes: S1. Use a large language model to generate motion scene descriptions, use a pre-trained text-driven human motion generation model to process the description text, and generate the corresponding 3D human motion model; S2. Use ray tracing to simulate the interaction between the millimeter-wave radar signal and the generated 3D human motion model. Collect all signal path information returning to the radar, superimpose and calibrate it to form a complete millimeter-wave simulation signal, and perform domain randomization. S3. Build a mmWave interpreter, which includes a pre-trained mmWave feature extractor and sentence feature extractor, a multimodal projection layer, and a pre-trained large language model backbone that supports mmWave modal input. During the pre-training process of the feature extractor, the millimeter wave signal features are aligned with the text features; S4. Use the millimeter wave interpreter to extract semantic information from millimeter waves.

2. The method for extracting millimeter wave semantic information based on a large language model and data synthesis according to claim 1, characterized in that: Generating a motion scene description specifically includes: Input the scenario description, apply syntax and synonym replacement strategies, enhance the diversity of actions with multiple syntactic structures, and output explicit action descriptions, implicit behaviors and result predictions.

3. The method for extracting millimeter wave semantic information based on a large language model and data synthesis according to claim 1, characterized in that: The text-driven human motion generation model processes text by: first, decomposing the text into tokens and expressing them as vector embeddings, wherein each vector in the vector embeddings is obtained by searching the embedding matrix, inputting the vector embeddings into an encoder to generate output features, wherein the output features include the representation of the entire input context at each position, and using a decoder to convert the features output by the encoder into an action sequence representing each action key point; converting the action sequence into a three-dimensional human skeleton key point representation through coordinate transformation and scaling, and using Gaussian filtering for smoothing.

4. The method for extracting millimeter wave semantic information based on a large language model and data synthesis according to claim 1, characterized in that: The process of acquiring the millimeter wave simulation signal specifically includes: During ray tracing, multiple rays are sent along the line of sight starting from the radar emission position. Each ray represents a possible signal path. These rays will interact with the human body model in three-dimensional space, and all interaction points, including reflection, scattering and absorption points, are recorded. For each interaction point, the time delay of the signal path and its energy intensity are calculated, taking into account the effects of material properties, incident and reflection angles on energy. Subsequently, the signal path information of all returning radars is collected, and the superposition effect of the signals on each path is simulated. The returned RF signals are superimposed on the time axis, and their energy intensities are accumulated to form a complete millimeter wave simulation signal.

5. The method for extracting millimeter wave semantic information based on a large language model and data synthesis according to claim 1, characterized in that: The domain randomization includes: By adjusting the radar viewing angle, random weight distribution in the segmented human mesh, and random antenna patterns, we ensure that the synthetic data remains valid in diverse and changing environments.

6. The method for extracting millimeter wave semantic information based on a large language model and data synthesis according to claim 1, characterized in that: The millimeter wave feature extractor includes: using a Transformer-based encoder, first dividing the input spectrum into a series of flattened two-dimensional patches, each patch is projected into a latent feature space, and retaining the contextual information of the time and Doppler axes by using learnable position embeddings. Next, the global features of the input are extracted through a multi-layer Transformer encoder, and the information is aggregated through learnable classification tokens to output a global feature embedding and a series of sequence embeddings, where the global feature embedding is used for semantic alignment pre-training and the sequence embedding is used for the millimeter wave modal input of the millimeter wave interpreter.

7. The method for extracting millimeter wave semantic information based on a large language model and data synthesis according to claim 1, characterized in that: The millimeter wave interpreter specifically includes: the millimeter wave features output by the pre-trained millimeter wave feature extractor are converted into a format that is easy to integrate through a multi-layer perceptron projection layer, aligned with the language features, and then merged and output to the pre-trained Phi3 large-scale language model backbone to understand multimodal content in an autoregressive manner; the large-scale language model backbone uses low-rank adaptation technology during the pre-training process and introduces a trainable low-rank matrix in the linear layer of the Transformer.

8. The method for extracting millimeter wave semantic information based on a large language model and data synthesis according to claim 1, characterized in that: The pre-training of the feature extractor specifically includes: by pairing millimeter wave features with sentence embeddings, the model learns to maximize the similarity of corresponding pairs while minimizing the similarity between non-corresponding pairs, and by using cosine similarity to evaluate the distance of multimodal features in the latent space, it supports seamless comparison between millimeter wave and text representations. The InfoNCE loss function is used in model training to achieve robust semantic alignment.

9. The method for extracting millimeter wave semantic information based on a large language model and data synthesis according to claim 1, characterized in that: The sentence feature extractor embeds the input token into a high-dimensional feature space to capture important grammatical and semantic information, and adds positional encoding. The processed token passes through the Transformer encoder to aggregate global sentence-level features; finally, a compact sentence-level embedding is generated through a convergence mechanism, which is used as a global feature aligned with the millimeter wave motion features.

10. A millimeter wave semantic information extraction device based on a large language model and data synthesis, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a millimeter wave semantic information extraction method based on a large language model and data synthesis as described in any one of claims 1 to 9.

Images