A method for constructing an end-to-end millimeter wave-language large model

Abstract

The application particularly relates to a construction method of an end-to-end millimeter wave-language large model, which comprises the following steps: firstly, performing three-dimensional FFT processing on a millimeter wave radar original I / Q signal in distance, Doppler and angle, generating a multi-dimensional spectrum diagram, and simultaneously analyzing a target time sequence sparse point cloud; then, constructing a parallel spectrum diagram encoder and a point cloud encoder double-branch network, respectively extracting time-space and geometric features of two modalities, and forming a unified multi-modal representation through cross-spectrum interaction and projection fusion; finally, introducing a millimeter wave special mark and embedding an adaptive adapter, mapping the representation to a pre-trained large language model semantic space, and based on a low-rank adaptive method, fine-tuning an instruction to realize end-to-end mapping from an original signal to an action category or a natural language description. The application effectively solves the problems of a traditional method process fragmentation, multi-modal fusion difficulty and weak generalization ability, and has the advantages of strong cross-scene zero-shot generalization ability and high output explainability.

Description

Technical Field

[0001] This invention relates to the fields of wireless sensing and artificial intelligence technology, and to an end-to-end method for constructing a large millimeter-wave language model. Specifically, it relates to an end-to-end construction method that uses spectrogram / point cloud representation and dual-branch feature extraction of raw millimeter-wave radar signals, and performs instruction fine-tuning on a large language model to achieve unified semantic understanding and task decision-making. Background Technology

[0002] In recent years, millimeter wave (mmWave) sensing applications in frequency bands such as 24 / 60 / 77 GHz have developed rapidly, and are widely used in scenarios such as vehicle environmental perception, indoor security, human motion recognition, and vital sign monitoring. Leveraging higher available bandwidth and scalable array apertures, millimeter waves possess more detailed spatial resolution at the physical layer. Meanwhile, basic models and the "large model + fine-tuning" paradigm have made significant progress in the fields of vision and language, with cross-modal models demonstrating strong generalization capabilities from pre-training to downstream adaptation.

[0003] However, existing research methods still face a series of interconnected and profound challenges in establishing a unified technical path from raw millimeter-wave I / Q signals to general semantic understanding. First, mainstream technical processes are generally fragmented, mostly following a cascaded model of "signal processing → manual feature extraction / small model classification → rule-based decision-making." This model compresses the richly informative continuous phase, time, and angular structures into limited statistics or simple labels, making it difficult for subtle evidence in the upstream signal to be fully understood and utilized by the downstream semantic model, thus creating a bottleneck in information transmission.

[0004] Secondly, millimeter-wave data itself presents significant heterogeneity challenges. Radar systems typically generate both spectral maps and sparse point clouds simultaneously, which differ fundamentally in coordinate systems, data densities, scale specifications, and noise distributions. Existing methods lack an effective joint modeling framework for the complementarity between these two heterogeneous modes and the underlying geometric consistency, limiting the full realization of the potential for multi-view information fusion.

[0005] More fundamentally, current methods suffer from significant limitations in generalization capabilities. Most models optimize only within a closed label space, struggling to achieve open-set recognition and zero-shot / few-shot generalization through flexible methods such as textual task descriptions or category prototypes. This makes them ill-suited to the dynamic characteristics of evolving task requirements and action categories in real-world applications. From a temporal modeling perspective, existing work largely focuses on single-frame spectrogram analysis or independent processing of inter-frame information, failing to fully utilize sequence-level temporal position encoding, lightweight temporal modeling units, or cross-frame association mechanisms. Consequently, it is difficult to stably and coherently characterize the various stages and subtle nuances of continuous actions. In the cutting-edge direction of integration with large language models, crucial multimodal alignment mechanisms remain immature. Robustly and efficiently mapping physical domain representations such as spectrograms and point clouds to the word vector space of large language models still lacks a unified and reusable technical paradigm. Existing attempts mostly remain at the level of shallow feature matching, failing to establish a complete technical pathway spanning "dedicated encoding → tokenization → semantic alignment → efficient parameter fine-tuning," hindering the deep integration of millimeter-wave sensing and general semantic understanding capabilities.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] This invention provides an end-to-end method for constructing a large-scale millimeter-wave language model, aiming to address the problems of existing millimeter-wave sensing technologies, such as process fragmentation leading to information transmission bottlenecks, lack of effective joint modeling of the complementarity of heterogeneous modalities in spectrograms and point clouds, limited generalization ability and difficulty in achieving open set recognition, insufficient temporal modeling to coherently characterize continuous actions, and immature multimodal alignment mechanisms with large-scale language models. It achieves a unified technical path for constructing a general semantic understanding from raw millimeter-wave I / Q signals.

[0008] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0009] According to a first aspect of the present invention, a method for constructing an end-to-end millimeter-wave language large model is provided, the method comprising: The raw echo signal from the millimeter-wave radar is processed by a cascaded fast Fourier transform in the range, Doppler, and angle dimensions to generate range-Doppler spectra, range-azimuth spectra, and range-elevation spectra. The distance, azimuth and elevation angle information of the target point are extracted from the original echo signal using a target detection and angle estimation algorithm, and then converted into a time-series sparse point cloud sequence in Cartesian coordinate system. A parallel dual-branch feature extraction network is constructed, comprising a spectral encoder and a point cloud encoder. The spectral encoder branch extracts features from the range-Doppler spectrum, range-azimuth spectrum, and range-elevation spectrum to obtain spectral features, while the point cloud encoder branch extracts features from the point cloud sequence to obtain point cloud features. The extracted spectral features and point cloud features are then fused to form a unified multimodal representation. The multimodal representation is embedded into the word vector space of the pre-trained large language model; an end-to-end millimeter-wave language model is constructed, which includes the dual-branch feature extraction network and the pre-trained large language model; and the end-to-end millimeter-wave language model is fine-tuned based on a low-rank adaptive method, ultimately achieving end-to-end mapping from millimeter-wave raw signals to action category recognition or natural language description.

[0010] In some exemplary embodiments, the specific steps for generating the range-Doppler spectrum, the range-azimuth spectrum, and the range-elevation spectrum include: Perform range-dimensional FFT processing on the original echo signal along the fast time dimension to obtain the range profile:

[0011] in, Let N be the nth sampling point of the mth chirp, N be the number of sampling points, and k be the distance-frequency index. The distance-dimensional FFT result is processed along the slow time dimension using a Doppler-dimensional FFT to obtain the distance-Doppler spectrum:

[0012] in, For the number of chirps, For Doppler frequency index; Based on the phase difference of the signals received by the multi-antenna array, angle information is obtained through angle-dimensional FFT processing to generate range-azimuth and range-elevation spectra:

[0013]

[0014] in, and They represent the first The horizontal antenna and the first Distance-Doppler data for each vertical antenna; , These represent the number of antenna array elements in the horizontal and vertical directions, respectively. It is the azimuth angle. The pitch angle; For the signal wavelength, Antenna spacing; exponential term It characterizes the phase difference between antennas caused by the path difference.

[0015] In some exemplary embodiments, the specific steps for generating the temporal sparse point cloud sequence include: Clutter suppression is performed on the signal after range-dimensional FFT processing. Static background interference is eliminated by the moving average method to obtain the clutter-suppressed range profile. Doppler FFT processing is performed on the clutter-suppressed range profile to obtain the clutter-suppressed range-Doppler spectrum; On the range-Doppler spectrum after clutter suppression, a constant false alarm rate (CFAR) detection method or a Top-K selection method based on energy ranking is used to detect effective target points and obtain their range and velocity information. By utilizing the phase difference between the multi-antenna arrays at the transmitting and receiving ends, the azimuth and elevation angles of the effective target point are calculated using an angle estimation algorithm. The target point parameters in polar coordinates are converted into point cloud data in Cartesian coordinates, and the point cloud data of multiple consecutive frames are regularized to generate a time-series sparse point cloud sequence with a fixed number of points.

[0016] In some exemplary embodiments, the specific processing steps of the spectral encoder include: The three-dimensional convolutional layer divides the range-Doppler, range-azimuth, and range-elevation spectral maps into image block sequences respectively; A cross-attention mechanism is employed to enable information interaction and fusion among the feature sequences of the three spectra, thereby capturing cross-spectral correlation features: Simultaneously, learnable positional and temporal encodings are introduced to preserve the spatiotemporal structure information of the spectral data, ultimately outputting spectral features.

[0017] In some exemplary embodiments, the point cloud encoder's specific processing steps include: Key points are selected from single-frame point clouds of the temporal sparse point cloud sequence using the farthest point sampling method to obtain a set of key points; Multi-scale local regions are constructed centered on the key points, and geometric features of each local region are extracted by a multilayer perceptron with shared weights and max pooling operation, respectively. Aggregate local features at different scales to form a single-frame global point cloud feature vector; Add timestamp encoding to the single-frame global point cloud feature vector to aggregate temporal context information and output point cloud features.

[0018] In some exemplary embodiments, the fusion method of the spectral features and point cloud features is to perform projection fusion after feature stitching through a fully connected layer:

[0019] in, This indicates concatenation along the feature dimension. and For the weights and biases of the projection layer, This is the activation function.

[0020] In some exemplary embodiments, embedding the multimodal representation into the word vector space of a pre-trained large language model specifically involves: In the original word segmenter vocabulary of the pre-trained large language model, special markers <|mmwave_start|>, <|mmwave_end|>, and <|mmwave_pad|> are introduced to identify the start, end, and padding of millimeter-wave data. The multimodal representation output by the dual-branch feature extraction network is mapped to an embedding space consistent with the dimension of the pre-trained large language model text word vectors. In the model input sequence, the embedding vectors are replaced with the corresponding millimeter-wave special markers, thereby organizing the multimodal data into a unified sequence that can be processed by a large language model. In some exemplary embodiments, the low-rank adaptive method is specifically implemented by freezing most of the model parameters of the pre-trained large language model and injecting a trainable low-rank decomposition matrix only into the self-attention layer or feedforward neural network layer of the Transformer module.

[0021] in, These are the fine-tuned model weights. The weight matrix of the frozen pre-trained model; It is a trainable low-rank matrix. For the hidden layer dimension of the model, It is a low-rank number; During the fine-tuning process, maintain Unchanged, only updated and The parameters are optimized to efficiently adapt to millimeter-wave multimodal tasks.

[0022] According to a second aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the end-to-end millimeter-wave-language large model construction method described in the first aspect above.

[0023] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the end-to-end millimeter-wave-language large model construction method described in the first aspect is implemented.

[0024] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the end-to-end millimeter-wave language large model construction method described in the first aspect above by executing the executable instructions.

[0025] The end-to-end millimeter-wave language large-scale model construction method provided by the embodiments of this invention effectively overcomes the core bottlenecks of process fragmentation, information loss, and difficulties in multimodal fusion in traditional technologies, achieving direct mapping from raw I / Q signals to advanced semantic understanding. This method employs a dual-branch encoder architecture, deeply fusing range-Doppler spectrum, range-azimuth spectrum, range-elevation spectrum, and point cloud data through a cross-attention mechanism, fully leveraging the complementary advantages of multimodal information and significantly improving the accuracy and robustness of action recognition. By utilizing the powerful semantic capabilities of the pre-trained large-scale language model combined with low-rank adaptive fine-tuning technology, this invention exhibits excellent open-set recognition and zero-shot generalization capabilities, flexibly adapting to the continuously evolving task requirements in real-world scenarios. Temporal coding and serialization processing ensure the coherent depiction of continuous actions, effectively capturing subtle dynamic details and overcoming the limitations of traditional single-frame analysis. The overall design not only provides high-precision perception performance but also offers a feasible engineering path for practical deployment through a modular structure and efficient parameter fine-tuning strategy, showing broad application prospects in fields such as smart homes, in-vehicle interaction, and medical monitoring.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0028] Figure 1 This is a flowchart of the millimeter-wave data preprocessing method of the present invention.

[0029] Figure 2 The flowchart shows the millimeter-wave spectral feature extraction method of this invention.

[0030] Figure 3 The flowchart shows the millimeter-wave point cloud feature extraction method of this invention.

[0031] Figure 4 This is a flowchart of an end-to-end millimeter-wave multimodal behavior recognition system. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0033] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] To address the shortcomings and deficiencies of existing technologies, this example implementation provides an end-to-end millimeter-wave language large-scale model construction method. Starting with the raw I / Q signals from a millimeter-wave radar, cascaded spectralization of range / Doppler / angle and target geometric analysis are sequentially performed to obtain RD / RA / RE spectra and temporally sparse point clouds. Subsequently, spatiotemporal / geometric features are extracted through two branches: a spectral encoder and a point cloud encoder. A unified multimodal representation is formed through cross-spectral interaction and projection fusion. Finally, this representation is fed into a pre-trained large-scale language model via dedicated tokenization and embedding mapping, and low-rank adaptive instruction fine-tuning is performed to complete the end-to-end mapping from raw signals to action categories or natural language descriptions. This scheme provides a unified engineering path to address challenges such as process fragmentation, cross-spectral and geometric heterogeneity, and insufficient temporal consistency and alignment mechanisms.

[0035] Specifically, this may include the following steps: Step 1: Signal Processing and Data Generation: Perform cascaded Fast Fourier Transform (FFT) processing on the raw echo I / Q signals from the millimeter-wave radar in the range, Doppler, and angular dimensions to generate range-Doppler spectra. Distance-azimuth spectrum and distance-elevation spectrum Simultaneously, using target detection and angle estimation algorithms, the distance, azimuth, and elevation angle information of the target point are extracted from the original echo I / Q signal and converted into a time-series sparse point cloud sequence in Cartesian coordinates. ; Step 2: Multimodal Feature Extraction and Fusion: Construct a parallel dual-branch feature extraction network, wherein the spectral encoder branch pairs with the distance-Doppler spectrum. Distance-azimuth spectrum and distance-elevation spectrum Feature extraction is performed, and the point cloud encoder branch processes the point cloud sequence. Feature extraction is performed; the extracted spectral features are fused with point cloud features to form a unified multimodal representation. Step 3: Large Language Model Integration and Fine-tuning: Using a tokenization process adapted to millimeter-wave data, the multimodal representations are mapped and embedded into the word vector space of a pre-trained large language model; an end-to-end millimeter-wave-language large model is constructed, including the dual-branch feature extraction network and the pre-trained large language model; and the end-to-end millimeter-wave-language large model is fine-tuned based on a low-rank adaptive method, ultimately achieving end-to-end mapping from raw millimeter-wave signals to action category recognition or natural language description.

[0036] The steps in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.

[0037] Example 1 This embodiment provides an end-to-end millimeter-wave language large model construction method. It utilizes millimeter-wave radar's TDM-MIMO technology to acquire three-dimensional information about the environment, including distance, velocity, and angle. A deep learning network is used to extract spatiotemporal features from the radar spectrogram and point cloud, respectively. These physical signal features are then mapped into semantic embeddings that the large language model can understand, thereby leveraging the model's reasoning capabilities for behavior recognition. Its main components include radar signal acquisition and three-dimensional FFT preprocessing, dynamic point cloud generation and clutter suppression, dual-stream feature extraction and fusion of the spectrogram and point cloud, and large language model adaptation based on low-rank fine-tuning. This invention uses TI AWR6843AOP radar to acquire data. First, the echo signal is processed by multidimensional FFT to generate a range-Doppler-angle spectrum and a sparse point cloud sequence. Second, a dual-path feature extraction network is constructed to capture the associated features of the multi-view spectrum using a cross-attention mechanism, and the geometric features of the point cloud are extracted through multi-scale local aggregation. Finally, the fused multimodal features are mapped to the embedding space of a pre-trained large language model through linear projection, and end-to-end efficient behavior recognition is achieved by injecting a trainable low-rank decomposition matrix (LoRA).

[0038] Step 1: Employ the TI AWR6843AOP. A three-channel sequential time-division multiplexing transmission and a four-channel parallel receiving and sampling system are used, with each frame consisting of multiple chirps. The radar acquires range-velocity-angle three-dimensional information simultaneously. TDM-MIMO is combined to form 12 virtual channels, providing a longer equivalent aperture in the azimuth direction and basic elevation resolution. The sampling duration is 5 seconds, the frame rate is 20 fps, and each frame contains P = 64 chirps. The radar is positioned at an altitude of 1.2 m and a distance of 2–4 m from the target.

[0039] Step 2: As Figure 1 The signal preprocessing workflow in the process first performs range-dimensional FFT processing on the original echo signal along the fast time dimension to obtain the range profile:

[0040] in, Let N be the nth sampling point of the m-th chirp, where N is the number of sampling points. k For distance-frequency index; Step 3: Perform Doppler-dimensional FFT processing on the distance-dimensional FFT result along the slow time dimension to obtain the distance-Doppler spectrum:

[0041] in, For the number of chirps, For Doppler frequency index; Step 4: Based on the phase difference of the received signals from the multi-antenna array, generate range-azimuth spectra by performing angular-dimensional FFT processing. With distance-elevation spectrum :

[0042] in, and They represent the first The horizontal antenna and the first Distance-Doppler data for each vertical antenna; , These represent the number of antenna array elements in the horizontal and vertical directions, respectively. It is the azimuth angle. The pitch angle; For the signal wavelength, Antenna spacing; exponential term It characterizes the phase difference between antennas caused by the path difference.

[0043] Step 5: Clutter suppression is applied to the range-dimensional FFT-processed signal. A moving average method is used to eliminate static background interference, resulting in a clutter-suppressed range profile.

[0044] in, The signal value of the m-th distance cell. This is the size of the sliding window. Based on the above... Perform Doppler FFT processing to obtain the clutter-suppressed range-Doppler spectrum (updated). ); Step 6: In the updated The method employs constant false alarm rate (CFAR) detection or a Top-K selection method based on energy ranking. First, the energy threshold for retaining the target number is calculated.

[0045] in, Energy threshold This represents the total number of units. To retain the target number, a set of valid target points is then selected based on the energy threshold. And extract the distance index of each target point. Speed ​​Index and energy intensity :

[0046] in, A set of indices representing valid targets. Number the target; Step 7: Utilizing the phase difference between the multi-antenna arrays at the transmitting and receiving ends, calculate the azimuth angle of each effective target point using an angle estimation algorithm. With pitch angle :

[0047] in, The phase difference between adjacent antennas in the horizontal direction. This represents the phase difference between adjacent antennas in the vertical direction; Antenna spacing, The wavelength of the signal; The distance index, velocity index, and energy intensity extracted in step 6, along with the azimuth and pitch angles calculated in this step, are used to construct the target point parameters in polar coordinates. These parameters are then converted into five-dimensional point cloud data in Cartesian coordinates, containing coordinate, velocity, and energy information. The point cloud data of multiple consecutive frames is regularized to generate a time-series sparse point cloud sequence with a fixed number of points.

[0048] Step 8: As Figure 2In order to extract features from the spectral map, a three-dimensional convolutional layer is used to convert the distance-Doppler spectrum. Distance-azimuth spectrum and distance-elevation spectrum The image is divided into image patch sequences to obtain the initial feature sequence. :

[0049] in, This represents any type of input spectrum. and These are the kernel weights and biases, respectively. Step 9: Employ a cross-attention mechanism to enable information exchange and fusion between the feature sequences of the three spectra, in order to capture cross-spectral correlation features:

[0050] in, From source sequence The query matrix, and From the target sequence The key matrix and value matrix, The dimension of the key vector; Step 10: Introduce learnable positional and temporal encodings to preserve the spatiotemporal structure information of the spectral data, and finally output the spectral features. :

[0051] in, For attention output features, , These are location encoding and time encoding, respectively.

[0052] Step 11: As Figure 3 In order to extract features from the point cloud, the farthest point sampling method is used from the temporal sparse point cloud sequence. Key points are selected from a single frame point cloud to obtain a key point set. :

[0053] in The number of key points is set. This indicates the farthest point sampling operation; Step 12: Based on the aforementioned key points Multi-scale local regions are constructed around the center, and the geometric features of each local region are extracted using a multilayer perceptron with shared weights and max pooling operations:

[0054] in, For Centered on, with radius The neighborhood point set, It consists of a fully connected layer and an activation function. It is a local geometric feature; Then, local features at different scales are aggregated to form a single-frame global point cloud feature vector. :

[0055] in, For scale quantity, For the first Aggregate weights at each scale; Step 13: The final result is the global point cloud feature vector for the single frame. Add timestamp encoding to aggregate temporal context information and output point cloud features. :

[0056] in, This indicates a point-by-point concatenation or addition operation. This is a learnable temporal encoding matrix.

[0057] Step 14: The fusion method for spectral features and point cloud features is to perform projection fusion after feature stitching through a fully connected layer:

[0058] in, This indicates concatenation along the feature dimension. and For the weights and biases of the projection layer, This is the activation function.

[0059] Step 15: In the original word segmenter vocabulary of the pre-trained large language model, introduce special tags <|mmwave_start|>, <|mmwave_end|>, and <|mmwave_pad|> to identify the start, end, and padding of millimeter-wave data; Step 16: Utilize the multimodal representation output by the dual-branch feature extraction network described in step S2. This is mapped to an embedding space with the same dimension as the word vectors in the pre-trained large language model text:

[0060] in, For learnable projective weight matrix, For bias vectors, Embed the mapped millimeter-wave features into a vector; In the model input sequence, the embedding vector Inserting between corresponding millimeter-wave special markers, thereby organizing multimodal data into a unified sequence that can be processed by large language models.

[0061] Step 17: Figure 4 The flowchart for the end-to-end millimeter-wave multimodal behavior recognition system shows that most of the model parameters of the pre-trained large language model are frozen, and only trainable low-rank decomposition matrices are injected into the self-attention layer or feedforward neural network layer of the Transformer module.

[0062] in, These are the fine-tuned model weights. The weight matrix of the frozen pre-trained model; and It is a trainable low-rank matrix. For the hidden layer dimension of the model, It is a low-rank rank and During the fine-tuning process, maintain Unchanged, only updated and The parameters are optimized to efficiently adapt to millimeter-wave multimodal tasks.

[0063] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0064] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0065] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.

Claims

1. A method for constructing an end-to-end millimeter-wave language large model, characterized in that, The method includes: The raw echo signal from the millimeter-wave radar is processed by a cascaded fast Fourier transform in the range, Doppler, and angle dimensions to generate range-Doppler spectra, range-azimuth spectra, and range-elevation spectra. The distance, azimuth and elevation angle information of the target point are extracted from the original echo signal using a target detection and angle estimation algorithm, and then converted into a time-series sparse point cloud sequence in Cartesian coordinate system. A parallel dual-branch feature extraction network is constructed, comprising a spectral encoder and a point cloud encoder. The spectral encoder branch extracts features from the range-Doppler spectrum, range-azimuth spectrum, and range-elevation spectrum to obtain spectral features, while the point cloud encoder branch extracts features from the point cloud sequence to obtain point cloud features. The extracted spectral features and point cloud features are then fused to form a unified multimodal representation. The multimodal representation is embedded into the word vector space of the pre-trained large language model; an end-to-end millimeter-wave language model is constructed, which includes the dual-branch feature extraction network and the pre-trained large language model; and the end-to-end millimeter-wave language model is fine-tuned based on a low-rank adaptive method, ultimately achieving end-to-end mapping from millimeter-wave raw signals to action category recognition or natural language description.

2. The method according to claim 1, characterized in that, The specific steps for generating the range-Doppler spectrum, the range-azimuth spectrum, and the range-elevation spectrum include: Perform range-dimensional FFT processing on the original echo signal along the fast time dimension to obtain the range profile: in, Let N be the nth sampling point of the mth chirp, N be the number of sampling points, and k be the distance-frequency index. The distance-dimensional FFT result is processed along the slow time dimension using a Doppler-dimensional FFT to obtain the distance-Doppler spectrum: in, M For the number of chirps, l For Doppler frequency index; Based on the phase difference of the signals received by the multi-antenna array, angle information is obtained through angle-dimensional FFT processing to generate range-azimuth and range-elevation spectra: in, and They represent the first The horizontal antenna and the first Distance-Doppler data for each vertical antenna; , These represent the number of antenna array elements in the horizontal and vertical directions, respectively. It is the azimuth angle. The pitch angle; For the signal wavelength, Antenna spacing; exponential term It characterizes the phase difference between antennas caused by the path difference.

3. The method according to claim 1, characterized in that, The specific steps for generating the temporal sparse point cloud sequence include: Clutter suppression is performed on the signal after range-dimensional FFT processing. Static background interference is eliminated by the moving average method to obtain the clutter-suppressed range profile. Doppler FFT processing is performed on the clutter-suppressed range profile to obtain the clutter-suppressed range-Doppler spectrum; On the range-Doppler spectrum after clutter suppression, a constant false alarm rate (CFAR) detection method or a Top-K selection method based on energy ranking is used to detect effective target points and obtain their range and velocity information. By utilizing the phase difference between the multi-antenna arrays at the transmitting and receiving ends, the azimuth and elevation angles of the effective target point are calculated using an angle estimation algorithm. The target point parameters in polar coordinates are converted into point cloud data in Cartesian coordinates, and the point cloud data of multiple consecutive frames are regularized to generate a time-series sparse point cloud sequence with a fixed number of points.

4. The method according to claim 1, characterized in that, The specific processing steps of the spectral encoder include: The three-dimensional convolutional layer divides the range-Doppler, range-azimuth, and range-elevation spectral maps into image block sequences respectively; A cross-attention mechanism is employed to enable information interaction and fusion among the feature sequences of the three spectra, thereby capturing cross-spectral correlation features: Simultaneously, learnable positional and temporal encodings are introduced to preserve the spatiotemporal structure information of the spectral data, ultimately outputting spectral features.

5. The method according to claim 1, characterized in that, The specific processing steps of the point cloud encoder include: Key points are selected from single-frame point clouds of the temporal sparse point cloud sequence using the farthest point sampling method to obtain a set of key points; Multi-scale local regions are constructed centered on the key points, and geometric features of each local region are extracted by a multilayer perceptron with shared weights and max pooling operation, respectively. Aggregate local features at different scales to form a single-frame global point cloud feature vector; Add timestamp encoding to the single-frame global point cloud feature vector to aggregate temporal context information and output point cloud features.

6. The method according to claim 1, characterized in that, The fusion method of the spectral features and point cloud features is to perform projection fusion after feature stitching through a fully connected layer: in, This indicates concatenation along the feature dimension. and For the weights and biases of the projection layer, This is the activation function.

7. The method according to claim 1, characterized in that, Embedding the multimodal representation into the word vector space of the pre-trained large language model specifically involves: In the original word segmenter vocabulary of the pre-trained large language model, special markers <|mmwave_start|>, <|mmwave_end|>, and <|mmwave_pad|> are introduced to identify the start, end, and padding of millimeter-wave data. The multimodal representation output by the dual-branch feature extraction network is mapped to an embedding space consistent with the dimension of the pre-trained large language model text word vectors. In the model input sequence, the embedding vectors are replaced with the corresponding millimeter-wave special markers, thereby organizing the multimodal data into a unified sequence that can be processed by a large language model.

8. The method according to claim 1, characterized in that, The low-rank adaptive method is specifically implemented by freezing most of the model parameters of the pre-trained large language model and injecting a trainable low-rank decomposition matrix only into the self-attention layer or feedforward neural network layer of the Transformer module. in, These are the fine-tuned model weights. The weight matrix of the frozen pre-trained model; It is a trainable low-rank matrix. For the hidden layer dimension of the model, It is a low-rank number; During the fine-tuning process, maintain Unchanged, only updated and The parameters are optimized to efficiently adapt to millimeter-wave multimodal tasks.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for constructing an end-to-end millimeter-wave-language large model as described in any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for constructing an end-to-end millimeter-wave-language large model as described in any one of claims 1 to 8.

Images