A millimeter wave human action understanding method and device based on a large language model
By combining a large language model with a spatiotemporal motion encoder and a projection layer, the modal differences in semantic understanding of millimeter-wave human behavior perception systems are solved, achieving stable and generalized semantic understanding of human motion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2026-03-24
- Publication Date
- 2026-06-05
AI Technical Summary
Existing millimeter-wave human behavior perception systems struggle to characterize the continuity, transitions, and fine-grained semantics of human movement. Furthermore, the modal differences between millimeter-wave signals and natural language lead to unstable semantic understanding.
By employing a large language model combined with a spatiotemporal motion encoder and a projection layer, motion embedding features are extracted by acquiring millimeter-wave point cloud sequences and mapped to the input semantic space of the large language model to achieve semantic alignment, ultimately outputting a text describing human motion.
It achieves stable and generalizable semantic understanding of human motion, overcomes modal differences, and improves fine-grained behavioral semantic understanding capabilities.
Smart Images
Figure CN122157364A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of IoT millimeter-wave sensing and artificial intelligence interdisciplinary technology, and in particular to a millimeter-wave human motion understanding method and device based on a large language model. Background Technology
[0002] With the development of smart IoT technology, the demand for non-invasive, high-precision sensing of human movement is increasing. While traditional cameras can provide rich visual information, they pose serious risks of privacy leaks; wearable devices, on the other hand, interfere with the user experience. In contrast, millimeter-wave radar, with its strong penetration, lack of lighting limitations, and the fact that it does not require wearable devices, is gradually becoming a highly promising non-contact sensing modality.
[0003] However, current mainstream millimeter-wave human behavior perception systems generally follow a "closed set classification" paradigm, mapping continuous dynamic human motion to a limited number of predefined labels, such as "walking" or "sitting," which makes it difficult to characterize the continuity, transitions, and fine-grained semantics between movements. In contrast, millimeter-wave point clouds preserve the three-dimensional spatial distribution and motion geometry of the human body, giving them a natural advantage in fine-grained behavioral semantic understanding. Therefore, it is worth considering using millimeter-wave point clouds to achieve semantic understanding of human motion.
[0004] Therefore, how to achieve semantic understanding of human motion based on millimeter-wave point clouds is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] This application provides a method and apparatus for understanding human motion based on a large language model in millimeter waves, which can realize semantic understanding of human motion based on millimeter wave point clouds, thereby improving the semantic understanding capability of human motion.
[0006] This application provides a millimeter-wave human motion understanding method based on a large language model, including: Obtain the millimeter-wave point cloud sequence of the target object; The millimeter-wave point cloud sequence is input into a spatiotemporal motion encoder, and the spatiotemporal motion encoder extracts the millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence; wherein, the spatiotemporal motion encoder is obtained by training the motion encoder based on millimeter-wave point cloud sequence samples; The millimeter-wave motion embedding features are input into the projection layer, and the millimeter-wave motion embedding features are mapped to the input semantic space of the large language model through the projection layer to obtain the millimeter-wave motion semantic features. The millimeter-wave motion semantic features and the preset semantic understanding prompt text features are concatenated, and the concatenated target feature sequence is input into the large language model. The large language model then outputs the human motion description text of the target object. The large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples.
[0007] According to the millimeter-wave human motion understanding method based on a large language model provided in this application, the target feature sequence sequentially includes: The features of the semantic understanding prompt text, the marking features of the starting position of the millimeter wave motion semantic features, the marking features of the ending position of the millimeter wave motion semantic features, and the features of the guiding text used to guide the large language model to output the human motion description text.
[0008] According to the millimeter-wave human motion understanding method based on a large language model provided in this application, the spatiotemporal motion encoder includes a convolutional layer and an encoder. The step of inputting the millimeter-wave point cloud sequence into the spatiotemporal motion encoder and extracting the millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder includes: The millimeter-wave point cloud sequence is convolved by the convolutional layer to extract the initial embedding features of the millimeter-wave point cloud sequence; The initial embedding features are compressed and encoded by the encoder to obtain the millimeter-wave motion embedding features.
[0009] According to the millimeter-wave human motion understanding method based on a large language model provided in this application, the motion encoder includes a convolutional layer and an autoencoder. The spatiotemporal motion encoder is trained based on millimeter-wave point cloud sequence samples, including: The millimeter-wave point cloud sequence samples are input into the convolutional layer, and the millimeter-wave point cloud sequence samples are convolved by the convolutional layer to extract the initial embedding features of the millimeter-wave point cloud sequence samples. The initial embedding features are randomly masked to obtain the masked embedding features; The masked embedded features are input into the autoencoder, and the model parameters of the convolutional layer and the autoencoder are jointly optimized. The trained autoencoder and convolutional layer are then extracted to obtain the spatiotemporal motion encoder.
[0010] According to the millimeter-wave human motion understanding method based on a large language model provided in this application, the autoencoder includes an encoder and a decoder. The method involves inputting the masked embedded features into the autoencoder, jointly optimizing the model parameters of the convolutional layer and the autoencoder, and extracting the trained autoencoder and convolutional layer to obtain the spatiotemporal motion encoder, comprising: The masked embedding features are input into the encoder, and the encoder compresses and encodes the masked embedding features to obtain millimeter-wave motion embedding features. The millimeter-wave motion embedding feature is input into the decoder, and the millimeter-wave motion embedding feature is reconstructed by the decoder to obtain the reconstructed millimeter-wave motion embedding feature; Based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, the model parameters of the convolutional layer, the encoder, and the decoder are jointly optimized, and the trained autoencoder and convolutional layer are extracted to obtain the spatiotemporal motion encoder.
[0011] According to the millimeter-wave human motion understanding method based on a large language model provided in this application, the method involves jointly optimizing the model parameters of the convolutional layer, the encoder, and the decoder based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, including: Based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, a first loss function is constructed corresponding to the millimeter-wave point cloud sequence samples. Based on the first loss function, the model parameters of the convolutional layer, the encoder, and the decoder are jointly optimized.
[0012] According to the millimeter-wave human motion understanding method based on a large language model provided in this application, the first loss function corresponding to the millimeter-wave point cloud sequence sample is constructed based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, including: based on Construct the first loss function; in, This represents the first loss function. This indicates the number of millimeter-wave point cloud sequence samples. express The first of the millimeter-wave point cloud sequence samples A millimeter-wave point cloud sequence sample This represents the initial embedding feature. This represents the reconstructed millimeter-wave motion embedding features. This represents the features in the initial embedded features. This refers to the features in the reconstructed millimeter-wave motion embedding features.
[0013] According to the millimeter-wave human motion understanding method based on a large language model provided in this application, the large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples, including: The motion embedding features corresponding to the millimeter-wave point cloud sequence samples are concatenated with the command text features, and the concatenated target features are input into the initial large language model. The initial large language model then outputs the predicted human motion description text. Based on the target features, the predicted human motion description text, and the real human motion description text, a second loss function is constructed corresponding to the millimeter-wave point cloud sequence samples. Based on the second loss function, the model parameters of the initial large language model are iteratively updated until the large language model is trained.
[0014] According to the millimeter-wave human motion understanding method based on a large language model provided in this application, the second loss function corresponding to the millimeter-wave point cloud sequence sample is constructed based on the target features, the predicted human motion description text, and the real human motion description text, including: based on Construct the second loss function; in, This represents the second loss function. This indicates the number of lexical units in the text describing the actual human movement. This represents the set of lexical units included in the large language model. This represents the word at position t in the text describing the actual human motion. Indicates an indicator function, if equals the lexicon in the lexicon set ,but =1, otherwise, =0, This represents the word at position t in the predicted human motion description text. This represents the model parameters of the large language model. This represents the target feature. Indicates that given the target feature Given the lexical units at the first t-1 positions of the previously generated real human motion description text, the lexical unit at the t-th position predicted by the large language model is a lexical unit. The probability of.
[0015] This application also provides a millimeter-wave human motion understanding device based on a large language model, including: The acquisition unit is used to acquire the millimeter-wave point cloud sequence of the target object; An extraction unit is used to input the millimeter-wave point cloud sequence into a spatiotemporal motion encoder, and extract millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder; wherein, the spatiotemporal motion encoder is obtained by training the motion encoder based on millimeter-wave point cloud sequence samples; The mapping unit is used to input the millimeter-wave motion embedding features into the projection layer, and to map the millimeter-wave motion embedding features to the input semantic space of the large language model through the projection layer to obtain the millimeter-wave motion semantic features. The processing unit is used to concatenate the millimeter wave motion semantic features and the prompt text features of the preset semantic understanding prompt text, and input the concatenated target feature sequence into the large language model, and output the human motion description text of the target object through the large language model; The large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples.
[0016] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the millimeter-wave human motion understanding method based on a large language model as described in any of the preceding claims.
[0017] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the millimeter-wave human motion understanding method based on a large language model as described in any of the preceding claims.
[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the millimeter-wave human motion understanding method based on a large language model as described in any of the preceding claims.
[0019] This application provides a method and apparatus for understanding millimeter-wave human motion based on a large language model. It acquires a millimeter-wave point cloud sequence of the target object and inputs it into a spatiotemporal motion encoder. The spatiotemporal motion encoder extracts millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence, thereby extracting robust spatiotemporal features with motion continuity from sparse millimeter-wave point clouds. The millimeter-wave motion embedding features are then input into a projection layer, which maps them to the input semantic space of the large language model to obtain millimeter-wave motion semantic features. This overcomes the significant modal differences between millimeter-wave signals and natural language, achieving stable and generalizable semantic alignment. The millimeter-wave motion semantic features are then concatenated with the features of a pre-defined semantic understanding prompt text. The resulting target feature sequence is input into the large language model, which outputs a description of the human motion of the target object. This combination of the spatiotemporal motion encoder, projection layer, and large language model enables semantic understanding of human motion based on millimeter-wave point clouds, thereby improving the semantic understanding capability of human motion. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a flowchart illustrating a millimeter-wave human motion understanding method based on a large language model, provided as an embodiment of this application.
[0022] Figure 2 This is a schematic diagram of a framework for semantic understanding of human actions provided in an embodiment of this application.
[0023] Figure 3 This is a schematic diagram illustrating a process for training a motion encoder based on millimeter-wave point cloud sequence samples, as provided in an embodiment of this application.
[0024] Figure 4 This is a schematic diagram of the structure of a motion encoder provided in an embodiment of this application.
[0025] Figure 5 This is a schematic diagram of a framework for training a large language model, provided in an embodiment of this application.
[0026] Figure 6 This is a schematic diagram showing a comparison between a real human motion description text and a human motion description text predicted by this application, provided in an embodiment of this application.
[0027] Figure 7 This is a schematic diagram comparing another real human motion description text provided in an embodiment of this application with the human motion description text predicted by this application.
[0028] Figure 8 This is a schematic diagram of the structure of a millimeter-wave human motion understanding device based on a large language model, provided in an embodiment of this application.
[0029] Figure 9 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0031] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0032] Current mainstream millimeter-wave human behavior perception systems generally follow a "closed-set classification" paradigm, mapping continuous dynamic human movement to a limited number of predefined labels, such as "walking" or "sitting," which makes it difficult to characterize the continuity, transitions, and fine-grained semantics between movements. In contrast, millimeter-wave point clouds preserve the three-dimensional spatial distribution and motion geometry of the human body, giving them a natural advantage in fine-grained behavioral semantic understanding. Therefore, it is worth considering using millimeter-wave point clouds to achieve semantic understanding of human movement.
[0033] However, when using millimeter-wave point clouds to achieve semantic understanding of human motion, the following problems arise: (1) How to extract robust spatiotemporal features with motion continuity from sparse millimeter-wave point clouds? (2) How to overcome the huge modal differences between millimeter wave signals and natural language, achieve stable and generalizable semantic alignment, and thus realize semantic understanding of human movement? Based on this, this application provides a method for understanding millimeter-wave human motion based on a large language model. It acquires a millimeter-wave point cloud sequence of the target object and inputs it into a spatiotemporal motion encoder. The spatiotemporal motion encoder extracts millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence, thereby extracting robust spatiotemporal features with motion continuity from sparse millimeter-wave point clouds. The millimeter-wave motion embedding features are then input into a projection layer, which maps them to the input semantic space of the large language model to obtain millimeter-wave motion semantic features. This overcomes the significant modal differences between millimeter-wave signals and natural language, achieving stable and generalizable semantic alignment. The millimeter-wave motion semantic features are then concatenated with the features of a preset semantic understanding prompt text. The resulting target feature sequence is input into the large language model, which outputs a description of the target object's human motion. By combining the spatiotemporal motion encoder, projection layer, and large language model, semantic understanding of human motion based on millimeter-wave point clouds can be achieved, thereby improving the semantic understanding capability of human motion.
[0034] For example, the millimeter-wave human action understanding method based on a large language model provided in this application embodiment can be applied to human behavior understanding in smart home scenarios, human behavior understanding in health monitoring scenarios, and human behavior understanding in privacy-sensitive environment scenarios.
[0035] It is understood that the execution entity of the millimeter-wave human motion understanding method based on a large language model provided in this application can be a computer, a server, or a specially set millimeter-wave human motion semantic understanding device, such as an intelligent robot or other electronic device. It can also be a millimeter-wave human motion understanding device based on a large language model set in such electronic device. The millimeter-wave human motion understanding device based on a large language model can be implemented by software, hardware, or a combination of both, and can be set according to actual needs.
[0036] The millimeter-wave human motion understanding method based on a large language model provided in this application will be described in detail below through several specific embodiments. It is understood that these specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0037] Figure 1 A flowchart illustrating a millimeter-wave human motion understanding method based on a large language model, provided as an embodiment of this application, is shown below. Figure 1 As shown, this millimeter-wave human motion understanding method based on a large language model may include: S101. Obtain the millimeter-wave point cloud sequence of the target object.
[0038] Among them, the millimeter-wave point cloud sequence refers to a set of 3D point cloud frames arranged in chronological order generated by continuous scanning of millimeter-wave radar. Each frame contains the spatial coordinates of multiple scattering points on the human body surface, which is used to depict the dynamic movement process of the human body.
[0039] For example, the millimeter-wave point cloud sequence of the target object can be obtained through hardware testing or through simulation generation. The specific settings can be configured according to actual needs.
[0040] Taking the acquisition of millimeter-wave point cloud sequences of target objects through hardware measurement as an example, a frequency-modulated continuous wave (FMCW) radar can be used to perform a real scan of the target object, and a two-dimensional fast Fourier transform can be performed on the received echo signals to obtain the range-Doppler spectrum. The target points are extracted using a constant false alarm rate (CFAR) detection algorithm, and the angle of arrival (DOA) is estimated using super-resolution algorithms such as digital beamforming (DBF) or multiple signal classification (MUSIC). This allows the calculation of the three-dimensional coordinates of each scattering point, generating the original millimeter-wave point cloud sequence.
[0041] To extract robust spatiotemporal features with motion continuity from sparse millimeter-wave point clouds, considering the sparsity and spatiotemporal discontinuity of millimeter-wave point clouds, such as the effect of specular reflection on signals from different parts of the human body, which causes them to appear and disappear intermittently in continuous time frames, this application introduces a transformer-based spatiotemporal motion encoder in its embodiments. For example, see [link to relevant documentation]. Figure 2 As shown, Figure 2 This is a schematic diagram of a framework for semantic understanding of human actions provided in an embodiment of this application, based on the acquisition of millimeter-wave point cloud sequences. , , ..., Then, the millimeter-wave point cloud sequence can be input into the spatiotemporal motion encoder, i.e., the following S102 is executed: S102. Input the millimeter-wave point cloud sequence into the spatiotemporal motion encoder, and extract the millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder; wherein, the spatiotemporal motion encoder is obtained by training the motion encoder based on the millimeter-wave point cloud sequence samples.
[0042] For example, in an embodiment of this application, the spatiotemporal motion encoder may include a convolutional layer and an encoder.
[0043] For example, the convolutional layer can be a Point 4D Convolution layer, which can be configured according to actual needs.
[0044] For example, the encoder can be a standard 6-layer transformer encoder, a pure Transformer variant encoder, or a hybrid architecture, such as a Convolutional Neural Network (CNN) + Transformer encoder, etc., which can be set according to actual needs.
[0045] For example, in this embodiment of the application, inputting a millimeter-wave point cloud sequence into a spatiotemporal motion encoder and extracting millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder may include: The millimeter-wave point cloud sequence is convolved by a convolutional layer to extract the initial embedding features of the millimeter-wave point cloud sequence; the initial embedding features are then compressed and encoded by an encoder to obtain the millimeter-wave motion embedding features.
[0046] After extracting millimeter-wave motion embedding features with motion continuity through a spatiotemporal motion encoder, in order to overcome the huge modal differences between millimeter-wave signals and natural language and achieve stable and generalizable semantic alignment, in this embodiment of the application, the millimeter-wave motion embedding features can be input into the projection layer, that is, the following S103 is executed, so as to realize the modal alignment between millimeter-wave signals and natural language through the projection layer, efficiently converting the millimeter-wave motion embedding features into millimeter-wave motion semantic features that can be recognized in the input semantic space of the large language model. At the same time, it can avoid introducing too many parameters that lead to overfitting or catastrophic forgetting.
[0047] S103. Input the millimeter-wave motion embedding features into the projection layer, and map the millimeter-wave motion embedding features to the input semantic space of the large language model through the projection layer to obtain the millimeter-wave motion semantic features.
[0048] The projection layer can be a lightweight projection layer, such as a projection layer with a single linear transformation or a projection layer of a multilayer perceptron (MLP), and can be set according to actual needs.
[0049] For example, a Large Language Model (LLM) can be a large language model that incorporates Low-Rank Adaptation (LoRA) technology. By introducing LoRA, a low-dimensional trainable matrix can be inserted into the attention layer of the large language model. W=BA, jointly optimizing the projection layer and LoRA data parameters, to achieve a fine understanding of human action semantics.
[0050] For example, the large language model can be a generative pre-trained Transformer (GPT) series model, or a T5 model or an LLaMA model, etc., which can be set according to actual needs.
[0051] By inputting millimeter-wave motion embedding features into the projection layer, modal alignment between millimeter-wave signals and natural language can be achieved through the projection layer. This efficiently transforms the millimeter-wave motion embedding features into millimeter-wave motion semantic features that can be recognized in the input semantic space of a large language model. At the same time, it can avoid introducing too many parameters that could lead to overfitting or catastrophic forgetting.
[0052] For example, in an embodiment of this application, when training the projection layer, the backbone and spatiotemporal motion encoder of the large language model can be frozen first, and only the projection layer can be trained. The millimeter-wave motion embedding features are mapped to the input semantic space of the large language model, and two special learnable tags are introduced. and ,in, Indicates the starting position of the semantic features of millimeter-wave motion. Representing the semantic features of millimeter wave motion, This indicates the end position of the semantic features of millimeter-wave motion. Only the projection layer parameters are optimized so that the large language model can accept millimeter-wave motion embedding features as "foreign language variants", thereby achieving modal alignment between millimeter-wave signals and natural language.
[0053] S104. The millimeter wave motion semantic features and the preset semantic understanding prompt text features are concatenated, and the concatenated target feature sequence is input into the large language model. The large language model outputs the human motion description text of the target object.
[0054] Among them, the large language model is trained based on the motion embedding features corresponding to millimeter-wave point cloud sequence samples and the real human motion description text corresponding to millimeter-wave point cloud sequence samples.
[0055] For example, in the embodiments of this application, the target feature sequence sequentially includes: features of semantic understanding prompt text, marker features of the starting position of millimeter wave motion semantic features, millimeter wave motion semantic features, marker features of the ending position of millimeter wave motion semantic features, and features of guiding text used to guide the large language model to output human motion description text.
[0056] For example, the format of the target feature sequence can be: .
[0057] in, Represents the target feature sequence. Features that indicate semantic understanding of the prompt text The marker feature representing the starting position of the semantic features of millimeter-wave motion. Representing the semantic features of millimeter wave motion, A marker feature indicating the end position of the semantic features of millimeter-wave motion. Features used to guide large language models in outputting text describing human motion.
[0058] For example, when obtaining the prompt text features of semantically understood prompt text, the semantically understood prompt text can be input into the instruction encoder, which then extracts the prompt text features. The instruction encoder can be the embedding layer built into a large language model, or it can be a standalone lightweight text encoder, etc., depending on the specific needs.
[0059] As can be seen, in this embodiment, by acquiring the millimeter-wave point cloud sequence of the target object and inputting the millimeter-wave point cloud sequence into a spatiotemporal motion encoder, the spatiotemporal motion encoder extracts the millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence to extract robust spatiotemporal features with motion continuity from sparse millimeter-wave point clouds. The millimeter-wave motion embedding features are then input into a projection layer, which maps the millimeter-wave motion embedding features to the input semantic space of a large language model to obtain millimeter-wave motion semantic features. This overcomes the huge modal differences between millimeter-wave signals and natural language, achieving stable and generalizable semantic alignment. The millimeter-wave motion semantic features are then concatenated with the prompt text features of a preset semantic understanding prompt text, and the concatenated target feature sequence is input into a large language model. The large language model outputs the human motion description text of the target object. By combining the spatiotemporal motion encoder, the projection layer, and the large language model, semantic understanding of human actions based on millimeter-wave point clouds can be achieved, thereby improving the semantic understanding ability of human motion.
[0060] Based on any of the above embodiments, for example, in this application embodiment, the millimeter-wave point cloud sequence includes point clouds of multiple millimeter-wave radar frames. In the above S101, when obtaining the millimeter-wave point cloud sequence of the target object, the initial millimeter-wave point cloud sequence of the target object can be obtained first. For each initial millimeter-wave point cloud sequence, data preprocessing can be performed on the initial millimeter-wave point cloud sequence. If the number of point clouds in each radar frame of the initial millimeter-wave point cloud sequence is less than the preset number of point clouds, a certain number of millimeter-wave point clouds are randomly copied, and the copied point cloud coordinates are... Random offsets are added to each of the above to expand the number of point clouds in the radar frame until the number of point clouds in the preprocessed millimeter-wave point cloud sequence reaches the preset number of point clouds. If the number of point clouds in each radar frame of the initial millimeter-wave point cloud sequence exceeds the preset number of point clouds, then a certain number of point clouds are randomly sampled to ensure that the number of point clouds in the preprocessed millimeter-wave point cloud sequence is the preset number of point clouds. .
[0061] The preset point cloud count can be set according to actual needs. For example, in this embodiment, the preset point cloud count can be denoted as... .
[0062] The resulting millimeter-wave point cloud sequence is obtained by supplementing the point cloud data as described above. ,in, This represents the number of frames in a series of consecutive millimeter-wave point clouds. This indicates the preset number of point clouds per frame. This unification of the inter-frame point count dimension of the millimeter-wave point cloud sequence not only meets the input requirements of the spatiotemporal motion encoder, but also enhances data diversity through random copying and offsetting, thereby improving the robustness of the spatiotemporal motion encoder to sparse point clouds.
[0063] Based on any of the above embodiments, for example, in this application embodiment, the motion encoder includes a convolutional layer and an autoencoder. The specific implementation of the spatiotemporal motion encoder, trained based on millimeter-wave point cloud sequence samples, can be found below. Figure 3 The example shown.
[0064] Figure 3 This application provides a schematic diagram of a process for training a motion encoder based on millimeter-wave point cloud sequence samples. For example, the reference... Figure 3 As shown, the method may include: S301. Input the millimeter-wave point cloud sequence samples into the convolutional layer, and perform convolution on the millimeter-wave point cloud sequence samples to extract the initial embedding features of the millimeter-wave point cloud sequence samples.
[0065] For example, see Figure 4 As shown, Figure 4This is a schematic diagram of the structure of a motion encoder provided in an embodiment of this application. The motion encoder includes a P4D convolutional layer and an autoencoder, which can input millimeter-wave point cloud sequence samples into the convolutional layer and perform convolution on the millimeter-wave point cloud sequence samples to extract the initial embedding features of the millimeter-wave point cloud sequence samples.
[0066] In this embodiment, an active embedding masking mechanism is introduced to randomly mask 50% of the initial embedding features during the training phase. This is used to simulate the loss of some observations caused by signal attenuation or specular reflection in a real environment, as shown in S302 below: S302. Randomly mask the initial embedded features to obtain the masked embedded features.
[0067] For example, in the embodiments of this application, a random independent masking method can be used to randomly mask the initial embedded features, or a temporal continuous block masking method or an adaptive importance sampling masking method can be used to randomly mask the initial embedded features. The specific method can be set according to actual needs.
[0068] S303. Input the masked embedded features into the autoencoder, jointly optimize the model parameters of the convolutional layer and the autoencoder, and extract the trained autoencoder and convolutional layer to obtain the spatiotemporal motion encoder.
[0069] For example, an autoencoder includes an encoder and a decoder, both of which can be a standard 6-layer transformer structure.
[0070] For example, in the embodiments of this application, when the masked embedding features are input into the autoencoder, the model parameters of the convolutional layer and the autoencoder are jointly optimized, and the trained autoencoder and convolutional layer are truncated to obtain the spatiotemporal motion encoder, the masked embedding features can be input into the encoder first, and the encoder can compress and encode the masked embedding features to obtain millimeter-wave motion embedding features; the millimeter-wave motion embedding features are then input into the decoder, and the decoder can reconstruct the millimeter-wave motion embedding features to obtain reconstructed millimeter-wave motion embedding features; then, based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, the model parameters of the convolutional layer, the encoder and the decoder are jointly optimized, and the trained autoencoder and convolutional layer are truncated to obtain the spatiotemporal motion encoder. That is, after training, the decoder is discarded, and the updated encoder is retained for extracting millimeter-wave motion embedding features.
[0071] For example, in the embodiments of this application, when jointly optimizing the model parameters of the convolutional layer, encoder and decoder based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, a first loss function corresponding to the millimeter-wave point cloud sequence sample can be constructed first based on the initial embedding features and the reconstructed millimeter-wave motion embedding features; and the model parameters of the convolutional layer, encoder and decoder can be jointly optimized based on the first loss function.
[0072] For example, in an embodiment of this application, the first loss function corresponding to the millimeter-wave point cloud sequence samples is constructed based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, which may include: based on Construct the first loss function.
[0073] in, Denotes the first loss function. This indicates the number of millimeter-wave point cloud sequence samples. express The first millimeter-wave point cloud sequence sample A millimeter-wave point cloud sequence sample This represents the initial embedded features. This represents the reconstruction of millimeter-wave motion embedding features. This represents the features in the initial embedded features. This represents the features embedded in the reconstructed millimeter-wave motion.
[0074] Based on any of the above embodiments, for example, in this application embodiment, the large language model is trained based on the target features corresponding to millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples. (See also...) Figure 5 As shown, Figure 5 This is a schematic diagram of a framework for training a large language model provided in an embodiment of this application. The motion embedding features corresponding to millimeter-wave point cloud sequence samples are concatenated with the command text features. The concatenated target features are then input into an initial large language model, which outputs predicted human motion description text. Based on the target features, the predicted human motion description text, and the actual human motion description text, a second loss function corresponding to the millimeter-wave point cloud sequence samples is constructed. Based on the second loss function, the model parameters of the initial large language model are iteratively updated until a large language model is trained.
[0075] For example, in an embodiment of this application, a second loss function is constructed based on target features, predicted human motion description text, and actual human motion description text, including: based on Construct the first loss function.
[0076] in, This represents the second loss function. The number of lexical units in a text describing actual human movement. This represents the set of lexical units included in a large language model. This represents the t-th word in a text describing real human motion. Indicates an indicator function, if equals the lexicon in the lexicon set ,but =1, otherwise, =0, This represents the t-th word in the text describing predicted human motion. The model parameters represent the large language model. Indicate target features, Indicates that given target features Given the lexical units at the first t-1 positions of the previously generated real human motion description text, the lexical unit at the t-th position predicted by the large language model is lexical unit t. The probability of.
[0077] After training the spatiotemporal motion encoder, mapping layer, and large language model respectively, the millimeter-wave point cloud sequence can be input into the spatiotemporal motion encoder. The spatiotemporal motion encoder extracts the millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence to extract robust spatiotemporal features with motion continuity from sparse millimeter-wave point clouds. The extracted millimeter-wave motion embedding features are then input into the projection layer, which maps the millimeter-wave motion embedding features to the input semantic space of the large language model to obtain millimeter-wave motion semantic features. This overcomes the huge modal differences between millimeter-wave signals and natural language, achieving stable and generalizable semantic alignment. The millimeter-wave motion semantic features are then concatenated with the features of the preset semantic understanding prompt text, and the concatenated target feature sequence is input into the large language model. The large language model outputs the human motion description text of the target object. In this way, by combining the spatiotemporal motion encoder, projection layer, and large language model, semantic understanding of human actions based on millimeter-wave point clouds can be achieved, thereby improving the semantic understanding ability of human motion.
[0078] The millimeter-wave human motion understanding method based on a large language model provided in this application can improve the semantic understanding ability of human motion. For example, see [link to relevant documentation]. Figure 6 and Figure 7 As shown, Figure 6 This is a schematic diagram illustrating the comparison between real human motion description text and predicted human motion description text provided in this application embodiment. Figure 7 A comparison diagram illustrating the relationship between real human motion description text and the predicted human motion description text provided in this application, in conjunction with an embodiment of this application. Figure 6 and Figure 7 It is evident that the human motion description text predicted by the millimeter-wave human motion understanding method based on a large language model provided in this application exhibits, on the one hand, a high degree of consistency with the real human motion description text in terms of motion category, direction, and body posture details. This verifies that millimeter-wave point cloud features, after cross-modal alignment, can be accurately interpreted by the large language model, effectively improving the semantic understanding capability of human motion. On the other hand, even under conditions that traditional vision methods struggle to handle, such as low light and partial occlusion, the millimeter-wave human motion understanding method based on a large language model provided in this application can still generate human motion description text that is almost identical to the real human motion description text. This demonstrates the non-invasive advantage of millimeter-wave perception and the robustness of point cloud features. Furthermore, detailed information from the real human motion description text, such as "arms outstretched," is successfully preserved in the predicted human motion description text. This indicates that the millimeter-wave human motion understanding method based on a large language model provided in this application does not lose key spatial geometric information due to modal conversion, overcoming the shortcomings of existing micro-Doppler methods.
[0079] The millimeter-wave human motion understanding device based on a large language model provided in this application is described below. The millimeter-wave human motion understanding device based on a large language model described below and the millimeter-wave human motion understanding method based on a large language model described above can be referred to and corresponded to each other.
[0080] Figure 8 A schematic diagram of a millimeter-wave human motion understanding device based on a large language model is provided for an embodiment of this application. For example, please refer to [link to relevant documentation]. Figure 8 As shown, the millimeter-wave human motion understanding device 80 based on a large language model may include: Acquisition unit 801 is used to acquire the millimeter-wave point cloud sequence of the target object; Extraction unit 802 is used to input the millimeter-wave point cloud sequence into a spatiotemporal motion encoder, and extract millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder; wherein, the spatiotemporal motion encoder is obtained by training the motion encoder based on millimeter-wave point cloud sequence samples; The mapping unit 803 is used to input the millimeter-wave motion embedding features into the projection layer, and to map the millimeter-wave motion embedding features to the input semantic space of the large language model through the projection layer to obtain millimeter-wave motion semantic features; Processing unit 804 is used to concatenate the millimeter wave motion semantic features and the prompt text features of the preset semantic understanding prompt text, and input the concatenated target feature sequence into the large language model, and output the human motion description text of the target object through the large language model; The large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples.
[0081] For example, in an embodiment of this application, the target feature sequence sequentially includes: The features of the semantic understanding prompt text, the marking features of the starting position of the millimeter wave motion semantic features, the marking features of the ending position of the millimeter wave motion semantic features, and the features of the guiding text used to guide the large language model to output the human motion description text.
[0082] For example, in this embodiment of the application, the spatiotemporal motion encoder includes a convolutional layer and an encoder. The extraction unit 802 is used to input the millimeter-wave point cloud sequence into the spatiotemporal motion encoder and extract the millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder, including: The millimeter-wave point cloud sequence is convolved by the convolutional layer to extract the initial embedding features of the millimeter-wave point cloud sequence; The initial embedding features are compressed and encoded by the encoder to obtain the millimeter-wave motion embedding features.
[0083] For example, in an embodiment of this application, the motion encoder includes a convolutional layer and an autoencoder, and the spatiotemporal motion encoder is trained based on millimeter-wave point cloud sequence samples, including: The millimeter-wave point cloud sequence samples are input into the convolutional layer, and the millimeter-wave point cloud sequence samples are convolved by the convolutional layer to extract the initial embedding features of the millimeter-wave point cloud sequence samples. The initial embedding features are randomly masked to obtain the masked embedding features; The masked embedded features are input into the autoencoder, and the model parameters of the convolutional layer and the autoencoder are jointly optimized. The trained autoencoder and convolutional layer are then extracted to obtain the spatiotemporal motion encoder.
[0084] For example, in an embodiment of this application, the autoencoder includes an encoder and a decoder. The step of inputting the masked embedded features into the autoencoder, jointly optimizing the model parameters of the convolutional layer and the autoencoder, and truncating the trained autoencoder and convolutional layer to obtain the spatiotemporal motion encoder includes: The masked embedding features are input into the encoder, and the encoder compresses and encodes the masked embedding features to obtain millimeter-wave motion embedding features. The millimeter-wave motion embedding feature is input into the decoder, and the millimeter-wave motion embedding feature is reconstructed by the decoder to obtain the reconstructed millimeter-wave motion embedding feature; Based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, the model parameters of the convolutional layer, the encoder, and the decoder are jointly optimized, and the trained autoencoder and convolutional layer are extracted to obtain the spatiotemporal motion encoder.
[0085] For example, in an embodiment of this application, the joint optimization of the model parameters of the convolutional layer, the encoder, and the decoder based on the initial embedding features and the reconstructed millimeter-wave motion embedding features includes: Based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, a first loss function is constructed corresponding to the millimeter-wave point cloud sequence samples. Based on the first loss function, the model parameters of the convolutional layer, the encoder, and the decoder are jointly optimized.
[0086] For example, in an embodiment of this application, constructing the first loss function corresponding to the millimeter-wave point cloud sequence sample based on the initial embedding features and the reconstructed millimeter-wave motion embedding features includes: based on Construct the first loss function; in, This represents the first loss function. This indicates the number of millimeter-wave point cloud sequence samples. express The first of the millimeter-wave point cloud sequence samples A millimeter-wave point cloud sequence sample This represents the initial embedding feature. This represents the reconstructed millimeter-wave motion embedding features. This represents the features in the initial embedded features. This refers to the features in the reconstructed millimeter-wave motion embedding features.
[0087] For example, in this embodiment of the application, the large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples, including: The motion embedding features corresponding to the millimeter-wave point cloud sequence samples are concatenated with the command text features, and the concatenated target features are input into the initial large language model. The initial large language model then outputs the predicted human motion description text. Based on the target features, the predicted human motion description text, and the real human motion description text, a second loss function is constructed corresponding to the millimeter-wave point cloud sequence samples. Based on the second loss function, the model parameters of the initial large language model are iteratively updated until the large language model is trained.
[0088] For example, in an embodiment of this application, the step of constructing a second loss function corresponding to the millimeter-wave point cloud sequence sample based on the target features, the predicted human motion description text, and the real human motion description text includes: based on Construct the second loss function; in, This represents the second loss function. This indicates the number of lexical units in the text describing the actual human movement. This represents the set of lexical units included in the large language model. This represents the word at position t in the text describing the actual human motion. Indicates an indicator function, if equals the lexicon in the lexicon set ,but =1, otherwise, =0, This represents the word at position t in the predicted human motion description text. This represents the model parameters of the large language model. This represents the target feature. Indicates that given the target feature Given the lexical units at the first t-1 positions of the previously generated real human motion description text, the lexical unit at the t-th position predicted by the large language model is a lexical unit. The probability of.
[0089] The millimeter-wave human motion understanding device 80 based on a large language model provided in this application embodiment can execute the technical solution of the millimeter-wave human motion understanding method based on a large language model in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the millimeter-wave human motion understanding method based on a large language model. Please refer to the implementation principle and beneficial effects of the millimeter-wave human motion understanding method based on a large language model. It will not be repeated here.
[0090] Figure 9 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 9As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communications bus 940. The processor 910 can call logical instructions in the memory 930 to execute a millimeter-wave human motion understanding method based on a large language model. This method includes: acquiring a millimeter-wave point cloud sequence of a target object; inputting the millimeter-wave point cloud sequence into a spatiotemporal motion encoder, and extracting millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder; wherein the spatiotemporal motion encoder is trained on a motion encoder based on millimeter-wave point cloud sequence samples; inputting the millimeter-wave motion embedding features into a projection layer, and mapping the millimeter-wave motion embedding features to the input semantic space of the large language model through the projection layer to obtain millimeter-wave motion semantic features; concatenating the millimeter-wave motion semantic features with the prompt text features of a preset semantic understanding prompt text, and inputting the concatenated target feature sequence into the large language model, and outputting a human motion description text of the target object through the large language model; wherein the large language model is trained on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples.
[0091] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0092] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the millimeter-wave human motion understanding method based on a large language model provided by the above methods. The method includes: acquiring a millimeter-wave point cloud sequence of a target object; inputting the millimeter-wave point cloud sequence into a spatiotemporal motion encoder; and extracting millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder; wherein the spatiotemporal motion encoder is trained on a motion encoder based on millimeter-wave point cloud sequence samples. The process involves: inputting the millimeter-wave motion embedding features into a projection layer, mapping the millimeter-wave motion embedding features to the input semantic space of a large language model, and obtaining millimeter-wave motion semantic features; concatenating the millimeter-wave motion semantic features with the prompt text features of a preset semantic understanding prompt text, and inputting the concatenated target feature sequence into the large language model, which then outputs the human motion description text of the target object; wherein the large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples.
[0093] In another aspect, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the millimeter-wave human motion understanding method based on a large language model provided by the above methods. This method includes: acquiring a millimeter-wave point cloud sequence of a target object; inputting the millimeter-wave point cloud sequence into a spatiotemporal motion encoder, and extracting millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder; wherein the spatiotemporal motion encoder is trained on a motion encoder based on millimeter-wave point cloud sequence samples; inputting the millimeter-wave motion embedding features into a projection layer, and mapping the millimeter-wave motion embedding features to the input semantic space of a large language model through the projection layer to obtain millimeter-wave motion semantic features; concatenating the millimeter-wave motion semantic features with the prompt text features of a preset semantic understanding prompt text, and inputting the concatenated target feature sequence into a large language model, and outputting a human motion description text of the target object through the large language model; wherein the large language model is trained on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples.
[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A millimeter-wave human motion understanding method based on a large language model, characterized in that, include: Obtain the millimeter-wave point cloud sequence of the target object; The millimeter-wave point cloud sequence is input into a spatiotemporal motion encoder, and the spatiotemporal motion encoder extracts the millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence; wherein, the spatiotemporal motion encoder is obtained by training the motion encoder based on the millimeter-wave point cloud sequence; The millimeter-wave motion embedding features are input into the projection layer, and the millimeter-wave motion embedding features are mapped to the input semantic space of the large language model through the projection layer to obtain the millimeter-wave motion semantic features. The millimeter-wave motion semantic features and the preset semantic understanding prompt text features are concatenated, and the concatenated target feature sequence is input into the large language model. The large language model then outputs the human motion description text of the target object. The large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples.
2. The method according to claim 1, characterized in that, The target feature sequence includes, in sequence: The features of the semantic understanding prompt text, the marking features of the starting position of the millimeter wave motion semantic features, the marking features of the ending position of the millimeter wave motion semantic features, and the features of the guiding text used to guide the large language model to output the human motion description text.
3. The method according to claim 1 or 2, characterized in that, The spatiotemporal motion encoder includes a convolutional layer and an encoder. The step of inputting the millimeter-wave point cloud sequence into the spatiotemporal motion encoder and extracting the millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder includes: The millimeter-wave point cloud sequence is convolved by the convolutional layer to extract the initial embedding features of the millimeter-wave point cloud sequence; The initial embedding features are compressed and encoded by the encoder to obtain the millimeter-wave motion embedding features.
4. The method according to claim 1 or 2, characterized in that, The motion encoder includes convolutional layers and an autoencoder. The spatiotemporal motion encoder is trained based on millimeter-wave point cloud sequence samples and includes: The millimeter-wave point cloud sequence samples are input into the convolutional layer, and the millimeter-wave point cloud sequence samples are convolved by the convolutional layer to extract the initial embedding features of the millimeter-wave point cloud sequence samples. The initial embedding features are randomly masked to obtain the masked embedding features; The masked embedded features are input into the autoencoder, and the model parameters of the convolutional layer and the autoencoder are jointly optimized. The trained autoencoder and convolutional layer are then extracted to obtain the spatiotemporal motion encoder.
5. The method according to claim 4, characterized in that, The autoencoder includes an encoder and a decoder. The masked embedded features are input into the autoencoder to jointly optimize the model parameters of the convolutional layer and the autoencoder. The trained autoencoder and convolutional layer are then truncated to obtain the spatiotemporal motion encoder, which includes: The masked embedding features are input into the encoder, and the encoder compresses and encodes the masked embedding features to obtain millimeter-wave motion embedding features. The millimeter-wave motion embedding feature is input into the decoder, and the millimeter-wave motion embedding feature is reconstructed by the decoder to obtain the reconstructed millimeter-wave motion embedding feature; Based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, the model parameters of the convolutional layer, the encoder, and the decoder are jointly optimized, and the trained autoencoder and convolutional layer are extracted to obtain the spatiotemporal motion encoder.
6. The method according to claim 5, characterized in that, The joint optimization of model parameters for the convolutional layer, the encoder, and the decoder based on the initial embedding features and the reconstructed millimeter-wave motion embedding features includes: Based on the initial embedding features and the reconstructed millimeter-wave motion embedding features, a first loss function is constructed corresponding to the millimeter-wave point cloud sequence samples. Based on the first loss function, the model parameters of the convolutional layer, the encoder, and the decoder are jointly optimized.
7. The method according to claim 6, characterized in that, The construction of the first loss function corresponding to the millimeter-wave point cloud sequence samples based on the initial embedding features and the reconstructed millimeter-wave motion embedding features includes: based on Construct the first loss function; in, This represents the first loss function. This indicates the number of millimeter-wave point cloud sequence samples. express The first of the millimeter-wave point cloud sequence samples A millimeter-wave point cloud sequence sample This represents the initial embedding feature. This represents the reconstructed millimeter-wave motion embedding features. This represents the features in the initial embedded features. This refers to the features in the reconstructed millimeter-wave motion embedding features.
8. The method according to claim 1 or 2, characterized in that, The large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples, including: The motion embedding features corresponding to the millimeter-wave point cloud sequence samples are concatenated with the command text features, and the concatenated target features are input into the initial large language model. The initial large language model then outputs the predicted human motion description text. Based on the target features, the predicted human motion description text, and the real human motion description text, a second loss function is constructed corresponding to the millimeter-wave point cloud sequence samples. Based on the second loss function, the model parameters of the initial large language model are iteratively updated until the large language model is trained.
9. The method according to claim 8, characterized in that, The second loss function corresponding to the millimeter-wave point cloud sequence samples is constructed based on the target features, the predicted human motion description text, and the actual human motion description text, including: based on Construct the second loss function; in, This represents the second loss function. This indicates the number of lexical units in the text describing the actual human movement. This represents the set of lexical units included in the large language model. This represents the word at position t in the text describing the actual human motion. Indicates an indicator function, if equals the lexicon in the lexicon set ,but =1, otherwise, =0, This represents the word at position t in the predicted human motion description text. This represents the model parameters of the large language model. This represents the target feature. Indicates that given the target feature Given the lexical units at the first t-1 positions of the previously generated real human motion description text, the lexical unit at the t-th position predicted by the large language model is a lexical unit. The probability of.
10. A millimeter-wave human motion understanding device based on a large language model, characterized in that, include: The acquisition unit is used to acquire the millimeter-wave point cloud sequence of the target object; An extraction unit is used to input the millimeter-wave point cloud sequence into a spatiotemporal motion encoder, and extract millimeter-wave motion embedding features corresponding to the millimeter-wave point cloud sequence through the spatiotemporal motion encoder; wherein, the spatiotemporal motion encoder is obtained by training a motion encoder based on the millimeter-wave point cloud sequence; The mapping unit is used to input the millimeter-wave motion embedding features into the projection layer, and to map the millimeter-wave motion embedding features to the input semantic space of the large language model through the projection layer to obtain the millimeter-wave motion semantic features. The processing unit is used to concatenate the millimeter wave motion semantic features and the prompt text features of the preset semantic understanding prompt text, and input the concatenated target feature sequence into the large language model, and output the human motion description text of the target object through the large language model; The large language model is trained based on the target features corresponding to the millimeter-wave point cloud sequence samples and the real human motion description text corresponding to the millimeter-wave point cloud sequence samples.