Millimeter wave radar-based end-to-end volley handwritten letter recognition method

Abstract

The invention discloses an end-to-end volley handwritten letter recognition method based on a millimeter wave radar, and belongs to the field of radar data processing. According to the invention, complete radar original data is used to identify volley handwritten letters, and a multi-modal optimization target of a millimeter wave radar end-to-end handwriting identification method based on writing mode guidance is provided. A multi-modal handwriting recognition model for millimeter wave radar original signals and two-dimensional handwritten letter stroke trajectory data is provided, and a targeted neural network training strategy is provided for the optimization target and the model architecture. According to the method, priori knowledge is fused in a training process, so that the extraction complexity of abstract features by a network is reduced, the training convergence of a radar original data feature extractor is accelerated, and the recognition accuracy of the radar original data feature extractor in a complex handwritten letter recognition task is improved.

Description

Technical Field

[0001] The present invention belongs to the field of radar data processing, and in particular relates to an end-to-end mid-air handwritten letter recognition method based on millimeter wave radar. Background Art

[0002] Against the backdrop of the rapid development of general-purpose computing, human-computer interaction (HCI) has become a highly sought-after research area. Human gestures, as a natural and intuitive form of interaction, have opened up new possibilities for HCI. In recent years, gesture recognition technologies based on vision and wearable sensors have made significant progress. However, these approaches have limitations in practical applications: optical camera sensors can raise privacy and security concerns, while wearable devices face challenges in portability and comfort, limiting their widespread adoption.

[0003] With the rise of radar technology in civilian applications, radar-based gesture recognition has become a promising research area. Early research focused on simple gesture classification tasks using single-antenna radar systems. However, since single-antenna radars can only measure distance and velocity and lack information about the angle of motion, they struggle to distinguish gesture changes in orientation. To address this issue, between 2019 and 2024, significant research shifted to millimeter-wave radars employing multiple-input, multiple-output (MIMO) architectures. These radars can provide three- or four-dimensional spatial information, enabling a more comprehensive description of complex spatial gestures.

[0004] In recent years, several studies have successfully applied MIMO radar to the task of airwriting, achieving accurate airwriting recognition through signal processing techniques and trajectory reconstruction algorithms. These studies demonstrate the significant potential of MIMO radar for complex motion recognition. These methods excel in specific scenarios, exhibit strong interpretability, and exhibit high computational efficiency. However, as the complexity of the task increases, traditional methods face numerous challenges:

[0005] 1) During the preprocessing stage, commonly used target detection algorithms such as fast Fourier transform and constant false alarm detection may inadvertently filter out some potentially valuable information;

[0006] 2) The design of signal preprocessing algorithms and the selection of radar signature image types often rely on empirical knowledge and may not be applicable to all mission scenarios;

[0007] 3) The integration of traditional algorithms with neural networks and their parallel computing still face certain difficulties, which to some extent limits the hardware deployment of end-to-end tasks;

[0008] Therefore, in recent years, researchers have begun to experiment with using specialized neural network models to extract features directly from raw radar data. The challenge lies in the fact that raw radar data is a multi-channel, high-dimensional, complex signal time series, where gesture features are contained in the signal's amplitude, phase, and time dependencies. Previous end-to-end radar gesture recognition research typically processed the real and imaginary parts of the complex signal separately, resulting in the loss of important phase information. Alternatively, when processing the complete raw signal, researchers faced slow network convergence and high computational resource consumption. Summary of the Invention

[0009] This paper proposes an innovative end-to-end method for airwritten handwritten letter recognition, which utilizes complete radar raw data for recognition. To address the challenges of network training and abstract feature extraction for high-dimensional radar data, a novel solution is introduced, namely, incorporating prior knowledge into the training process to reduce the complexity of the network's extraction of abstract features. Based on the idea of multimodal feature alignment, the semantic features of the two-dimensional trajectory sequence are used to guide the end-to-end network in extracting semantic features from the raw signal. Theoretical analysis and experimental results show that this method accelerates the training convergence of the radar raw data feature extractor and improves its recognition accuracy in complex handwritten letter recognition tasks.

[0010] The purpose of the present invention is achieved through the following technical solution: a method for end-to-end mid-air handwritten letter recognition based on millimeter wave radar, the method comprising the following steps:

[0011] Step 1) Use MIMO millimeter wave radar to detect the handwritten letters in the air, pass the echo through the mixer and low-pass filter to obtain the intermediate frequency signal, and sample the intermediate frequency signal to obtain the original data of the radar, which is recorded as ,in , this is a radar raw data sequence corresponding to a handwritten letter action, N represents the number of equivalent transmit and receive channels, H represents the number of linear frequency modulation pulses per frame, L represents the number of sampling points of each linear frequency modulation pulse, and T represents the number of frames;

[0012] Step 2) normalizes the radar raw data obtained in step 1), and the specific formula is as follows:

[0013] ;

[0014] in For the The mean of all data amplitudes in a radar raw data sequence, For the The standard deviation of all data amplitudes in a radar raw data sequence.

[0015] Step 3) Normalize the data from step 2) Input the original radar data feature extractor, complete the preliminary data dimension reduction and feature expression, and output a representation vector of size 1×512 ( );

[0016] The specific implementation methods include:

[0017] Step a) Consider a batch in the training process, the original radar data dimension is ,in Represents the number of samples used in a batch, that is, the batch size, while T represents the number of frames, H represents the number of linear frequency modulation pulses per frame, N represents the number of equivalent transmit and receive channels, and L represents the number of sampling points of each linear frequency modulation pulse. The fast time, slow time, and equivalent transmit and receive channel dimensions of the original radar signal are sequentially reduced and feature extracted through three layers of complex-valued linear layers to obtain the data dimension: , the complex-valued linear layer is expressed as:

[0018] ;

[0019] Among them, the weight Initialized as:

[0020] ;

[0021] Step b) Then, the characteristic The dimension is fed into multiple layers of 3D convolutional layers for further fine-grained feature extraction. After the convolutional layers, the 3D features are flattened to obtain Fine-grained handwriting features;

[0022] Step c) Finally, a two-layer long short-term memory network is used to further extract the temporal correlation features of the handwriting in the time dimension, and the encoding is obtained as the representation vector (B, 128) .

[0023] Step 4) The representation vector obtained in step 3) is The input variational autoencoder extracts its distribution information. The variational autoencoder consists of a linear layer and outputs the distribution information of the corresponding handwriting trajectory features, including the mean and variance Then, we randomly sample the Gaussian distribution of mean and variance to obtain Corresponding handwriting trajectory features ;

[0024] Step 5) Use 2D handwritten letter trajectory sequence data Conduct guidance training, including , represents the two-dimensional sequence trajectory of a handwritten letter, which is also normalized using the method described in step 2), as follows:

[0025] ;

[0026] in For the The mean of all data amplitudes in a two-dimensional handwriting trajectory sequence, For the The standard deviation of all data amplitudes in a two-dimensional handwriting trajectory sequence.

[0027] Step 6) Normalize the data in step 5) Input is a two-dimensional handwriting trajectory feature extractor composed of a two-layer long short-term memory network, and output is a representation vector of size 1×512 ( );

[0028] Step 7) Get the normalized representation vector After that, similar to step 4), the distribution information is extracted through the variational self-encoder. The variational self-encoder consists of a linear layer and outputs the distribution information of the corresponding handwriting trajectory features, including the mean and variance Then, we sample the Gaussian distribution of mean and variance by random sampling. Corresponding handwriting trajectory features ;

[0029] Step 8) Use the decoder composed of linear layers to decode the handwriting trajectory features obtained in step 4) and step 7) obtained Reconstruct and regenerate 512 dimensions and , and input these two vectors into the discriminator to obtain the loss , as part of the overall loss, the detailed implementation steps are:

[0030] Step a) The multimodal optimization goal of training the millimeter wave radar end-to-end handwriting recognition method is to obtain a feature extractor To approximate the distribution of real handwriting trajectory features , using KL divergence, the optimization objective is equivalent to minimizing:

[0031] ;

[0032] according to is a constant, minimizing the KL divergence is equivalent to maximizing the variational lower bound :

[0033] ;

[0034] and Expressed as reconstruction loss and regularization loss, minimizing KL divergence is equivalent to maximizing reconstruction loss and minimizing regularization loss;

[0035] Step b) maximizes the reconstruction loss described as the following minimum mean square error:

[0036] ;

[0037] in A reconstructor that outputs the radar raw signal for a given handwritten trajectory feature;

[0038] Step c) The two-dimensional handwriting trajectory data and the radar original signal have similar semantic features, namely, the handwriting trajectory feature and The feature space is similar, using an easy-to-train handwriting trajectory reconstructor To describe the maximum reconstruction loss:

[0039] ;

[0040] Step d) Minimize the regularization loss as the true distribution of the handwritten trajectory features of the original radar signal and the posterior distribution of the handwriting trajectory features calculated by the encoder using the original radar signal The KL divergence of the original radar signal is considered to be the true distribution of the handwritten trajectory features. Satisfies the standard Gaussian distribution, and the encoder calculates the posterior distribution of the handwriting trajectory features through the original radar signal Satisfying the mean and the standard deviation is Gaussian distribution ,get:

[0041] ;

[0042] Step 9) The handwriting trajectory features obtained in step 4) are Through linear classification, a 26-dimensional probability vector P of handwritten letter recognition is directly output for recognition and classification.

[0043] Step 10) A targeted neural network training strategy is used to optimize the parameters of the two-dimensional handwriting trajectory feature extractor and encoder, the two-dimensional handwriting trajectory decoder, and the radar raw signal feature extractor and encoder. The specific training steps are as follows:

[0044] Step a) Training the 2D handwriting trajectory feature extractor and encoder: Using the labeled 2D handwriting trajectory sequence data, the 2D handwriting trajectory feature extractor and encoder are trained in a standard supervised learning manner, using the cross entropy loss function Expressed as:

[0045] ;

[0046] Where N is the number of samples, C is the number of categories, Indicates the samples in the category The unique hot tag, are the model parameters of the 2D handwriting trajectory feature extractor and encoder.

[0047] Step b) Training the 2D handwriting trajectory decoder: Fix the 2D handwriting trajectory feature extractor and encoder parameters, and set the loss function for training the 2D handwriting trajectory decoder to:

[0048] ;

[0049] Where M is the number of samples, are the model parameters of the 2D handwriting trajectory decoder.

[0050] Step c) training the radar original signal feature extractor and encoder: When optimizing the parameters of the radar original signal feature extractor and encoder, the parameters of the two-dimensional handwriting trajectory feature extractor, encoder, and decoder are fixed, and the optimization goal is:

[0051] ;

[0052] Represents the parameters of the radar original signal feature extractor and encoder, and its loss function The cross entropy loss between the probability vector P obtained by recognition and the radar original signal label and the losses described in claim 2 composition:

[0053] ;

[0054] and By reconstruction loss and regularization loss It consists of two parts:

[0055] ;

[0056] in, 、 is a hyperparameter, Determines the degree of influence of the two-dimensional handwriting trajectory feature information on the optimization, It determines the influence of KL divergence on optimization, that is, the similarity between the original radar signal trajectory characteristics and the standard normal distribution.

[0057] Compared with the prior art, the present invention has the following innovative advantages and significant effects:

[0058] 1) For the task of end-to-end handwritten letter recognition using millimeter-wave radar, a multimodal optimization objective for the end-to-end handwriting recognition method using stroke patterns as guidance is proposed.

[0059] 2) In order to optimize the above objectives, a multimodal handwriting recognition model is proposed based on millimeter-wave radar raw signals and two-dimensional handwritten letter stroke trajectory data.

[0060] 3) Based on the above optimization objectives and model architecture, a targeted neural network training strategy is proposed.

[0061] 4) This method performs handwritten letter recognition based on complete raw radar data. By incorporating prior knowledge into the training process, it reduces the complexity of the network's extraction of abstract features, accelerates the training convergence of the radar raw data feature extractor, and improves its recognition accuracy in complex handwritten letter recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is the overall framework of the end-to-end handwritten letter recognition of the present invention;

[0063] Figure 2 It is a structural diagram of the two-dimensional handwriting trajectory feature extractor and radar original signal feature extractor proposed in the present invention;

[0064] Figure 3 1 is a structural diagram of a variational encoder and decoder proposed in the present invention;

[0065] Figure 4 : This is the confusion matrix of the classification results of handwritten recognition of 26 English letters of the model in one embodiment of the present invention. DETAILED DESCRIPTION

[0066] In this example, an Intel RealSense SR300 camera was used to collect a dataset of handwritten stroke trajectories for 26 letters. The writing style of these trajectories was set to the most typical letter writing strokes, representing common stroke patterns. Ten volunteers participated in the data collection, with a total of 30,000 samples. The average trajectory length ranged from 21 to 173, and the data storage format was similar to ,in, It is the first The two-dimensional handwriting trajectory position of the sequence points. The detailed implementation steps are as follows:

[0067] Step 1) Following the writing pattern of the above-mentioned handwriting stroke trajectory dataset, this embodiment constructs an aerial handwritten letter raw signal dataset using MIMO millimeter-wave radar, and the data collection is carried out in real environments with frequent human activities, such as offices and conference rooms. In order to maintain the user's natural writing style, no restrictions are imposed on the participants except for the specified writing gesture templates. The data was collected using Texas Instruments' IWR6843ISK radar sensor, which operates in the 60-64 GHz frequency range. This MIMO radar sensor is equipped with 3 transmitting antennas and 4 receiving antennas. To ensure data consistency, this embodiment limits the execution time of each handwriting action to within 3 seconds. The writing area is defined as a space of 40 cm × 40 cm × 20 cm above the sensor, which is considered to be the optimal range for capturing handwriting actions. Ultimately, each handwriting sample for 4D radar raw data, T represents the number of frames, H represents the number of linear frequency modulation pulses per frame, N represents the number of equivalent transmit and receive channels, and L represents the number of sampling points for each linear frequency modulation pulse;

[0068] Step 2) normalizes the radar raw data obtained in step 1), and the specific formula is as follows:

[0069] ;

[0070] in For the The mean of all data amplitudes in a radar raw data sequence, For the The standard deviation of all data amplitudes in a radar raw data sequence.

[0071] Step 3) Normalize the data from step 2) Input the original radar data feature extractor, complete the preliminary data dimension reduction and feature expression, and output a representation vector of size 1×512 ( );

[0072] The specific implementation methods include:

[0073] Step a) In this embodiment, the batch size is set to 24, so the data dimension of the feature extractor inputting the raw radar data is The fast time, slow time, and equivalent transmit and receive channel dimensions of the original radar signal are sequentially reduced and feature extracted through three layers of complex-valued linear layers to obtain the data dimension: , the complex-valued linear layer is expressed as:

[0074] ;

[0075] Among them, the weight Initialized as:

[0076] ;

[0077] Step b) Then, the characteristic The dimension is fed into multiple layers of 3D convolutional layers for further fine-grained feature extraction. After the convolutional layers, the 3D features are flattened to obtain Fine-grained handwriting features;

[0078] Step c) Finally, the double-layer long short-term memory network is used to further extract the temporal correlation features of the handwriting in the time dimension, and the encoding is obtained as the representation vector of (24,128) .

[0079] Step 4) Represent the vector The distribution information is extracted through the variational autoencoder, which consists of a linear layer and outputs the distribution information of the corresponding handwriting trajectory features, including the mean and variance Then, we randomly sample the Gaussian distribution of the mean and variance to obtain Corresponding handwriting trajectory features ;

[0080] Step 5) In order to ensure the consistency of data length, the data in the public RTC dataset is downsampled or interpolated. The sequence length is converted to 50, and the data is normalized using the method in step 2), as follows:

[0081] ;

[0082] in For the The mean of all data amplitudes in a two-dimensional handwriting trajectory sequence, For the The standard deviation of all data amplitudes in a two-dimensional handwriting trajectory sequence.

[0083] Step 6) Normalize the data in step 5) Input is a two-dimensional handwriting trajectory feature extractor composed of a two-layer long short-term memory network, and output is a representation vector of size 1×512 ( );

[0084] Step 7) Get the normalized representation vector Then, similar to step 4), a variational autoencoder composed of linear layers is used to extract its distribution information, and the output is the distribution information of the corresponding handwriting trajectory features, including the mean and variance Then, we randomly sample the Gaussian distribution of the mean and variance to obtain Corresponding handwriting trajectory features ;

[0085] Step 8) Use the decoder composed of linear layers to decode the handwriting trajectory features obtained in step 4) and step 7) obtained Reconstruct and regenerate 512 dimensions and , and input these two vectors into the discriminator to obtain the loss , as part of the overall loss, the detailed implementation steps are:

[0086] Step a) The multimodal optimization goal of training the millimeter wave radar end-to-end handwriting recognition method is to obtain a feature extractor To approximate the distribution of real handwriting trajectory features , using KL divergence, the optimization objective is equivalent to minimizing:

[0087] ;

[0088] according to is a constant, minimizing the KL divergence is equivalent to maximizing the variational lower bound :

[0089] ;

[0090] and Expressed as reconstruction loss and regularization loss, minimizing KL divergence is equivalent to maximizing reconstruction loss and minimizing regularization loss;

[0091] Step b) maximizes the reconstruction loss described as the following minimum mean square error:

[0092] ;

[0093] in A reconstructor that outputs the radar raw signal for a given handwritten trajectory feature;

[0094] Step c) The two-dimensional handwriting trajectory data and the radar original signal have similar semantic features, namely, the handwriting trajectory feature and The feature space is similar, using an easy-to-train handwriting trajectory reconstructor To describe the maximum reconstruction loss:

[0095] ;

[0096] Step d) Minimize the regularization loss as the true distribution of the handwritten trajectory features of the original radar signal and the posterior distribution of the handwriting trajectory features calculated by the encoder using the original radar signal The KL divergence of the original radar signal is considered to be the true distribution of the handwritten trajectory features. Satisfies the standard Gaussian distribution, and the encoder calculates the posterior distribution of the handwriting trajectory features through the original radar signal Satisfying the mean and the standard deviation is Gaussian distribution ,get:

[0097] ;

[0098] Step 9) The handwriting trajectory features obtained in step 4) are Through linear classification, a 26-dimensional probability vector P of handwritten letter recognition is directly output for recognition and classification.

[0099] Step 10) A targeted neural network training strategy is used to optimize the parameters of the two-dimensional handwriting trajectory feature extractor and encoder, the two-dimensional handwriting trajectory decoder, and the radar raw signal feature extractor and encoder. The specific training steps are as follows:

[0100] Step a) Training the 2D handwriting trajectory feature extractor and encoder: Using the labeled 2D handwriting trajectory sequence data, the 2D handwriting trajectory feature extractor and encoder are trained in a standard supervised learning manner, using the cross entropy loss function Expressed as:

[0101] ;

[0102] Where N is the number of samples, C is the number of categories, Indicates the samples in the category The unique hot tag, are the model parameters of the 2D handwriting trajectory feature extractor and encoder.

[0103] Step b) Training the 2D handwriting trajectory decoder: Fix the 2D handwriting trajectory feature extractor and encoder parameters, and set the loss function for training the 2D handwriting trajectory decoder to:

[0104] ;

[0105] Where M is the number of samples, are the model parameters of the 2D handwriting trajectory decoder.

[0106] Step c) training the radar original signal feature extractor and encoder: When optimizing the parameters of the radar original signal feature extractor and encoder, the parameters of the two-dimensional handwriting trajectory feature extractor, encoder, and decoder are fixed, and the optimization goal is:

[0107] ;

[0108] Represents the parameters of the radar original signal feature extractor and encoder, and its loss function The cross entropy loss between the probability vector P obtained by recognition and the radar original signal label and the losses described in claim 2 composition:

[0109] ;

[0110] and By reconstruction loss and regularization loss It consists of two parts:

[0111] ;

[0112] Among them, the hyperparameters Determines the degree of influence of the two-dimensional handwriting trajectory feature information on the optimization, It determines the influence of KL divergence on optimization (i.e. the similarity between the original radar signal trajectory characteristics and the standard normal distribution).

[0113] Step 11) This example uses Python 3.11 to develop and implement the above model construction and training in the PyTorch framework. During deep network training, the Adam optimizer with adaptive learning rate was used, and the initial learning rate was set to 2e-3. A total of 70 training rounds were performed, using a linear learning rate decrease strategy, decreasing the learning rate every 20 rounds. All experiments were performed on an Nvidia RTX4060 GPU with CUDA acceleration. The confusion matrix of the classification results of this model is shown in Figure 1. Figure 4 As shown, good classification results were achieved.

Claims

1. A method for end-to-end mid-air handwritten letter recognition based on millimeter wave radar, characterized in that: The method comprises the following steps: Step 1) Use MIMO millimeter wave radar to detect the handwritten letters in the air, pass the echo through the mixer and low-pass filter to obtain the intermediate frequency signal, and sample the intermediate frequency signal to obtain the original data of the radar, which is recorded as ,in , This is a radar raw data sequence corresponding to a handwritten letter action, N represents the number of equivalent transmit and receive channels, H represents the number of linear frequency modulation pulses per frame, L represents the number of sampling points of each linear frequency modulation pulse, T represents the number of frames, and k represents the number of sampling points; Step 2) normalizes the radar raw data obtained in step 1) as follows: ； in For the The mean of all data amplitudes in a radar raw data sequence, For the The standard deviation of all data amplitudes in a radar raw data sequence; Step 3) Normalize the data from step 2) Input the radar raw data feature extractor to complete the preliminary data dimensionality reduction and feature expression, and output a representation vector of size 1×512 , ; Step 4) Represent the vector The distribution information is extracted through the variational autoencoder, which consists of a linear layer and outputs the distribution information of the corresponding handwriting trajectory features, including the mean and variance ; Then, we randomly sample the Gaussian distribution of mean and variance to obtain Corresponding handwriting trajectory features ; Step 5) Use 2D handwritten letter trajectory sequence data Conduct guidance training, including , represents the two-dimensional sequence trajectory of a handwritten letter, which is also normalized using the same method in step 2), as follows: ； in For the The mean of all data amplitudes in a two-dimensional handwriting trajectory sequence, For the The standard deviation of all data amplitudes in the model in a two-dimensional handwriting trajectory sequence; Step 6) Normalize the data in step 5) Input is a two-dimensional handwriting trajectory feature extractor composed of a two-layer long short-term memory network, and output is a representation vector of size 1×512 , ; Step 7) Get the normalized representation vector After that, similar to step 4), the distribution information is extracted through the variational self-encoder. The variational self-encoder consists of a linear layer and outputs the distribution information of the corresponding handwriting trajectory features, including the mean and variance ; Then, we sample the Gaussian distribution of mean and variance by random sampling. Corresponding handwriting trajectory features ; Step 8) Use the decoder composed of linear layers to decode the handwriting trajectory features obtained in step 4) and step 7) obtained Reconstruct and regenerate 512 dimensions and , and input these two vectors into the discriminator to obtain the loss , as part of the overall loss; Step 9) The handwriting trajectory features obtained in step 4) are Through linear classification, a 26-dimensional probability vector P of handwritten letter recognition is directly output for recognition and classification. Step 10) A targeted neural network training strategy is used to optimize the parameters of the two-dimensional handwriting trajectory feature extractor and encoder, the two-dimensional handwriting trajectory decoder, and the radar raw signal feature extractor and encoder.

2. The method for end-to-end mid-air handwritten letter recognition based on millimeter wave radar according to claim 1, characterized in that: The radar raw data feature extractor is used to perform preliminary data dimension reduction and feature expression on the radar raw data. In the step 3), the representation vector is obtained by the radar raw data feature extractor. , the structure and calculation process of the feature extractor are as follows: Step a) Consider a batch in the training process, the original radar data dimension is ,in Represents the number of samples used in a batch, that is, the batch size, while T represents the number of frames, H represents the number of linear frequency modulation pulses per frame, N represents the number of equivalent transmit and receive channels, and L represents the number of sampling points of each linear frequency modulation pulse. The fast time, slow time, and equivalent transmit and receive channel dimensions of the original radar signal are sequentially reduced and feature extracted through three layers of complex-valued linear layers to obtain the data dimension: , the complex-valued linear layer is expressed as: ； Among them, the weight Initialized as: ； Step b) Then, the characteristic The dimension is fed into multiple layers of 3D convolutional layers for further fine-grained feature extraction. After the convolutional layers, the 3D features are flattened to obtain Fine-grained handwriting features; Step c) Finally, a two-layer long short-term memory network is used to further extract the temporal correlation features of the handwriting in the time dimension, and the encoding is obtained as the representation vector (B, 128) .

3. The method for end-to-end mid-air handwritten letter recognition based on millimeter wave radar according to claim 1, characterized in that: A multimodal optimization objective of an end-to-end handwriting recognition method using millimeter-wave radar guided by writing mode is proposed. In step 8), the decoder output and Input the discriminator to get the loss , the detailed calculation process and the corresponding optimization objectives are: Step a) The multimodal optimization goal of training the millimeter wave radar end-to-end handwriting recognition method is to obtain a feature extractor To approximate the distribution of real handwriting trajectory features , using KL divergence, the optimization objective is equivalent to minimizing: ； according to is a constant, minimizing the KL divergence is equivalent to maximizing the variational lower bound : ； and Expressed as reconstruction loss and regularization loss, minimizing KL divergence is equivalent to maximizing reconstruction loss and minimizing regularization loss; Step b) maximizes the reconstruction loss described as the following minimum mean square error: ； in A reconstructor that outputs the radar raw signal for a given handwritten trajectory feature; Step c) The two-dimensional handwriting trajectory data and the radar original signal have similar semantic features, namely, the handwriting trajectory feature and The feature space is similar, using an easy-to-train handwriting trajectory reconstructor To describe the maximum reconstruction loss: ； d) Minimizing the regularization loss is expressed as the true distribution of the handwritten trajectory features of the original radar signal and the posterior distribution of the handwriting trajectory features calculated by the encoder using the original radar signal The KL divergence of the original radar signal is considered to be the true distribution of the handwritten trajectory features. Satisfies the standard Gaussian distribution, and the encoder calculates the posterior distribution of the handwriting trajectory features through the original radar signal Satisfying the mean and the standard deviation is Gaussian distribution ,get: 。 4. The method for end-to-end mid-air handwritten letter recognition based on millimeter wave radar according to claim 1, characterized in that: In view of the optimization objectives and model architecture described in claim 1, a targeted neural network training strategy is proposed. In step 10), the specific training steps are as follows: Step a) Training the 2D handwriting trajectory feature extractor and encoder: Using the labeled 2D handwriting trajectory sequence data, the 2D handwriting trajectory feature extractor and encoder are trained in a standard supervised learning manner, using the cross entropy loss function Expressed as: ； Where N is the number of samples, C is the number of categories, Indicates the samples in the category The unique hot tag, are the model parameters of the 2D handwriting trajectory feature extractor and encoder; Step b) Training the 2D handwriting trajectory decoder: Fix the 2D handwriting trajectory feature extractor and encoder parameters, and set the loss function for training the 2D handwriting trajectory decoder to: ； Where M is the number of samples, are the model parameters of the 2D handwriting trajectory decoder; Step c) training the radar original signal feature extractor and encoder: When optimizing the parameters of the radar original signal feature extractor and encoder, the parameters of the two-dimensional handwriting trajectory feature extractor, encoder, and decoder are fixed, and the optimization goal is: ； Represents the parameters of the radar original signal feature extractor and encoder, and its loss function The cross entropy loss between the probability vector P obtained by recognition and the radar original signal label and the losses described in claim 2 composition: ； and By reconstruction loss and regularization loss It consists of two parts: ； in, 、 is a hyperparameter, Determines the degree of influence of the two-dimensional handwriting trajectory feature information on the optimization, Determines the impact of KL divergence on optimization.

Images