Millimeter wave radar based few-shot lip reading method and apparatus

Abstract

This invention relates to the field of lip-reading technology, specifically disclosing a few-shot lip-reading recognition method and apparatus based on millimeter-wave radar. In this application, the influence of illumination, occlusion, and noise on recognition is avoided at the source by preprocessing the echo signal to obtain a micro-Doppler image. The micro-Doppler image is processed by the radar encoder branch of the lip-reading recognition model to obtain radar temporal features, which are then globally aggregated to obtain a radar global embedding vector, solving the problem of lost temporal correlation before and after pronunciation. The trained lip-reading recognition model obtains text pronunciation anchor points for each category label in a preset lip-reading command set, and calculates the similarity between the radar global embedding vector of the echo signal and the text pronunciation anchor points to obtain the lip-reading recognition result. By embedding linguistic and physical priors into the training of the lip-reading recognition model and superimposing a two-stage training strategy, the effectiveness of model learning is significantly improved, and the accuracy of lip-reading recognition under few-shot conditions is enhanced.

Description

Technical Field

[0001] This invention relates to the field of lip reading technology, and particularly to a method and apparatus for few-sample lip reading based on millimeter-wave radar. Background Technology

[0002] Radar lip reading typically utilizes millimeter-wave FMCW (Frequency-Modulated Continuous Wave) radar to capture the micro-movements of the lips, jaw, and local facial tissues during pronunciation, converting them into a micro-Doppler time-frequency representation. FMCW radar does not acquire identifiable facial images while maintaining high sensitivity to minute displacement and velocity changes, thus providing an attractive alternative for privacy-preserving IoT interactions.

[0003] Most existing methods employ closed-set classification: a finite set of commands is predefined, sufficient radar samples are collected, and then convolutional neural networks or attention networks are trained for classification. These methods achieve good performance under fixed vocabulary and controlled environments, but users often need to add new commands, modify phrases, or express similar interaction intentions in different languages. Re-collecting large amounts of radar data and retraining the model for each new command is impractical. Text prior and cross-modal contrastive learning offer a new approach to unseen category recognition.

[0004] However, directly transferring general semantic embeddings is not entirely applicable to radar lip movement perception. The fundamental reason is that FMCW radar primarily observes the micro-movements of the external lips, jaw, and local facial tissues, rather than semantic, grammatical, or intraoral articulation events. Two semantically significantly different commands may have similar visible lip trajectories; conversely, two texts or semantically similar words may differ primarily in intraoral phonemes, which are difficult for radar to directly observe. Most existing methods employ the approach of directly inputting radar micro-Doppler images into convolutional neural networks for image classification. This deep learning model heavily relies on massive and precisely labeled training data, making radar data acquisition costly and hindering the system's ability to infinitely expand its vocabulary like large text models. When encountering unseen commands, the model can only force a classification or fails. Therefore, there is an urgent need for a lip-reading method that overcomes the overfitting problem of existing radar lip-reading methods with limited samples, thereby improving lip-reading accuracy with fewer samples. Summary of the Invention

[0005] To address the problems existing in the prior art, embodiments of this application provide a method and apparatus for few-sample lip reading recognition based on millimeter-wave radar, thereby improving the accuracy of few-sample lip reading recognition.

[0006] This application provides a few-sample lip-reading recognition method based on millimeter-wave radar, including the following steps: The test echo signal transmitted back by the FMCW radar is acquired, and the test echo signal is preprocessed to obtain a micro Doppler image characterizing lip changes, thereby distinguishing different articulation movements. A batch of sample data is acquired, and a two-stage warm-up strategy is used to train a lip-reading recognition model that aligns the image and text modes. Each batch of samples in the sample data contains a micro-Doppler image, the lip-reading instruction text corresponding to the micro-Doppler image, and a category label, thus obtaining the trained lip-reading recognition model. The micro-Doppler image of the echo signal to be tested is input into the trained lip reading recognition model to obtain the radar global embedding vector of the echo signal to be tested. The similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label in the offline pre-stored preset lip reading command set is calculated, and the result with the highest similarity is taken as the lip reading recognition result.

[0007] In the above scheme, the step of preprocessing the echo signal to obtain a micro-Doppler image characterizing lip changes includes: The windowed echo signal to be measured is subjected to a range-dimensional Fourier transform to obtain an initial time-range heatmap; wherein, the initial time-range heatmap reflects the echo energy at different distances at different times during the detection. The initial time-distance heatmap is processed by first-order difference according to the same distance and adjacent frames to obtain a time-distance heatmap; The echo energy of all frames in each distance chamber within the preset distance chamber set is accumulated, and the distance chamber with the largest accumulated energy is selected as the lip reference target distance chamber; wherein, the distance chamber set consists of all distance chambers within the preset effective near-range detection range of the human mouth; Centered on the lip reference target distance chamber, multiple distance chambers are extended to the left and right sides to obtain the optimal distance window for the lips; the reserved distance chambers effectively avoid the problem of the target running out of the detection range when the person slightly shakes their head or moves back and forth, reducing the impact of micro-head movements; After extracting the echo signal to be tested within the optimal lip distance window of the time-distance heatmap, the frequencies outside the preset range are filtered out by a bandpass filter. Then, a short-time Fourier transform is performed on the filtered echo signal to be tested to extract a micro-Doppler image with time on the horizontal axis and Doppler frequency on the vertical axis. The instantaneous speed and direction of the lip movement can be seen intuitively from the image, thus distinguishing different articulation actions.

[0008] In the above scheme, the training of the image and text bimodal aligned lip-reading recognition model using a two-stage warm-up strategy includes: The first step is to perform the first forward propagation: the micro-Doppler images in a single batch of samples are input into the radar branch encoder for processing to obtain the corresponding radar temporal features and radar global embedding vector. Simultaneously, the lip-reading text corresponding to the micro-Doppler images is input into the text branch encoder for processing to obtain the corresponding phoneme temporal features and text pronunciation anchors of each category. The radar global embedding vector and the text pronunciation anchors are then L2 normalized and mapped to the unit hypersphere. The radar branch encoder includes a multi-layer two-dimensional convolutional network, a lightweight self-attention module, and a first Bi-LSTM network, while the text branch encoder includes a second Bi-LSTM network. The second step is to perform the first batch of parameter updates: construct a symmetrical form of supervised contrast loss based on the normalized radar global embedding vector and text pronunciation anchor points, use the calculated supervised contrast loss as the first target loss function, perform the first backpropagation to update the model parameters in the first batch, and iterate until the embedding space meets the conditions. The third step involves performing a second forward propagation and parameter update: A second forward propagation, consistent with step 21, is executed, followed by a second backpropagation based on the calculated second objective loss function to update the model parameters in the second batch. The second objective loss function is a weighted sum of supervised contrastive loss, Soft-DTW loss, and joint mask regularization loss. The joint mask regularization loss is a weighted sum of the first and second regularization terms. The first regularization term constrains the correction magnitude of the preset prior weights, and the second regularization term constrains the mean of the soft mask to be within a preset reasonable range. Initially, only the first objective loss function stabilizes the feature space, achieving precise category-level matching between radar global features and text pronunciation anchors, constructing clear feature clustering boundaries. Later, the second objective loss function, which superimposes Soft-DTW loss and joint mask regularization loss, achieves dynamic matching between radar temporal features and phonemes, effectively avoiding noise gradients caused by feature instability in the early training stages. Repeat steps one through three until the lip reading model's performance reaches the convergence condition, then output the trained lip reading model.

[0009] In the above scheme, the step of inputting the micro-Doppler images from a single batch of samples into the radar branch encoder for processing to obtain the corresponding radar time-series features and radar global embedding vector includes: The micro-Doppler images from a single batch of samples are input into a multi-layer two-dimensional convolutional network to extract the time-frequency texture features corresponding to lip vibrations layer by layer. This method can automatically learn the most discriminative texture features in the micro-Doppler images and maintain good feature extraction performance even in complex environments such as low signal-to-noise ratio, avoiding the subjectivity and limitations of manually designed features. The time-frequency texture features are used as input to a lightweight self-attention module. The time-frequency response of lip pronunciation in the time-frequency texture features is enhanced by the lightweight self-attention module, and the invalid time-frequency response corresponding to environmental clutter and physiological interference is suppressed, resulting in time-frequency texture enhancement features. The time-frequency texture enhancement features are reorganized into a one-dimensional temporal feature sequence along the time dimension and input into the first Bi-LSTM network to model the sequential dependencies in the pronunciation process. The output radar temporal features are then globally aggregated to obtain the radar global embedding vector.

[0010] In the above scheme, the step of inputting the lip-reading instruction text corresponding to the micro-Doppler image into a text branch encoder for processing to obtain the corresponding phoneme temporal features and text pronunciation anchors for each category includes: After converting the lip-reading command text corresponding to the micro-Doppler image into a pinyin sequence, it is decomposed into syllables to obtain a phoneme feature sequence. The initial phoneme embedding vector and the prior pronunciation attribute vector describing the physical pronunciation features of each phoneme in the phoneme feature sequence are obtained. A continuous and smooth soft mask is obtained by superimposing a preset prior weight with a correction term of the preset prior weight; wherein, the larger the preset prior weight, the higher the observability of the corresponding phoneme relative to the radar. The initial phoneme embedding vector of each phoneme in the phoneme feature sequence is multiplied element-wise with the soft mask to obtain the weighted phoneme feature sequence; the weighting of the initial phoneme embedding vector by the soft mask preserves the high-visibility phonemes completely and weakens the low-visibility phonemes to transition the context. The weighted phoneme feature sequence is input into the second Bi-LSTM network to model the contextual dependency of phoneme pronunciation. The output phoneme temporal features are weighted and summed using the soft mask to obtain the text pronunciation anchor point.

[0011] In the above scheme, the step of adding a correction term to the preset prior weights to obtain a soft mask includes: The prior articulation attribute vector of each phoneme is mapped to an initial correction term of the preset prior weight of each phoneme, wherein the initial correction term is used to reflect the prior strength of the external articulation action of the corresponding phoneme as observed by millimeter-wave radar. The initial correction terms of each phoneme are restricted by the hyperbolic tangent function and the preset correction amplitude coefficient, respectively, to obtain the correction terms of the preset prior weights of each phoneme. The preset prior weights of each phoneme are summed with the corresponding correction terms to obtain the soft mask of each phoneme, ensuring that the correction does not violate the physical laws of pronunciation. In the above scheme, the embedding space meets the conditions, and the determination process includes: The label mask matrix constructed based on the category labels uses radar global embedding features of the same category and text pronunciation anchors as positive sample pairs, and combinations of different categories as negative sample pairs. When the similarity fluctuation amplitude of the positive sample pairs and the similarity fluctuation amplitude of the negative sample pairs in the verification set are both less than the preset fluctuation threshold in multiple consecutive iteration cycles, it is determined that the cross-modal embedding space formed by the radar global embedding vector and the text pronunciation anchors meets the conditions.

[0012] In the above scheme, the calculation of the similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label in the offline pre-stored preset lip-reading command set, and taking the result with the highest similarity as the lip-reading recognition result, includes: The trained lip-reading recognition model obtains the text pronunciation anchor points of each category label in the preset lip-reading command set; In Zero-shot mode, the cosine similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label stored offline is calculated, and the category with the highest cosine similarity is taken as the lip reading recognition result; no labeled data is required during the recognition process. In K-shot mode, for each candidate category, multiple radar global embedding vectors of preset echo signals with the candidate category label are obtained through the radar encoder branch of the lip reading recognition model. The mean value of the corresponding radar global embedding vector is used as the reference radar global embedding vector of the candidate category label. The comprehensive similarity score of the echo signal to be tested relative to the reference radar global embedding vector of a single candidate category is calculated. The category with the highest comprehensive similarity score is selected as the lip reading recognition result.

[0013] In the above scheme, calculating the comprehensive similarity score of the echo signal to be tested relative to the global embedding vector of a reference radar of a single candidate category includes: The cosine similarity between the radar global embedding vector of the L2-normalized echo signal and the text pronunciation anchor points of each category label is obtained as the first similarity. The larger the value, the higher the similarity between the echo signal and the text pronunciation anchor points. The cosine similarity between the radar global embedding vector of the L2-normalized echo signal under test and the global embedding vector of the reference radar for each category is obtained as the second similarity. The larger the value, the higher the similarity between the echo signal under test and the global embedding vector of the reference radar. The first similarity and the second similarity are weighted and summed to obtain the comprehensive similarity score of the echo signal to be tested relative to the category. By adaptively balancing the physical prior of the text pronunciation anchor point representation with the global embedding vector of the reference radar, it has strong environmental robustness and user generalization ability.

[0014] This application provides a few-sample lip-reading recognition device based on millimeter-wave radar, including: The echo signal acquisition and processing module is used to acquire the echo signal to be tested transmitted back by the FMCW radar, and to perform signal preprocessing on the echo signal to be tested to obtain a micro-Doppler image characterizing lip changes. The lip reading recognition model training module is used to acquire sample batch data and train the image and text bimodal aligned lip reading recognition model using a two-stage warm-up strategy. Each batch of samples in the sample batch data contains a micro-Doppler image, the lip reading instruction text corresponding to the micro-Doppler image, and a category label, to obtain the trained lip reading recognition model. The lip reading module is used to input the micro-Doppler image of the echo signal to be tested into the trained lip reading model to obtain the radar global embedding vector of the echo signal to be tested. It calculates the similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label in the offline pre-stored preset lip reading command set, and takes the result with the highest similarity as the lip reading result.

[0015] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. This application achieves static clutter filtering by performing a distance-dimensional Fourier transform and first-order difference on the windowed echo signal to obtain a time-distance heatmap. Then, the distance bin for the lip reference target is determined by accumulating the echo energy to suppress the influence of random noise. The optimal distance window for the lips is obtained by superimposing left and right expansion, ensuring continuous signal capture even with small head movements. Finally, the echo signal within the optimal distance window for the lips is extracted from the time-distance heatmap. After filtering out frequencies of human dynamic interference such as breathing and heartbeat, a short-time Fourier transform is performed to obtain a micro-Doppler image that accurately represents the micro-movement speed changes of the lips and vocal cords. This reduces the impact of invalid noise on recognition accuracy from the source, significantly improves signal stability in complex indoor environments, provides a high-quality input foundation for subsequent steps, effectively solves the problem of effective features being submerged in existing lip reading recognition, and reduces the misjudgment rate of lip reading recognition.

[0016] 2. For the radar encoder branch in the lip-reading model, firstly, a multi-layer two-dimensional convolutional network is used to extract the time-frequency texture features corresponding to lip vibrations layer by layer from the micro-Doppler images in a single batch of samples. Then, a lightweight self-attention module is used to enhance the time-frequency response of lip pronunciation in the time-frequency texture features to obtain time-frequency texture enhancement features. Then, the time-frequency texture enhancement features are reorganized along the time dimension and the contextual dependencies of the entire pronunciation process are modeled through the first Bi-LSTM network to obtain radar temporal features. Finally, the radar temporal features are globally aggregated to obtain the corresponding radar global embedding vector, which is used for cross-modal similarity retrieval in the first stage of training and inference of the lip-reading model. This solves the defect of existing lip-reading models that only extract static spatial textures and lose the temporal correlation before and after pronunciation.

[0017] 3. Regarding the text encoder branch in the lip-reading recognition model, this application converts the lip-reading instruction text corresponding to the micro-Doppler map in a single batch of samples into a phoneme feature sequence. Then, it converts each phoneme in the sequence into a computable prior pronunciation attribute vector. The prior pronunciation attribute vector is then mapped to obtain a correction term for the preset prior weights, avoiding weight shift. This is then superimposed with the preset prior weights to obtain a soft mask. The soft mask is then used to multiply the initial phoneme embedding vectors of each phoneme in the phoneme feature sequence element-wise to amplify the feature assignment of highly visible pronunciations. Finally, a second Bi-LSTM network is used to model the linguistic contextual dependencies before and after pronunciation, outputting phoneme temporal features to obtain text speech anchors as a text modality benchmark representation. This embeds linguistic and physical priors into the model training process, significantly improving the effectiveness of model learning and solving the problems of weak generalization ability and easy overfitting in small sample scenarios in existing lip-reading recognition models.

[0018] 4. In this application, a warm-up strategy is used to train the lip-reading recognition model. In the first stage, supervised contrastive loss is used to achieve accurate matching between the radar global embedding vector and the text pronunciation anchor point, and to build a stable embedding space. In the second stage, supervised contrastive loss, soft-DTW loss and joint mask regularization loss are superimposed to achieve accurate matching between the radar global embedding vector and the phoneme. At the same time, the noise gradient caused by the instability of the initial features is avoided, making the alignment of the two-branch features more stable, which greatly improves the model's adaptability to different speech rates and different pronunciation habits.

[0019] 5. Regarding lip reading recognition, this application, in Zero-shot mode, relies solely on pre-stored text pronunciation anchors of various categories to achieve lip reading command recognition without any labeled samples. In K-shot mode, it performs fusion inference based on a small number of labeled reference radar global embedding vectors and text pronunciation anchors to obtain lip reading recognition results. It can adapt to zero-sample and few-sample inference, solving the problems of high data annotation cost and few samples in existing lip reading recognition, and has extremely strong environmental robustness and user generalization ability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0021] Figure 1 A flowchart illustrating the few-sample lip reading recognition method based on millimeter-wave radar provided by this invention; Figure 2 The time-distance heatmap provided for this invention; Figure 3 The micro-Doppler image provided by this invention; Figure 4 The micro-displacement variation diagram of the lip provided by this invention; Figure 5 The confusion matrix under Random split provided by this invention; Figure 6 The confusion matrix under subject-independent split provided by this invention; Figure 7 The recognition rate under K-shot provided by this invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0023] This application provides a few-sample lip-reading recognition method based on millimeter-wave radar, such as... Figure 1 As shown, the method includes the following steps: (1) Obtain the echo signal detected in real time by FMCW radar, i.e. the echo signal to be measured, and perform signal preprocessing on the echo signal to be measured to obtain a micro-Doppler image characterizing the change in the speed of lip micro-movement.

[0024] To achieve lip-reading recognition, the hardware configuration is first required: a 77GHz millimeter-wave radar sensor with two transmit and four receive antennas (TI AWR1642BOOS+DCA1000), employing a MIMO architecture with two transmit antennas and four receive antennas. This architecture supports various application scenarios, including gesture recognition, micro-vibration detection, human presence sensing, and blind spot monitoring. The operating frequency band is 77-81GHz, with a bandwidth of 3.9GHz, a frame rate of 100fps, and each frame contains 128 linear frequency modulated pulses. The radar is placed horizontally (initial angle 0°) at a distance of approximately 0.5-0.8m from the mouth area of ​​a seated subject.

[0025] The process involves signal preprocessing of the echo signal to obtain micro-Doppler images characterizing the micro-movement velocity changes of the lips and vocal cords, including: (11) Perform range-dimensional Fourier transform on the windowed echo signal to obtain the initial time-range heatmap and the corresponding frequency; wherein, the initial time-range heatmap reflects the echo energy at different distances at each time of detection, that is, the horizontal axis is time, the vertical axis is distance, and the pixel amplitude represents the radar echo energy at that time and distance.

[0026] Understandably, millimeter-wave radar transmits a long string of continuous I and Q sampled digital signals. First, according to hardware timing specifications, I and Q signals must be paired one-to-one to form complex signals. Then, following Fortran's column-first memory rules, the one-dimensional complex sequence is rearranged into a four-dimensional matrix (Frames, Chirps, RX, ADC) to describe the acquired multi-dimensional signal data, corresponding to frames, chirps, receiving antennas, and sampled points after analog-to-digital conversion. This ensures that each position in the array corresponds to the actual physical sampling scenario, providing structurally correct baseband data for subsequent multi-dimensional Fourier transforms of distance, velocity, and angle.

[0027] The above steps apply Hanning window weighting to the four-dimensional matrix ADC, then perform a range-dimensional Fourier transform along the ADC axis, fusing the echo energy of all frequency-modulated pulses and multiple receiving antennas within a single frame to obtain an initial time-range heatmap, i.e., the echo signal of the target echo signal after pulse compression, such as... Figure 2 As shown in the figure, the echo energy at different collection times and distances is as follows: static objects (walls, tables) appear as continuous horizontal bright lines, while slight movements of the human head and lips appear as small up-and-down undulating bright trajectories.

[0028] The Hanning window is used to suppress Fourier transform spectral leakage, meaning that when the target distance is not within the standard range, the echo energy will not spread extensively to other range cells. Rectangular windows, Hamming windows, and Lexman windows can also be used to reduce spectral leakage. Since the frequency of the echo signal being measured is proportional to the target distance, the frequency points obtained after the Fourier transform correspond to the target distance, thus providing the target's distance information.

[0029] (12) The initial time-distance heatmap is subjected to first-order difference according to the same distance and adjacent frames to suppress static clutter, and the time-distance heatmap is obtained. Among them, the time-distance heatmap only retains the signal component of the moving lip target. That is, the square of the echo signal amplitude generated by local movements such as lips and jaw has been completely canceled out by the energy corresponding to the stationary object.

[0030] First-order difference is one of the core methods in Moving Target Indication (MTI) cancellation processing. It involves differentiating adjacent frames at the same distance to remove static background interference (such as walls or tables), or in other words, eliminating the signal components corresponding to the static background while retaining the signal components of the moving target, such as the lips. Equivalent to a time-domain high-pass filter, it creates a notch at stationary targets, making it ideal for near-field extraction of minute Doppler motion signals like those from the lips.

[0031] (13) Accumulate the echo energy of all frames of each range cell in the preset range cell set, and select the range cell with the largest accumulated energy as the lip reference target range cell.

[0032] Among them, the candidate distance chamber is a pre-defined effective near-range detection range for human lips, and all distance chambers within the range constitute the distance chamber set.

[0033] (14) Taking the lip reference target distance compartment as the center, expand the preset window width to the left and right sides, that is, multiple distance compartments, to obtain the optimal lip distance window that covers the micro-movement of the mouth.

[0034] The aforementioned frames represent the total duration of continuous radar data acquisition corresponding to a complete lip-reading sound. Random noise is suppressed and the center of mouth micro-movements is stably located through full-duration energy accumulation. In other words, the sum of all echo energy at various distances is calculated, and the distance with the highest energy is the position of the mouth on the face.

[0035] Understandably, when a person speaks, their head will move slightly back and forth. If only a single distance chamber is locked, the signal will be lost frequently. By expanding left and right, the signal can be continuously captured even when the head moves slightly.

[0036] (15) Extract the echo signal within the optimal distance window of the lip from the time-distance heatmap, filter out frequencies outside the preset range using a bandpass filter, perform short-time fourier transform (STFT) processing on the filtered signal, and extract a micro-Doppler image with time as the horizontal axis and Doppler frequency as the vertical axis, as shown below. Figure 3 As shown.

[0037] Understandably, based on the typical velocity range of lip and jaw movements, such as 5-30 cm / s, a bandpass filter is used to filter out frequencies outside the preset range of 12Hz-150Hz, filtering out dynamic human interference such as breathing and heartbeat, while accurately preserving the micro-movement signals of the lips. STFT processing is performed with a sliding window size of 512 and a step size of 128. The micro-Doppler image is used as the input to the radar encoder, and its brightness represents the micro-movement energy.

[0038] It should be noted that by extracting the effective distance zone of the lips from the time-distance heatmap, and then extracting the features within this distance zone to form a micro-Doppler image, the differences in the corresponding echo signals can be distinguished. The phase displacement reflected by the lip micro-movement curve can also be superimposed to fully describe the spatiotemporal dynamic characteristics of the lip articulation movement.

[0039] For example, from Figure 2 It can be seen that the range bins corresponding to the continuous appearance of high-energy color blocks in the 0-1.5s interval prove that the lip target is stably within the effective radar observation range. The energy color blocks fluctuate intermittently over time, corresponding to the changes in radar echo intensity caused by lip opening and closing and facial muscle movements during pronunciation. Figure 3 In the 0-1.5s interval, numerous concentrated Doppler energy clusters appear, with velocity distribution spanning both positive and negative ranges. This indicates that the lips alternately approach and move away from the radar. The multiple discrete energy peaks in the figure correspond to the instantaneous micro-motion velocities generated by the rapid changes in lip shape during different phoneme switching. Figure 4 The curve represents the micro-displacement of the lips, which is the waveform of continuous micro-displacement of the lips calculated from radar phase information. The horizontal axis represents time, and the vertical axis represents radial micro-displacement, which intuitively records the displacement fluctuations of the lips opening and closing and the coordinated movement of facial muscles. Figure 4 The waveform oscillates violently, with peak displacement reaching ±0.3mm, and contains numerous rapid peaks and troughs. These dense fluctuations correspond to a series of actions during the pronunciation of continuous syllables, including lip closure, lip rounding, and jaw movement, with high frequency and large amplitude fluctuations in displacement changes. As shown in the three figures above, the 0-1.5s interval represents the vocalization phase, during which the lips remain continuously within the radar's observation range, exhibiting continuous and rapid lip movement, simultaneously generating rich speed changes and millimeter-level displacement fluctuations.

[0040] Similarly, within the 1.5-5 second time period, Figure 2 It can be seen that there is still a weak echo at the target distance, but the high-intensity red and yellow energy spots no longer appear. Figure 3 There are almost no obvious Doppler energy clusters, only scattered and weak noise spots. After filtering, low-frequency physiological micro-movements such as heartbeat and respiration are suppressed, and there are no effective Doppler features without lip movement. Figure 4 In this segment, the waveform transforms into a smooth, regular, small-amplitude, low-frequency oscillation, with a significantly reduced fluctuation amplitude. As can be seen from the above figure, after 1.5 seconds, there is a silent state. In the silent state, there is no lip-pronouncing action, the dynamic micro-motion characteristics are greatly attenuated, and only weak basic physiological noise is retained.

[0041] (2) Obtain sample batch data and use a two-stage warm-up strategy to train the image and text dual-modal alignment lip reading recognition model. Each batch of samples in the sample batch data contains micro-Doppler images, lip reading instruction text corresponding to the micro-Doppler images, and category labels to obtain the trained lip reading recognition model.

[0042] The training process for the lip-reading recognition model includes: (21) Perform the first forward propagation: Input the micro-Doppler image in the single batch of samples into the radar branch encoder for processing to obtain the corresponding radar temporal features and radar global embedding vector. Simultaneously input the lip-reading command text corresponding to the micro-Doppler image into the text branch encoder for processing to obtain the corresponding phoneme temporal features and text pronunciation anchors of each category. After performing L2 normalization on the radar global embedding vector and the text pronunciation anchors, map them to the unit hypersphere.

[0043] The category labels are used to construct a label mask matrix for positive and negative samples. The radar branch encoder includes a multi-layer 2D convolutional network, a lightweight self-attention module, and a first Bi-LSTM network, while the text branch encoder includes a second Bi-LSTM network.

[0044] Furthermore, L2 normalization is used to unify the feature measurement scale. Without normalization, the vector magnitude can interfere with similarity calculation. Cosine similarity is equivalent to vector dot product. Normalization eliminates the measurement bias caused by the magnitude, making the contrastive loss calculation purer. It also constrains the feature distribution to a unit hypersphere, stabilizing the contrastive loss gradient because the unit sphere restricts all representations to the same bounded manifold, resulting in stable gradient magnitudes and accelerating the convergence of the bi-branch embedding space alignment. After normalization and mapping, two types of features with drastically different magnitude distributions are placed in the same metric space.

[0045] (22) Perform the first batch of parameter updates: Based on the normalized radar global embedding vector and text pronunciation anchor points, construct a symmetric form of supervised contrast loss, use the calculated supervised contrast loss as the first target loss function, perform the first backpropagation to perform the first batch of model parameter updates, and iterate until the embedding space meets the conditions.

[0046] Based on the label mask matrix constructed by the above category labels, the radar global embedding features of the same category and the text pronunciation anchor points are used as positive sample pairs, and different category combinations are used as negative sample pairs. Cosine similarity is obtained by feature dot product operation to optimize the supervised contrast loss, which brings the cross-modal representation distance of the same category closer and pushes the representation distance of different categories further apart, forming category cluster boundaries in the feature space.

[0047] For example, the normalized radar global embedding vector is used as the query and the text pronunciation anchor point is used as the key to calculate the first cross-entropy loss, which maximizes the similarity of positive samples and minimizes the similarity of negative samples; the normalized text pronunciation anchor point is used as the query and the normalized radar global embedding vector is used as the key to calculate the second cross-entropy loss; the average of the two types of cross-entropy losses is used to obtain the supervised contrast loss.

[0048] The embedding space meets the conditions, and the determination process includes: The label mask matrix constructed based on the category labels uses radar global embedding features of the same category and text pronunciation anchors as positive sample pairs, and combinations of different categories as negative sample pairs. When the similarity fluctuation amplitude of the positive sample pairs and the similarity fluctuation amplitude of the negative sample pairs in the verification set are both less than the preset fluctuation threshold in multiple consecutive iteration cycles, it is determined that the cross-modal embedding space formed by the radar global embedding vector and the text pronunciation anchors meets the conditions.

[0049] (23) Perform the second forward propagation and parameter update: Perform the second forward propagation consistent with (21), and perform the second backpropagation based on the calculated second objective loss function to perform the second batch update of model parameters; wherein, the second objective loss function is composed of supervised contrastive loss, Soft-DTW loss and joint mask regularization loss weighted together; Repeat steps (21) to (23) until the lip reading recognition model reaches the convergence condition, and output the trained lip reading recognition model.

[0050] For example, let the batch size be B, and the normalized radar global embedding vector of the i-th sample be... The text pronunciation anchor point is The set of positive samples for the i-th sample is The temperature coefficient is .

[0051] The first item is a comparative loss monitoring. The expression for the first objective loss is: ; Among them, the loss of radar in the text direction The expression is: ; Text-to-radar direction loss Use the same format.

[0052] set up Let be the soft monotonic dynamic alignment score between the i-th radar temporal feature and the j-th phoneme temporal feature, then the Soft-DTW loss is... The expression is: .

[0053] Joint mask regularization loss The expression is: ; in: ; .

[0054] In the formula, Let be the soft mask weight for the l-th phoneme. is the preset prior weight corresponding to the l-th phoneme, and L is the length of the phoneme sequence; The overall training objective of the model is the second objective loss function. The expression is: ; Where e represents the training round, , The weights of the corresponding terms change as the training phase progresses.

[0055] The supervised contrastive loss is used to maintain cross-modal feature alignment constraints at the global category level, while the joint mask regularization loss includes a prior preservation regularization term and a mean constraint regularization term. Soft-DTW loss is a dynamic time warping (DTW) algorithm that transforms the originally non-differentiable DTW distance into a differentiable loss function by introducing a smoothing parameter. Its core idea is to calculate the weighted distance between two time series under different time alignment methods. The lip-reading model trained using these methods solves the problems of speech rate scaling and mismatched pronunciation lengths.

[0056] The aforementioned joint mask regularization loss is a weighted sum of the prior preservation regularization term and the mean constraint regularization term. The prior preservation regularization term, i.e., the first regularization term, is used to constrain the correction magnitude of the preset prior weights, i.e., to penalize the mask from large deviations from the artificial physical prior; the mean constraint regularization term, i.e., the second regularization term, is used to constrain the mean of the soft mask to be within a preset reasonable range, avoiding simultaneous amplification or reduction of the weights.

[0057] (3) Input the micro-Doppler image of the echo signal to be tested into the trained lip reading recognition model to obtain the radar global embedding vector of the echo signal to be tested. Calculate the similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label in the offline pre-stored preset lip reading command set. Use the result with the highest similarity as the lip reading recognition result.

[0058] This involves calculating the similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label in a pre-stored offline set of lip-reading commands. The result with the highest similarity is taken as the lip-reading recognition result, including: (31) Obtain the text pronunciation anchor points of each category label in the preset lip reading command set through the trained lip reading recognition model.

[0059] (32) In Zero-shot mode, calculate the cosine similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label stored offline, and take the category with the highest cosine similarity as the lip reading recognition result.

[0060] (33) In K-shot mode, for each candidate category, multiple radar global embedding vectors of preset echo signals with category labels are obtained through the radar encoder branch of the lip reading recognition model. The mean value of the corresponding radar global embedding vector is used as the reference radar global embedding vector of the category label. The comprehensive similarity score of the echo signal to be tested relative to the reference radar global embedding vector of a single candidate category is calculated. The category with the highest comprehensive similarity score is selected as the lip reading recognition result.

[0061] The above scheme calculates similarity to complete the inference of text translation. It supports adaptive adjustment of the contribution ratio of text pronunciation anchors and reference radar global embedding vectors, ensuring recognition accuracy with few samples.

[0062] Specifically, the comprehensive similarity score of the echo signal to be tested relative to the global embedding vector of a reference radar for a single candidate category is calculated, including: (331) Obtain the cosine similarity between the radar global embedding vector of the L2-normalized echo signal and the text pronunciation anchor points of each category label, and use it as the first similarity. .

[0063] (332) Obtain the cosine similarity between the radar global embedding vector of the L2-normalized echo signal to be measured and the global embedding vector of the reference radar of each category, and use it as the second similarity. .

[0064] (333) The first similarity and the second similarity are weighted and summed to obtain the comprehensive similarity score of the echo signal to be tested relative to the category. .

[0065] The specific expression for the overall similarity score is as follows: ; in, For the reference radar global embedding vector of category c, The fusion coefficient is... This represents the overall similarity score of the echo signal x to be tested relative to category c. The vector is the radar global embedding vector output by the radar encoder branch of the echo signal x to be measured, and then normalized by L2. This is the text pronunciation anchor point corresponding to category c.

[0066] In the above formula For cross-modal text anchor similarity, For reference, the global embedding vector similarity of the radar, Specifically, when When it degenerates into plain text anchor retrieval; when It degenerates into traditional prototype network-based few-shot retrieval. Experiments show that the optimal 1-shot result appears in... This indicates that real radar support samples can provide primary category evidence, while vocal observability anchors provide auxiliary constraints.

[0067] In a specific embodiment of the present invention, the micro-Doppler images in a single batch of samples are input into a radar branch encoder for processing to obtain the corresponding radar time-series features and radar global embedding vector, including: (201) Input the micro-Doppler images in a single batch of samples into a multi-layer two-dimensional convolutional network to extract the time-frequency texture features corresponding to lip vibration layer by layer; that is, use the local perception, weight sharing and pooling characteristics of the convolutional network to automatically extract features from the micro-Doppler images.

[0068] (202) The time-frequency texture features are used as input to a lightweight self-attention module. The time-frequency response of lip pronunciation in the time-frequency texture features is enhanced by the lightweight self-attention module to obtain time-frequency texture enhancement features.

[0069] Among them, the lightweight self-attention module is used to adaptively learn the weights of regions in the time-frequency plane that are strongly correlated with lip articulation movements, amplify the lip movement Doppler ridge features, suppress the invalid time-frequency responses corresponding to environmental clutter and physiological interference, and obtain time-frequency texture enhancement features.

[0070] (203) The time-frequency texture enhancement features are reorganized into a one-dimensional temporal feature sequence along the time dimension and input into the first Bi-LSTM network to model the preceding and following dependencies in the pronunciation process. The output radar temporal features are then globally aggregated to obtain the radar global embedding vector.

[0071] In this system, the first Bi-LSTM network's forward LSTM processes the input in the natural order of the sequence, from the beginning to the end of the pronunciation, primarily capturing the influence of preceding lip movements on subsequent mouth shapes. The backward LSTM processes the input in reverse order, traversing from the end of the pronunciation to the beginning, primarily capturing the constraints of subsequent lip movements on preceding pronunciation. At each time step, the forward and backward hidden states are concatenated as vectors to fully cover the contextual dependencies of the entire pronunciation process, resulting in radar temporal features.

[0072] In the above steps, the radar temporal features retain the hidden state at each time step for calculating the local dynamic temporal alignment loss, while the radar global embedding vector is used for supervised contrastive loss training and cross-modal similarity retrieval during the inference phase. The purpose of global aggregation is to integrate this temporal information into a fixed-dimensional vector. Common aggregation methods include average pooling and max pooling. The resulting vector contains global information of the entire sequence and can also be called global embedding features.

[0073] By modeling the dynamic temporal patterns of continuous lip vibrations in micro-Doppler spectra using a first Bi-LSTM network, this method overcomes the shortcomings of traditional methods that only extract static spatial textures and lose the temporal correlation before and after pronunciation.

[0074] In a specific embodiment of the present invention, the lip-reading instruction text corresponding to the micro-Doppler image is input into a text branch encoder for processing to obtain the corresponding phoneme temporal features and pronunciation anchors for each category of text, including: (211) After converting the lip-reading instruction text corresponding to the micro-Doppler image into a pinyin sequence, it is decomposed into syllables to obtain a phoneme feature sequence. The initial phoneme embedding vector and the prior pronunciation attribute vector describing the physical pronunciation features of each phoneme in the phoneme feature sequence are obtained.

[0075] That is, the text string is first converted into Pinyin, and then the Pinyin is broken down into phonemes, such as the initials b, p, m, f, zh, ch, sh, and the finals ang, ong, ao, ian, etc. Each phoneme is mapped to a learnable initial phoneme embedding vector and a prior articulation attribute vector. The prior articulation attribute vector describes physical articulation features such as bilip closure, lip-teeth contact, degree of lip rounding, mouth opening, mandibular movement, visible vowels, alveolar / dental sounds, palatal / retroflex sounds, soft palatal sounds, nasals, fricatives, plosives / affricates, and glissandos.

[0076] The prior articulation attribute vector is a vector defined by professionals based on linguistics and oral articulation; it is a pre-defined vector describing the lip movements corresponding to each phoneme. It is used to calculate pre-defined prior weights and generally consists of 0-1 values ​​corresponding to the physical articulation feature dimension. It is not used in training. In other words, the prior articulation attribute vector of each phoneme indicates its visibility to radar, preventing the model from deviating from actual articulation rules. The learnable initial phoneme embedding vector is an adaptive vector for each phoneme, automatically learning the differences in the true physical articulation features of different phonemes. This vector is iteratively updated by the model.

[0077] (212) The preset prior weights are superimposed with the correction terms of the preset prior weights to obtain a soft mask; wherein, the larger the preset prior weights are, the higher the observability of the corresponding phonemes relative to the radar.

[0078] The preset prior weights are based on a weighted scoring of various artificially defined physical features of pronunciation. High scores are assigned to external lip movement features (rounded lips, bilip closure, fricatives) and low scores are assigned to internal oral sounds (soft palate, alveolar plosives). The values ​​range from 0 to 1, representing the intensity of micro-movements that the phoneme can theoretically be captured by millimeter-wave radar, or the inherent strength of the lip radar micro-movement response that the phoneme can theoretically produce.

[0079] In actual pronunciation, even if the primary articulation point of some phonemes is located inside the oral cavity, indirect radar responses may occur through jaw movements, facial muscle coordination, or cophonation transitions. Therefore, the core function of soft masking is to adaptively adjust the importance of different time steps or features based on the current context or task requirements, thereby enhancing the model's ability to capture key information. This dynamic weighting approach helps improve the model's performance and generalization ability, especially when processing long sequences or complex semantics.

[0080] (213) Multiply the initial phoneme embedding vector of each phoneme in the phoneme feature sequence element by element with the soft mask to amplify the feature amplitude of the high lip movement visible phonemes and retain the weak temporal occupancy information of the low visible oral cavity phonemes to obtain the weighted phoneme feature sequence.

[0081] (214) Input the weighted phoneme feature sequence into the second Bi-LSTM network to model the contextual dependency of phoneme pronunciation. The output phoneme temporal features are weighted and summed through the soft mask to obtain the text pronunciation anchor. Simply put, the text pronunciation anchor is a fixed-length feature prototype vector that is compressed into a single instruction after integrating all the physical pronunciation knowledge such as initials, finals, lip opening / rounding / jaw movement with temporal context.

[0082] Understandably, the input to the text encoder branch is the text of the lip-reading command to be recognized, and the output is the text pronunciation anchor point after weighting the phoneme temporal features. This is achieved by using a continuous, smooth soft mask as the weighting coefficient for pooling, where highly visible phonemes contribute more to the final text anchor point, while low-visibility internal sounds provide temporal context supplementation, enabling the text anchor point to possess both lip-movement prior and complete pronunciation temporal information. This output serves as the global baseline representation of the text modality, participating in the global supervised contrastive loss calculation together with the radar encoder output (i.e., the radar global embedding vector). It is also used for cross-modal cosine similarity retrieval and matching in Zero-shot or K-shot inference. This design allows highly visible phonemes to contribute more to the text anchor point while retaining the weak contribution and temporal placeholder role of low-visibility units. In other words, the text encoder branch and the radar encoder branch form a two-branch network of the lip-reading recognition model.

[0083] The second Bi-LSTM network uses a forward LSTM to traverse the phoneme order (e.g., b→a→m), primarily capturing the sequence of pronunciation. A backward LSTM traverses in reverse order, mainly capturing the linkage constraints of subsequent phonemes on the lip shapes of preceding phonemes. At each time step, the forward and backward hidden states are concatenated as vectors to obtain the temporal hidden states representing the complete context of each phoneme. By modeling the temporal logic of phonemes through the second Bi-LSTM network, physical pronunciation priors such as lip and teeth movements, lip rounding, and jaw movements are fused with the temporal context to construct text features with physical interpretability.

[0084] The comparison shows that the first Bi-LSTM network outputs continuous, equally spaced radar temporal features, while the second Bi-LSTM network outputs discrete temporal hidden states, i.e., phoneme temporal features. In subsequent steps, the radar global vector output by the first Bi-LSTM network and the text pronunciation anchor points output by the second Bi-LSTM network are used for category-level cross-modal matching to solve the problems of speech rate scaling and inconsistent lip-sync duration.

[0085] The above scheme converts the command text into a phoneme feature sequence and constructs a priori pronunciation attribute vector and a learnable initial phoneme embedding vector based on physical pronunciation features such as bilip closure, lip-teeth contact, lip rounding, mouth opening, and mandibular movement. An adaptive soft mask is obtained by superimposing the priori pronunciation attribute vector with double constraints to perform visibility weighting on the initial phoneme embedding vector, thereby achieving cross-modal alignment between radar micro-Doppler features and pronunciation physical priors. This enables high-precision, occlusion-resistant millimeter-wave radar lip reading recognition with only a few samples required.

[0086] The soft mask is obtained by adding a correction term to the preset prior weights, including: First, the prior articulation attribute vector of each phoneme is mapped to the initial correction term of the preset prior weight of each phoneme. The value range of the initial correction term is [0,1], which is used to reflect the prior strength of the external articulation action of the corresponding phoneme being observed by the millimeter-wave radar. This is used to model the actual scenario in which theoretically low visible sounds rely on the linkage of the jaw and facial muscles to generate indirect radar echoes.

[0087] That is, a priori articulation attribute vector is constructed based on the physical articulation characteristics such as lip closure, lip-teeth contact, lip rounding, mouth opening, and mandibular movement. , is a 16-dimensional vector, that is Simultaneously, the preset prior weights of the phoneme are obtained. This weight represents the prior strength of the external articulation action of the corresponding phoneme as observed by millimeter-wave radar. For example, bilip closure / release phonemes have a higher initial prior, while velar and glottal sounds have a lower initial prior.

[0088] The prior articulation attribute vector of the l-th phoneme The mapping is to a scalar correction value, and defined as the initial correction term for the l-th phoneme. .

[0089] Then, using the hyperbolic tangent function and a preset correction amplitude coefficient... The initial correction terms for each phoneme are constrained to obtain the correction terms with preset prior weights for each phoneme. .

[0090] For example, The expression is: ; Finally, the preset prior weights of each phoneme are summed with the corresponding correction terms to obtain the soft mask for each phoneme.

[0091] That is, the l-th phoneme and Add them together and limit the result to the range of 0 to 1 to obtain the soft mask for that speed of sound. The expression is: .

[0092] This application provides a few-sample lip reading recognition device based on millimeter-wave radar, including an echo signal acquisition and processing module, a lip reading recognition model training module, and a lip reading recognition module.

[0093] The echo signal acquisition and processing module is used to acquire the echo signal to be tested transmitted back by the FMCW radar, and to perform signal preprocessing on the echo signal to be tested to obtain a micro-Doppler image characterizing the velocity changes of the lips and vocal cords.

[0094] The lip reading recognition model training module is used to acquire sample batch data and train an image and text bimodal lip reading recognition model using a two-stage warm-up strategy. Each batch of samples in the sample batch data contains a micro-Doppler image, the lip reading instruction text corresponding to the micro-Doppler image, and a category label, thus obtaining the trained lip reading recognition model.

[0095] The lip reading module is used to input the micro-Doppler image of the echo signal to be tested into the trained lip reading model to obtain the radar global embedding vector of the echo signal to be tested. It calculates the similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label in the offline pre-stored preset lip reading command set, and takes the result with the highest similarity as the lip reading result.

[0096] In summary, this application significantly improves the accuracy and environmental adaptability of lip reading by signal preprocessing and a trained lip reading model. On the other hand, it breaks the dependence of traditional methods on massive labeled data by using soft masking and few-shot inference, ensuring the generalization performance of recognition. It can adapt to complex and extreme scenarios such as no light, strong light, and high noise, and has a strong resistance to environmental interference.

[0097] For the aforementioned lip-reading recognition model, a lip-reading dataset of 25 categories in Chinese and English (including single characters, words, phrases, and sentences, encompassing words with similar forms but different meanings, words with reversed word order, and very short words) was used, with 10 subjects and a total of 9500 valid samples. Under the current validation protocol, with a combination of various data augmentation strategies (temporal masking, temporal scaling, Gaussian noise, and temporal shifting), the model achieved an accuracy of 81.31% in recognizing 1-shot no-sight commands, demonstrating good stability in distance, angle robustness, and ablation experiments. Regarding closed-set performance, the model achieved accuracies of 92.40% and 74.20% under random split and subject-independent split, respectively. The confusion matrix under random split is shown below. Figure 5 As shown, the confusion matrix under subject-independent split (8 users training, 2 users testing) is as follows: Figure 6 As shown.

[0098] exist Figure 5 and Figure 6 In the diagram, the horizontal axis represents the model's predicted class, and the vertical axis represents the sample's true class. From... Figure 5 As can be seen from the diagonal lines, the single-class recognition accuracy for a large number of categories reaches over 90%, and some categories achieve 100% accuracy. The off-diagonal lines show only a few light-colored blocks, indicating low inter-class confusion; only a few categories exhibit slight confusion, such as a small number of cross-class misclassifications around categories 13, 14, 23, and 25. Compared to Random Split, Figure 6 The diagonal values ​​decreased overall, while the number of off-diagonal light-colored blocks increased significantly, indicating a marked increase in inter-category confusion. Commands with similar pronunciations and lip movement patterns were more likely to be confused with each other, such as those in categories 3, 9, and 23.

[0099] from Figure 5 and Figure 6 It can be seen that the model performance under the Random split condition is significantly better than that under the subject-independent condition. Under the subject-independent condition, the training set and the test set consist of completely different subjects. There are individual differences in the amplitude of lip movements, facial structure, and relative posture to the radar among different people, resulting in a shift in the distribution of the millimeter-wave radar micro-Doppler signal. However, even in the more challenging cross-person testing scenario, the model still maintains effective class discrimination ability. The categories where confusion occurs mostly correspond to instructions with similar lip pronunciation movement patterns, which is consistent with the formation mechanism of the millimeter-wave radar micro-motion signal, and the model's predicted behavior has physical interpretability.

[0100] To further evaluate cross-user few-shot adaptation capability, a leave-one-out protocol was adopted. The model was trained using 20 seen-class episodes from 9 subjects each time, and 25-way K-shot identification was performed on the unseen-class episode from the remaining subject. Results for different subjects are as follows: Figure 7 As shown.

[0101] Visible differences in performance existed among subjects, but the overall trend remained consistent. From Figure 7 It can be seen that as the preset echo signal increased from 1-shot to 7-shot, the recognition accuracy of each subject steadily improved. Most test users achieved an accuracy of approximately 75% or higher under the 1-shot condition, indicating that a single labeled radar sample can provide an effective individualized prototype for unseen commands. Subject 6's 1-shot performance was relatively low, possibly related to its smaller vocal amplitude or weaker micro-Doppler energy; when the number of preset echo signals increased, the subject's accuracy recovered significantly, indicating that the reference radar global embedding vector can partially compensate for individual vocal habits and signal strength differences.

[0102] This application provides a computer-readable storage medium for storing a computer program used in the method described.

[0103] This application provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the method.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A few-sample lip-reading recognition method based on millimeter-wave radar, characterized in that, Includes the following steps: The echo signal to be tested transmitted back by the FMCW radar is acquired, and the echo signal to be tested is preprocessed to obtain the micro-Doppler image to be analyzed, which characterizes the changes in the lips. A batch of sample data is acquired, and a warm-up strategy is used to train a lip-reading recognition model that aligns the image and text modes. Each batch of samples in the sample data contains a micro-Doppler image, the lip-reading instruction text corresponding to the micro-Doppler image, and a category label, thus obtaining the trained lip-reading recognition model. The lip-reading recognition model includes a dual-branch encoder, and each encoder includes a Bi-LSTM network. The micro-Doppler image to be analyzed is input into the trained lip-reading recognition model to obtain the lip-reading recognition results.

2. The few-sample lip-reading recognition method based on millimeter-wave radar as described in claim 1, characterized in that, The preprocessing of the echo signal to obtain the micro-Doppler image characterizing lip changes includes: The windowed echo signal to be measured is subjected to a range-dimensional Fourier transform to obtain an initial time-range heatmap; wherein, the initial time-range heatmap reflects the echo energy at different distances at different times during the detection. The initial time-distance heatmap is processed by first-order difference according to the same distance and adjacent frames to obtain a time-distance heatmap; The echo energy of all frames in each distance chamber within the preset distance chamber set is accumulated, and the distance chamber with the largest accumulated energy is selected as the lip reference target distance chamber; wherein, the distance chamber set consists of all distance chambers within the preset effective near-range detection range of the human mouth; Using the lip reference target distance bin as the center, multiple distance bins are expanded to the left and right sides to obtain the optimal lip distance window; After extracting the echo signal to be tested within the optimal distance window of the lip in the time-distance heatmap, the frequency outside the preset range is filtered out by a bandpass filter. Then, a short-time Fourier transform is performed on the filtered echo signal to be tested to extract the micro-Doppler image to be analyzed with time as the horizontal axis and Doppler frequency as the vertical axis.

3. The few-sample lip-reading recognition method based on millimeter-wave radar as described in claim 1, characterized in that, The training process for the lip-reading recognition model includes: The first step is to perform the first forward propagation: the micro-Doppler images in a single batch of samples are input into the first branch encoder for processing. The result of global aggregation of the temporal hidden states output by the first Bi-LSTM network of the first branch encoder is used as the radar global embedding vector. Simultaneously, the lip-reading text corresponding to the micro-Doppler images is input into the second branch encoder for processing. The result of weighting the temporal hidden states output by the second Bi-LSTM network of the second branch encoder is used as the text pronunciation anchor points for each category. The radar global embedding vector and the text pronunciation anchor points are then mapped to the unit hypersphere after L2 normalization. The first branch encoder includes a multi-layer two-dimensional convolutional network, a lightweight self-attention module, and a first Bi-LSTM network, while the second branch encoder includes a second Bi-LSTM network. The second step is to perform the first batch of parameter updates: based on the normalized radar global embedding vector and text pronunciation anchor points, construct a symmetric form of supervised contrast loss function, use the calculated supervised contrast loss as the first target loss, perform the first backpropagation to update the model parameters in the first batch, and iterate until the embedding space meets the conditions. The third step is to perform a second forward propagation and parameter update: the second forward propagation is performed in the same manner as the first forward propagation, and a second backpropagation is performed based on the calculated second target loss to perform a second batch update of the model parameters; wherein, the second target loss is composed of a weighted sum of supervised contrast loss, soft-DTW loss, and joint mask regularization loss, and the joint mask regularization loss is the weighted result of the first regularization term and the second regularization term. The first regularization term is used to constrain the correction magnitude of the preset prior weights, and the second regularization term is used to constrain the mean of the soft mask to be within a preset reasonable range; Repeat steps one through three until the lip reading model converges, and output the trained lip reading model.

4. The few-sample lip-reading recognition method based on millimeter-wave radar as described in claim 3, characterized in that, The process involves inputting the micro-Doppler images from a single batch of samples into the first branch encoder for processing, and then globally aggregating the temporal hidden states output by the first Bi-LSTM network of the first branch encoder as the radar global embedding vector, including: The micro-Doppler images from a single batch of samples are input into a multi-layer two-dimensional convolutional network to extract the time-frequency texture features corresponding to lip vibrations layer by layer. The time-frequency texture features are used as input to a lightweight self-attention module. The time-frequency response of lip pronunciation in the time-frequency texture features is enhanced by the lightweight self-attention module to obtain time-frequency texture enhancement features. The time-frequency texture enhancement features are reorganized into a one-dimensional temporal feature sequence along the time dimension and input into the first Bi-LSTM network to model the sequential dependencies in the pronunciation process. The output temporal hidden states are then used as radar temporal features for global aggregation to obtain the radar global embedding vector.

5. The few-sample lip-reading recognition method based on millimeter-wave radar as described in claim 3, characterized in that, The step of inputting the lip-reading instruction text corresponding to the micro-Doppler image into the second branch encoder for processing, and using the weighted result of the temporal hidden state output of the second Bi-LSTM network of the second branch encoder as the text pronunciation anchor point for each category, includes: After converting the lip-reading command text corresponding to the micro-Doppler image into a pinyin sequence, it is decomposed into syllables to obtain a phoneme feature sequence. The initial phoneme embedding vector and the prior pronunciation attribute vector describing the physical pronunciation features of each phoneme in the phoneme feature sequence are obtained. A soft mask is obtained by superimposing a preset prior weight with a correction term of the preset prior weight; wherein, the larger the preset prior weight, the higher the observability of the corresponding phoneme relative to the radar. The vector formed by multiplying the initial phoneme embedding vector of each phoneme in the phoneme feature sequence with the soft mask element by element is input into the second Bi-LSTM network to model the contextual dependency relationship of phoneme pronunciation. The output temporal hidden state is used as the phoneme temporal feature. The output phoneme temporal feature is weighted and summed by the soft mask to obtain the text pronunciation anchor point.

6. The few-sample lip-reading recognition method based on millimeter-wave radar as described in claim 5, characterized in that, The step of adding a correction term to the preset prior weights to obtain a soft mask includes: The prior articulation attribute vector of each phoneme is mapped to an initial correction term of the preset prior weight of each phoneme, wherein the initial correction term is used to reflect the prior strength of the external articulation action of the corresponding phoneme as observed by millimeter-wave radar. The initial correction terms of each phoneme are restricted by the hyperbolic tangent function and the preset correction amplitude coefficient, respectively, to obtain the correction terms of the preset prior weights of each phoneme. The soft mask for each phoneme is obtained by summing the preset prior weights of each phoneme with the corresponding correction terms.

7. The few-sample lip-reading recognition method based on millimeter-wave radar as described in claim 3, characterized in that, The embedded space meets the conditions, and the determination process includes: The label mask matrix constructed based on the category labels uses radar global embedding features of the same category and text pronunciation anchors as positive sample pairs, and combinations of different categories as negative sample pairs. When the similarity fluctuation amplitude of the positive sample pairs and the similarity fluctuation amplitude of the negative sample pairs in the verification set are both less than the preset fluctuation threshold in multiple consecutive iteration cycles, it is determined that the cross-modal embedding space formed by the radar global embedding vector and the text pronunciation anchors meets the conditions.

8. The few-sample lip-reading recognition method based on millimeter-wave radar as described in claim 3, characterized in that, The process of obtaining the lip reading results includes: The trained lip-reading recognition model obtains the text pronunciation anchor points of each category label in the preset lip-reading command set; In Zero-shot mode, the micro-Doppler image to be analyzed is input into the trained lip-reading recognition model to obtain the radar global embedding vector of the echo signal to be tested. The cosine similarity between the radar global embedding vector of the echo signal to be tested and the text pronunciation anchor points of each category label stored offline is calculated, and the category with the highest cosine similarity is taken as the lip-reading recognition result. In K-shot mode, for each candidate category, multiple radar global embedding vectors of preset echo signals with labels of that candidate category are obtained through the trained lip-reading recognition model. The mean value of the corresponding radar global embedding vector is used as the reference radar global embedding vector of the candidate category label. The comprehensive similarity score of the echo signal to be tested relative to the reference radar global embedding vector of a single candidate category is calculated. The category with the highest comprehensive similarity score is selected as the lip-reading recognition result.

9. The few-sample lip-reading recognition method based on millimeter-wave radar as described in claim 8, characterized in that, The calculation of the comprehensive similarity score of the echo signal to be tested relative to the global embedding vector of a reference radar of a single candidate category includes: The cosine similarity between the radar global embedding vector of the L2-normalized echo signal and the text pronunciation anchor points of each category label is used as the first similarity. The cosine similarity between the radar global embedding vector of the L2-normalized echo signal under test and the global embedding vector of the reference radar of each category is obtained as the second similarity. The first similarity and the second similarity are weighted and summed to obtain the comprehensive similarity score of the echo signal under test relative to the category.

10. A few-sample lip-reading recognition device based on millimeter-wave radar, characterized in that, include: The echo signal acquisition and processing module is used to acquire the echo signal to be tested transmitted back by the FMCW radar, perform signal preprocessing on the echo signal to be tested, and obtain the micro-Doppler image to be analyzed, which characterizes the changes in the lips. The lip reading recognition model training module is used to acquire sample batch data and train an image and text bimodal aligned lip reading recognition model using a warm-up strategy. Each batch of samples in the sample batch data contains a micro-Doppler image, the lip reading instruction text corresponding to the micro-Doppler image, and a category label to obtain the trained lip reading recognition model. The lip reading recognition model includes a dual-branch encoder, and each encoder includes a Bi-LSTM network. The lip reading module is used to input the micro-Doppler image to be analyzed into the trained lip reading model to obtain the lip reading result.

Images