Speech content recognition and identity recognition method and system based on single millimeter wave radar

Abstract

The application relates to the technical field of lip reading, and particularly discloses a speech content recognition and identity recognition method and system based on a single millimeter wave radar. In the application, a plurality of target distance bins are first screened, and then a plurality of target azimuth angles are estimated. When the estimation is performed, an improved mode based on a CORDIC algorithm is used to solve the inverse to avoid the problem of numerical divergence of a covariance matrix, and the accuracy of the estimation is improved. The spatial micro-motion power with high precision is quickly calculated through two rounds of scanning of a Capon beam, so that a high-precision joint heat map including a distance dimension and an azimuth dimension is spliced. A rhombus shape is used to perform constant false alarm detection on the heat map to suppress clutter false alarms, and an effective cluster is accurately determined in combination with physiological constraints, so that two types of windows that do not overlap each other are adaptively segmented, and finally corresponding micro-Doppler time-frequency diagrams are formed, the high-precision effective separation of lip and throat signals under the single millimeter wave radar is realized, and the accuracy of speech content recognition and speaker authentication under the single millimeter wave radar is improved.

Description

Technical Field

[0001] This invention relates to the field of lip reading technology, and in particular to a method and system for speech content recognition and identity recognition based on single millimeter-wave radar. Background Technology

[0002] Accurate acquisition of language content and speaker recognition information is crucial in scenarios such as hearing impairment assistance, human-computer interaction, medical care, and daily communication, especially for people with hearing impairments.

[0003] Millimeter-wave radar, with its advantages of non-contact operation, resistance to light interference, certain penetration capability, and sensitivity to minute movements, is increasingly being applied in fields such as human behavior recognition, biosignal recognition, and speech-related motion perception tasks. For example, the invention patent CN116416996B discloses a multimodal speech recognition system and method based on millimeter-wave radar, which further improves the accuracy of speech recognition by fusing lip movement features and vocal cord vibration features. Compared to traditional acoustic and visual solutions, millimeter-wave radar has potential advantages such as adaptability to noisy environments, strong confidentiality, and the ability to operate continuously for 24 hours, making it an important research direction in the field of speech perception.

[0004] However, existing methods still have significant drawbacks. They rely solely on a single type of motion information during recognition, making them susceptible to interference from background noise, echoes, and simultaneous speech by multiple speakers. Furthermore, they fail to fully utilize additional information such as vocal cord vibration, macroscopic lip movements, and microscopic throat vibrations, resulting in incomplete representation of the pronunciation content and speaker characteristics. There is an urgent need to propose a speech content recognition and identity verification method based on single-millimeter-wave radar, achieving effective separation of lip and throat signals under single-millimeter-wave radar, thereby enabling speech content recognition and speaker authentication. Summary of the Invention

[0005] To address the problems existing in the prior art, this application provides a method and system for speech content recognition and identity recognition based on single millimeter-wave radar. This method achieves high-precision and effective separation of lip and throat signals under single millimeter-wave radar, thereby improving the accuracy of speech content recognition and speaker authentication under single millimeter-wave radar.

[0006] This application provides a method for speech content recognition and identity recognition based on a single millimeter-wave radar, including the following steps: The preprocessed multi-echo signals are subjected to distance-dimensional Fourier transform to obtain a two-dimensional heat map including distance and time dimensions. Based on the phase of each distance cell in the two-dimensional heat map, multiple target distance cells that match human micro-movements are selected. Based on the target range cell, the preprocessed multi-echo signals are subjected to two-stage DOA estimation using the Capon algorithm, and the spatial micro-motion power at each spatial angle is calculated to construct a joint heatmap including range and azimuth dimensions. Each spatial cell in the joint heatmap stores the spatial micro-motion power at each spatial angle. The two-stage DOA estimation involves first performing DOA estimation on multiple spatial angles with zero pitch to obtain candidate spatial angles, and then performing DOA estimation on candidate spatial angles with non-zero pitch to obtain effective spatial angles. Constant false alarm rate (CFAR) detection is performed on the joint heatmap, and spatial cells in the joint heatmap with spatial micro-motion power greater than the CFAR detection threshold are considered effective spatial cells. The window geometry of the measured unit in the CFAR detection is rhomboid. Connected component clustering is performed on the effective spatial cells of the joint heatmap, and multiple groups of spatial windows are divided in combination with human physiological constraints. Each group of spatial windows includes a lip spatial window and a throat spatial window. The phase difference of the signal in each group of windows is calculated frame by frame to obtain the first micro-Doppler signal reflecting the opening and closing of the target's lips, and the second micro-Doppler signal reflecting the vibration of the target's vocal cords. Short-time Fourier transforms were performed on the first and second micro-Doppler signals after time synchronization to obtain the first and second micro-Doppler time-frequency diagrams with time on the horizontal axis and Doppler frequency on the vertical axis.

[0007] Preferably, the step of performing two-stage DOA estimation on the preprocessed multi-echo signals based on the target range bin to obtain multiple target azimuth angles and constructing a joint heatmap including range and azimuth dimensions includes: The M-order covariance matrix of the target range cell is calculated based on the preprocessed M echo signals with different chirps under a single target range cell, and the inverse matrix of the covariance matrix of the target range cell is solved by the improved Gauss-Jordan iterative method based on the CORDIC algorithm; where M is the number of receiving antennas of a single millimeter-wave radar. The steering vectors of each spatial angle of a single target range cell are obtained, and the two-level DOA estimation is performed by combining the inverse of the covariance matrix of the target range cell to obtain the Capon beam cepstral values ​​of the target range cell at different spatial angles. The reciprocal of the Capon beam cepstral value at each spatial angle is taken as the spatial micro-motion power at each spatial angle under the target range cell, and the spatial micro-motion power corresponding to all spatial angles under the target range cell is used to form the power row vector corresponding to the target range cell. The power row vectors corresponding to all target distance bins are stitched together along the distance dimension to form a joint heatmap.

[0008] Preferably, the step of solving for the inverse matrix of the covariance matrix of the target distance bin using the improved Gauss-Jordan iterative method based on the CORDIC algorithm includes: The M-order covariance matrix of the target range cell is calculated based on the preprocessed M echo signals of different chirps under the target range cell; where M is the number of receiving antennas of a single millimeter-wave radar. The first row and first column of the M-order covariance matrix of the target distance to the bin are used as the pivot element, and the rotation angle of the pivot element is calculated using the vector pattern of the CORDIC algorithm. The identity matrix is ​​used as the current inverse matrix of the covariance matrix. The covariance matrix and the current inverse matrix are grouped by row, with the number of groups equal to the number of rows in the matrix. Each group includes one row of complex data of the covariance matrix and one row of complex data of the current inverse matrix. Using the opposite of the rotation angle of the principal component and each set of data as input, the rotation mode of the CORDIC algorithm is called to rotate the real and imaginary parts of each set of data synchronously, and the covariance matrix and the current inverse matrix are updated with the row dimension coordinates after rotation. Based on the principal component and its rotation angle, calculate the rotation angle of each element in the first column of the covariance matrix. Then, call the rotation mode of the CORDIC algorithm to rotate each covariance matrix and the current inverse matrix in the column dimension. Update each covariance matrix and the current inverse matrix with the rotated column dimension coordinates. Circularly shift each column of the covariance matrix and the current inverse matrix to the left by one column, and circularly shift each row of the covariance matrix and the current inverse matrix to the up by one row; The above steps are executed M times iteratively. After the iteration is completed, the rotation angles of the diagonal elements in the covariance matrix are obtained. The rotation angles are inverted and the rotation mode of the CORDIC algorithm is called to calculate the compensated rotation coordinates of the current inverse matrix to replace the original coordinates in the current inverse matrix. The final inverse matrix is ​​then output.

[0009] Preferably, the step of obtaining the steering vectors of each spatial angle of a single target range cell, and performing two-stage DOA estimation by combining the inverse of the covariance matrix of the target range cell to obtain the Capon beam cepstral values ​​at different spatial angles under the target range cell includes: For a single target distance chamber, multiple preset spatial angle guidance vectors are constructed, wherein the pitch angle of the spatial angle is zero; The inverse matrix of the covariance matrix of the target distance to the warehouse is decomposed into M columns of data streams. The rotation mode of the CORDIC algorithm is called to rotate the M columns of data streams and the guide vector of the current spatial angle. The results of the rotation operation are accumulated and summed according to the real part and the imaginary part of the data to obtain the first summation result in complex form. The result of the conjugate operation on the guide vector of the current spatial angle is used as the conjugate guide vector. Taking the phase of the conjugate guide vector of the current spatial angle as the rotation reference, CORDIC rotation and real and imaginary part accumulation are performed again on the first summation result to obtain the second summation result in complex form. The modulus of the second summation result is used as the Capon beam cepstral value of the current spatial angle, and the set of Capon beam cepstral values ​​of the target range cell is formed by traversing all spatial angles. The data in the Capon beam cepstral value set that are smaller than the cepstral values ​​of the left and right adjacent Capon beams are taken as local minima, and the spatial angle corresponding to the local minima is obtained. The effective spatial angle is defined as the angle after assigning a non-zero pitch angle to the spatial angle corresponding to the local minimum. The Capon beam cepstral value of the effective spatial angle is then recalculated, thus obtaining the Capon beam cepstral values ​​for different spatial angles.

[0010] Preferably, the constant false alarm rate (CFAR) detection of the joint heatmap includes: Traverse all pixels of the joint heatmap in the order of distance dimension first, then orientation dimension, and take each pixel obtained as the measured unit. The rhombus shape is used as the window geometry of the unit under test, and a reference window area and a protective window area with window geometry are formed relative to the unit under test. The constant false alarm detection threshold of the unit under test is calculated based on the spatial micro-motion power within the reference window area and the protection window area. If the spatial micro-motion power of the tested unit is greater than the constant false alarm detection threshold, the constant false alarm detection result of the tested unit is recorded as the existence of a target and marked as a valid spatial cell; otherwise, the constant false alarm detection result of the tested unit is recorded as the absence of a target.

[0011] Preferably, the step of using a rhombus shape as the window geometry of the unit under test, and forming a reference window area and a protective window area with the window geometry relative to the unit under test, includes: The location index of a single unit under test in the joint heatmap is obtained. The boundary of the joint heatmap is expanded by combining the given protection window width and reference window width to prevent the window from going out of bounds, resulting in an expanded range-azimuth joint heatmap. The location index includes the indexes of the range dimension and the azimuth dimension, and the reference window width is greater than the protection window width. Using the position index of the measured unit within the expanded joint heat map as the center point, and combining the protection window width, cells that meet the Manhattan distance determination condition of the diamond shape are selected from the expanded joint heat map to form the protection window area; Using the position index of the measured unit within the expanded joint heatmap as the center point, and combining it with the width of the reference window, a set of candidate cells corresponding to the outer rhombus is obtained. Cells that have been included in the protection window area are removed from this set of candidate cells of the outer rhombus, and the remaining cells together constitute the reference window area.

[0012] Preferably, the step of calculating the constant false alarm detection threshold of the unit under test based on the spatial micro-motion power within the reference window area and the protection window area includes: For each unit under test, the number of samples in the reference cell within the reference window area of ​​that unit under test is taken as the number of reference samples for that unit under test. The micro-motion power values ​​corresponding to each reference cell are obtained and sorted in order according to the power magnitude. The micro-motion power value at the preset position after sorting is used as the constant false alarm detection threshold of the tested unit.

[0013] Preferably, the step of performing connected component clustering on the effective spatial cells of the joint heatmap and dividing multiple groups of spatial windows in conjunction with human physiological constraints includes: Each spatially adjacent valid spatial cell is aggregated into multiple spatially connected clusters, which serve as candidate head spatial clusters. The centroid of each candidate head spatial cluster is obtained. The candidate center positions of the lips and throat are obtained by tilting the centroid upward and downward, respectively, based on the angular offset range of the lips and throat relative to the centroid, thus obtaining multiple effective candidate head spatial clusters. Among them, the physical distance between the candidate center positions of the lips and throat of the effective candidate head middle cluster is within the physiologically permissible range. For a single valid candidate head spatial cluster, non-overlapping lip spatial windows and throat spatial windows are defined with the lip candidate center position and throat candidate center position of the corresponding candidate head middle cluster as the center; wherein, both the lip spatial window and the throat spatial window are contained within the valid candidate head spatial cluster.

[0014] Preferably, the step of calculating the phase difference of the signals in each group of windows frame by frame to obtain a first micro-Doppler signal reflecting the opening and closing motion of the target's lips and a second micro-Doppler signal reflecting the vibration of the target's vocal cords includes: After capturing the frame-by-frame echo signals of each spatial cell within the lip and throat spatial windows of the same valid candidate head spatial cluster, coherent accumulation is performed on the frame-by-frame echo signals within the lip and throat spatial windows to obtain the composite echo signal of the lip and throat spatial windows of the valid candidate head spatial cluster. After extracting the phases of the two types of composite echo signals, phase unwrapping is performed to obtain two unwrapped phase sequences to eliminate phase folding jumps. These are denoted as the unwrapped phase sequence of the lip space window and the unwrapped phase sequence of the throat space window of the effective candidate head space cluster. The two unwound phase sequences are subjected to time-domain difference to obtain the time-domain difference signal of the lip spatial window and the time-domain difference signal of the throat spatial window of the effective candidate head spatial cluster. The time-domain difference signal of the lip spatial window is filtered by a bandpass filter to obtain the first micro-Doppler signal, and the time-domain difference signal of the throat spatial window is filtered by a high-pass filter to obtain the second micro-Doppler signal.

[0015] Secondly, this application provides a voice content recognition and identity recognition system based on a single millimeter-wave radar, including: The target distance chamber acquisition unit is used to perform distance-dimensional Fourier transform on the preprocessed multi-echo signals to obtain a two-dimensional heat map including distance and time dimensions, and to filter out multiple target distance chambers that match human micro-movements based on the phase of each distance chamber in the two-dimensional heat map. The joint heatmap construction unit is used to construct a joint heatmap including distance and azimuth dimensions by performing two-stage DOA estimation on the preprocessed multi-echo signals according to the Capon algorithm based on the target distance bin and calculating the spatial micro-motion power at each spatial angle. Each spatial cell in the joint heatmap stores the spatial micro-motion power at each spatial angle. The two-stage DOA estimation means that DOA estimation is first performed on multiple spatial angles with zero pitch angle to obtain candidate spatial angles, and then DOA estimation is performed on candidate spatial angles with non-zero pitch angle to obtain effective spatial angles. The constant false alarm detection unit is used to perform constant false alarm detection on the joint heat map. The spatial cells in the joint heat map whose spatial micro-motion power is greater than the constant false alarm detection threshold are taken as the valid spatial cells of the joint heat map. The window geometry of the tested cell in the constant false alarm detection is a rhombus shape. The micro-Doppler signal separation unit is used to perform connected component clustering on the effective spatial cells of the joint heatmap and divide multiple groups of spatial windows in combination with human physiological constraints. Each group of spatial windows includes a lip spatial window and a larynx spatial window. The phase difference of the signals in each group of windows is calculated frame by frame to obtain the first micro-Doppler signal reflecting the opening and closing of the target's lips and the second micro-Doppler signal reflecting the vibration of the target's vocal cords. The Doppler time-frequency diagram generation unit is used to perform short-time Fourier transform on the time-synchronized first micro-Doppler signal and the second micro-Doppler signal respectively to obtain the first micro-Doppler time-frequency diagram and the second micro-Doppler time-frequency diagram with time as the horizontal axis and Doppler frequency as the vertical axis.

[0016] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. To address the issue of highly coherent echo signals caused by multiple objects reflecting off the body and near-micro-movements of the lips and throat in indoor environments, this application, unlike traditional direct matrix inversion methods, employs an improved Gauss-Jordan iterative method based on the CORDIC algorithm to solve for the inverse of the covariance matrix of each target distance cell. This effectively avoids the defects of ill-conditioned, rank-deficient, and numerically divergent covariance matrices in highly coherent echo signals. Consequently, it improves the accuracy of DOA estimation performed using the Capon algorithm combined with the inverse of the covariance matrix and the steering vector, significantly suppresses Capon spatial spectrum spurious peaks and angle jumps, and enhances the accuracy of multi-target orientation estimation. This provides stable data for distinguishing different speakers and accurately locating areas of weak human movement in multi-person scenarios.

[0017] 2. Traditional Capon beam DOA estimation uses zero-pitch angle scanning, which cannot identify the physiological height difference between the lips and throat in the pitch dimension, easily causing spatial aliasing of two adjacent weak and slightly moving signals. This application first uses the zero-pitch angle spatial angle for the first round of screening to obtain the set of Capon beam cepstral values ​​of a single target range cell, thereby screening out the local minimum value and the corresponding spatial angle. Then, after assigning non-zero pitch angles to these angles, the effective spatial angles are formed, and the Capon beam cepstral values ​​of the effective spatial angles are calculated, thereby achieving high-precision spatial separation of two adjacent weak and slightly moving targets, the lips and throat, within the same head region.

[0018] 3. This application employs a rhomboid shape for constant false alarm detection (CFAR) of the joint heatmap. Compared to the traditional rectangular window shape, this shape better reflects the energy diffusion characteristics of human body micro-motion, effectively suppressing clutter false alarms. Simultaneously, through spatial connectivity clustering and combining the physiological distance constraint between the throat and lips, it filters out false target clusters generated by multipath propagation and clutter, accurately identifying valid candidate head spatial clusters. Furthermore, within a single valid candidate head spatial cluster, it adaptively segments non-overlapping lip and throat spatial windows that are completely contained within the cluster.

[0019] 4. In this application, the echo signals within the dual windows are coherently accumulated to improve the signal-to-noise ratio. Phase unwrapping is used to eliminate millimeter-wave phase folding jumps, and then time-domain differential analysis is used to extract the true micro-motion phase change information. Addressing the differences between the low-frequency opening and closing movements of the lips and the high-frequency vocal cord vibrations of the larynx, bandpass filters are used to filter the time-domain differential signals of the lip spatial window, and high-pass filters are used to filter the time-domain differential signals of the larynx spatial window. This specifically filters out head rigid body swaying, low-frequency drift, and high-frequency noise, ultimately obtaining two physically independent, non-interfering micro-Doppler signals from the lips and larynx. This solves the problem that existing technologies cannot distinguish the movement modes of the vocal organs and the aliasing of speech features. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0021] Figure 1 A flowchart of the speech content recognition and identity recognition method based on single millimeter-wave radar provided by the present invention; Figure 2 A flowchart of the speech content recognition and identity recognition system based on single millimeter-wave radar provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0023] First, it's important to note that before transmitting the frequency-modulated continuous wave (FM) signal using the millimeter-wave radar, hardware configuration is required. A 77GHz millimeter-wave radar sensor with two transmit and four receive antennas (TI AWR1642BOOS+DCA1000) is used, employing a MIMO architecture with two transmit antennas and four receive antennas. This architecture supports various applications, including gesture recognition, micro-vibration detection, human presence sensing, and blind spot monitoring. The operating frequency band is 77-81GHz, with a bandwidth of 3.9GHz, a frame rate of 100fps, and each frame contains 128 linear frequency modulated pulses. The radar is placed horizontally (initial angle 0°) at a distance of approximately 0.5-0.8m from the mouth area of ​​a seated subject.

[0024] Furthermore, after the radar transmits the frequency-modulated continuous wave signal, it sends back a long string of continuous I and Q sampled digital signals. The aforementioned preprocessing refers to first pairing I and Q signals one-to-one according to hardware timing specifications to form complex signals, and then rearranging the one-dimensional complex sequence into a four-dimensional matrix according to Fortran's column-first memory rules. This describes the collected multi-dimensional signal data, corresponding to the total number of frames, chirp, receiving antenna, and sampling points after analog-to-digital conversion. This allows each position in the array to correspond to the real physical sampling scenario, providing structurally correct baseband data for subsequent multi-dimensional Fourier transforms of distance, velocity, and angle.

[0025] This application provides a method for speech content recognition and identity recognition based on single millimeter-wave radar, such as... Figure 1 As shown, the method includes the following steps: Step 1, Obtain the target distance bin: Perform a distance-dimensional Fourier transform on the preprocessed multi-echo signals along the fast time axis to obtain a two-dimensional heatmap including both distance and time dimensions, denoted as the time-distance heatmap. Based on the phase of each distance bin in the time-distance heatmap, select the target distance bin that matches the micro-movement of the human body, i.e., the radial distance range of the target, which greatly reduces the amount of data for subsequent processing.

[0026] In this system, radar echoes carry range information on the fast time axis. Fast-time Fourier transform (FTFT) can focus the signal energy, achieving range resolution. Furthermore, radar data is sampled in fast time sequence; performing a FTFT on the fast time axis first enables "streaming processing," meeting real-time processing requirements. The resulting time-range heatmap uses time as the horizontal axis and the distance between the target and the radar as the vertical axis. Pixel amplitude represents the radar echo energy at that moment and distance. Each data point in the graph is a complex number; the real part represents the signal amplitude, and the imaginary part represents phase information, meaning it simultaneously carries both amplitude and phase information.

[0027] Furthermore, the criteria for selecting target distance chambers are based on the fact that human speech and head movements cause minute radial displacements, and the phase of complex echoes fluctuates continuously over slow time, while the phase of static objects (walls, furniture) hardly changes over time. Specifically, the variance of the phase of each distance chamber along the slow time dimension is first calculated, and then the calculated average variance is compared with a preset variance threshold. Distance chambers with a variance greater than the preset threshold are designated as target distance chambers, while the remaining distance chambers are judged as static background distance chambers and can be eliminated. Variance is used to characterize the intensity of micro-movements. The phase of micro-movements of the human mouth / throat fluctuates continuously with small amplitudes, and the variance is significantly higher than that of noise, while the phase of clutter or noise is random and irregular, with extremely small variance. The phase of reflections from static background objects remains almost unchanged, with a variance close to 0.

[0028] It is important to understand that the above method of screening target distance bins is only a preliminary screening in terms of distance dimension. This is because indoor multipath reflections can cause additional phase fluctuations in non-target distance bins, or in other words, random phase noise under low signal-to-noise ratio can lead to an increase in average variance. All of these factors result in the target distance bins being inaccurate and require further precise positioning by combining them with the angle dimension.

[0029] Step 2, constructing a joint heatmap: Based on the target range cell, perform two-level DOA (Direction of Arriva) estimation on the preprocessed echo signal according to the Capon algorithm to obtain the target azimuth angle, obtain the spatial micro-motion power of each target azimuth angle under each target range cell, and generate a joint heatmap including the range dimension and the azimuth dimension, denoted as the range-azimuth joint heatmap.

[0030] If there is no pitch angle, the image is a range-azimuth joint heatmap with distance on the horizontal axis and horizontal azimuth on the vertical axis. The pixel values ​​of this image represent the spatial micro-motion power amplitude of the spatial distance unit formed by distance and horizontal azimuth. The horizontal azimuth can be directly estimated using Angle FFT, which is suitable for time-series tracking scenarios with no pitch angle and a single speaker. Conversely, the range-azimuth joint heatmap represents the spatial micro-motion power amplitude of the spatial distance unit formed by distance, horizontal azimuth, and pitch angle. It can achieve complete separation of multiple people and background objects at different angles at the same distance, avoiding signal aliasing in scenarios with multiple people at the same distance where only the range-dimensional Fourier transform exists. In this case, the Capon algorithm is required to achieve DOA estimation.

[0031] The Capon algorithm is a beamforming method based on the linearly constrained minimum variance criterion, widely used in DOA estimation in radar signal processing. This algorithm achieves constrained beamforming under conditions considering spatial noise and interference signals. Specifically, it uses a portion of the degrees of freedom to set the gain in the desired direction to a constant, while using the remaining degrees of freedom to create nulls in the interference direction. The optimization objective is to minimize the output power in the noise and interference signal directions, while keeping the output power in the desired signal direction constant. Its core foundation lies in the signal's covariance matrix and its inverse. The covariance matrix provides spatial statistical information about the signal, while the inverse matrix provides the numerical solution for optimal weights; together, they constitute the mathematical basis of Capon beamforming. The inversion operation enables the algorithm to adaptively suppress interference and noise, minimizing the total array output power while maintaining distortion-free signal in the desired direction.

[0032] Step 3, Constant False Alarm Rate (CFAR) Detection: Perform CFAR detection on the joint heatmap. Spatial cells in the joint heatmap whose spatial micro-motion power is greater than the CFAR detection threshold are taken as valid spatial cells of the joint heatmap. The window geometry of the tested cell in the CFAR detection is a rhombus shape.

[0033] Step 4, Separate the micro-Doppler signals: Perform connected component clustering on the effective spatial cells of the joint heatmap, and divide multiple groups of spatial windows in combination with human physiological constraints. Each group of spatial windows includes a lip spatial window and a larynx spatial window. Calculate the phase difference of the signals in each group of windows frame by frame to obtain the first micro-Doppler signal reflecting the opening and closing of the target's lips, and the second micro-Doppler signal reflecting the vibration of the target's vocal cords.

[0034] One window is designated as the lip space window, which is closer in distance and positioned at a narrow angle. This window retains only the echo signal from the lip region, filtering out irrelevant micro-motion signals from the torso, forehead, and other areas. The other window, the throat space window, is farther away than the lip space window and is positioned at the same angle below the throat. Data within this window is independently extracted to avoid spatial crosstalk with the lip region. The first micro-Doppler signal includes the lip micro-Doppler signal, the second micro-Doppler signal, and the throat micro-Doppler signal. It is understood that the above window division is based on the geometric spatial constraints of the target head, and the two types of signals are output after time-sequential pairing.

[0035] Step 5, Generate Doppler time-frequency diagrams: Perform short-time Fourier transforms on the first and second micro-Doppler signals after time synchronization to obtain the first and second micro-Doppler time-frequency diagrams with time as the horizontal axis and Doppler frequency as the vertical axis.

[0036] The first micro-Doppler time-frequency map, namely the lip micro-Doppler map, represents macroscopic articulation actions such as lip opening and closing, and movement. The second micro-Doppler time-frequency map, namely the laryngeal micro-Doppler map, is used to represent fundamental frequency changes, voiced intensity, and vocal stability.

[0037] By aligning the timestamps of the two independently extracted micro-Doppler signals, the time axis and sampling frame rate of the lip and throat dual-modal signals are ensured to be completely consistent, avoiding timing misalignment issues. Independent micro-Doppler time-frequency maps are generated based on the synchronized dual signals, providing accurate, feature-complete, and highly discriminative dual-modal time-frequency features for subsequent speech content recognition and speaker identification. This significantly improves the accuracy and stability of speech perception and identity recognition in complex indoor multi-person scenarios. In other words, the method in this application can simultaneously acquire lip movement micro-Doppler and vocal cord mechanical vibration dual-modal features based on a single millimeter-wave radar. Then, speech activity detection is performed based on the joint dual-modal features to distinguish between phonological and non-phonological segments. Speech content is extracted from the phonological segments, and features related to personal habits are extracted from the non-phonological segments. Identity is then modeled to achieve speaker authentication.

[0038] In a specific embodiment of the present invention, before performing the range-dimensional Fourier transform in step 1, preprocessing is required, such as mixing the received and transmitted signals to obtain the echo signal, and two-line cancellation. Two-line cancellation involves using the range pulse compression results corresponding to two adjacent transmitted waveforms for differential calculation. The difference between the two range pulse compression results eliminates the signal components corresponding to the static scene. After differential calculation of the moving target, the signal components of the moving lip target are retained, thus completing the cancellation of the static scene and eliminating it.

[0039] In a specific embodiment of the present invention, it should be understood that in indoor scenarios with multiple people, multipath clutter, and coherent micro-motion, the covariance matrix is ​​close to singular, the matrix inversion value diverges, the spatial spectrum exhibits numerous spurious peaks, and the angle estimation is severely offset, making it impossible to accurately distinguish the small spatial regions of the lips / throat. To address these issues, step 2, the generation of the joint heatmap, includes the following sub-steps: Step 21: For each target range cell, obtain the preprocessed echo signals of all receiving antennas at different chirps under a single target range cell to obtain the M-order covariance matrix of the target range cell, and use an improved iterative method to calculate the inverse matrix of the covariance matrix of the target range cell to solve the problem of matrix singularity and inversion divergence caused by coherent micro-motion.

[0040] In calculating the covariance matrix, the complex echo signals from each receiving antenna at the target range cell, after undergoing range-dimensional Fourier transforms at different chirps, are first obtained. For each chirp, a data vector is constructed, and the product of each vector and its corresponding transpose and conjugate is calculated. Then, the summation and averaging of these results for all vectors yield the covariance matrix corresponding to the target range cell. When calculating the covariance matrix, only the values ​​of the upper triangular matrix are calculated in parallel; the values ​​of the lower triangular matrix are mapped according to the principle of conjugate symmetry.

[0041] For example, the original covariance matrix , represented as In the formula, N represents the number of chirps in each frame. The nth vector representing the data vectors formed by the data from each received channel. represent The transpose and conjugate of .

[0042] Full-rank covariance matrix , represented as In the formula, is the weighting coefficient, with a value ranging from 0.05 to 0.2, and M is the number of receiving antennas. It is an M-order identity matrix. for The traces.

[0043] During implementation, the inverse of the covariance matrix of the target distance warehouse is calculated, including: Step 21-1: For the M-order covariance matrix of the target distance bin, take the elements in the first row and first column of the matrix as the pivot elements, and calculate the rotation angle, i.e. the rotation phase, of each pivot element using the vector pattern of the CORDIC (Coordinate Rotation Digital Computer) algorithm.

[0044] In the above scheme, fixing the pivot avoids the need to select the pivot for each shift; instead, it only requires elementary row and column shifts of the matrix after each iteration. In implementation, if parallel processing is considered, the M-order covariance matrix can be further stored in an M×M parallel register set, with the identity matrix set as the initial value of the inverse matrix.

[0045] Step 21-2: Output the covariance matrix and current inverse matrix of the target distance bin by row grouping. The number of groups is equal to the number of rows of the matrix. Each group includes a complex number of the covariance matrix and a complex number of the current inverse matrix. At the same time, using the opposite of the rotation angle of each principal element and the output data of each group as input, call the rotation mode of the CORDIC algorithm to rotate the real and imaginary parts of each group of output data synchronously. Replace the initial coordinates with the rotated coordinates in the row dimension, that is, update each covariance matrix and the current inverse matrix.

[0046] During implementation, the complex numbers of the corresponding row of the covariance matrix and the corresponding row of the current inverse matrix are taken as the complex numbers to be rotated in each group. The opposite of the pivot rotation angle is taken as the rotation phase. The rotation operation is completed by calling the rotation mode of the CORDIC algorithm. Then, the real and imaginary coordinates obtained by the row dimension rotation are used to replace the original complex data in the corresponding group to complete the update of the covariance matrix and the current inverse matrix.

[0047] Step 21-3: Based on the principal elements and their rotation angles in each covariance matrix, derive the rotation angles of each element in the first column of each covariance matrix. Rotate each covariance matrix and the current inverse matrix using the rotation mode of the CORDIC algorithm to obtain the rotated coordinates in the column dimension. Update each covariance matrix and the current inverse matrix with the rotated coordinates.

[0048] During implementation, the complex numbers of each non-principal element in the first column of the covariance matrix and the corresponding complex numbers in the current inverse matrix are taken as the complex numbers to be rotated. The compensation rotation angle of each element in the corresponding column is taken as the rotation phase. The rotation operation is completed by calling the rotation mode of the CORDIC algorithm. Then, the real and imaginary coordinates of the column-dimensional rotation output are used to update the complex numbers at the corresponding positions of the covariance matrix and the current inverse matrix.

[0049] For example, after the column containing the pivot is converted to a real number, the elements of the corresponding column are converted to... Where k is an integer from 1 to M, through We can see that the rotation angle of the element in the k-th row of the first column is... for Understandably, The element in the k-th row and first column. The element in the first row and first column is the pivot.

[0050] Understandably, during the real part processing in the above steps, the data stream is divided into M groups. Each group follows the same method as in step 21-2, i.e., first inputting the covariance matrix and then its inverse, and synchronously importing the data and the rotation angles of each element in the first column calculated in step 21-3 into the rotation mode. The imaginary part processing is the same as the real part processing. After the calculation is completed, the real and imaginary part data are arranged and combined appropriately to finally calculate the covariance matrix and inverse matrix after the first iteration.

[0051] Step 21-4: Circularly shift each column of each covariance matrix and the current inverse matrix to the left by one column, and then circularly shift each row of each covariance matrix and the current inverse matrix of each covariance matrix to the top by one row. Iterate through the above steps M times. After the iteration is complete, obtain the rotation angle of the diagonal elements in the covariance matrix. After inverting the rotation angle, call the rotation mode of the CORDIC algorithm to calculate the compensated rotation coordinates of the current inverse matrix to replace the original coordinates in the current inverse matrix, so as to complete the rotation angle correction of the diagonal elements and output the final inverse matrix, thereby alleviating the problem of numerical divergence in inverse calculation caused by ill-conditioned singularity of the matrix.

[0052] During implementation, the diagonal elements of the covariance matrix and the complex numbers corresponding to the diagonal positions of the current inverse matrix are taken sequentially as the complex numbers to be rotated. The opposite of the rotation angle of the corresponding diagonal elements is taken as the rotation phase, and the rotation mode of the CORDIC algorithm is called to complete the diagonal element correction rotation operation.

[0053] Understandably, after the cyclic shift is complete, the elements in the first column of the covariance matrix become the elements in the next column from the previous iteration, and the first element in the first column also becomes the main diagonal element of the next column from the previous iteration; while the elements in the first column from the previous iteration become the last column at this point, and the pivot element becomes the last element of the last column, while all other elements in the same column as the pivot are 0. The second iteration then begins, and after the Mth iteration is completed, the iteration loop is exited.

[0054] It's important to understand that this method relies entirely on the vector and rotation modes of the CORDIC algorithm for complex rotation elimination. Each round of row rotation, column rotation, and pivoting adds an extra rotation phase to the complex elements of the matrix. After M iterations, although the covariance matrix has been reduced to near the identity matrix, residual phase shifts and amplitude scaling biases from the iterations remain on the diagonal elements. Without correction, the output inverse matrix will have a fixed complex phase bias. Therefore, after M iterations, the complex elements at each diagonal position of the covariance matrix are read. The CORDIC algorithm's vector mode is used to obtain the amplitude and the accumulated rotation angle for each diagonal element. The rotation angle is inverted to obtain the compensation phase. The CORDIC algorithm's rotation mode is then called, and this compensation phase is used to perform a uniform complex rotation on all complex elements of the intermediate matrix in the current inverse matrix iteration, completing the correction. By using the improved Gauss-Jordan iteration based on CORDIC and the diagonal element correction processing, the problem of numerical divergence in matrix inversion caused by ill-conditioned singularity of the covariance matrix in indoor coherent micro-motion scenarios is alleviated, significantly reducing computational overhead and improving processing timeliness.

[0055] Step 22: Obtain the steering vectors of a single target range cell at different spatial angles, and combine them with the inverse of the covariance matrix of the target range cell to obtain the Capon beam cepstral values ​​at different spatial angles under the target range cell; wherein, the steering vector is used to characterize the phase difference of the echo signal arriving at each receiving antenna at different angles.

[0056] During implementation, the Capon beam cepstral values ​​of the target at different spatial angles below the binoculars are obtained, including the following sub-steps: Step 22-1: For a single target distance chamber, construct multiple preset spatial angle guidance vectors, where the pitch angle of the spatial angle is zero.

[0057] Understandably, conventional spatial angles are set by first iterating through the pitch angles and then the horizontal angles. However, the vertical movement of a human face is very small, and the interference from multiple speakers is also in the horizontal direction. Scanning the pitch angles first would result in a large amount of invalid calculations and would not be able to lock the speaker's horizontal range, with clutter peaks covering the subtle lip movements. Therefore, in this embodiment, the horizontal angle is first used to determine the horizontal range where the face exists, and then the pitch angle is assigned to redefine the spatial angles, thereby filtering out invalid angle calculations and improving the signal-to-noise ratio of subtle vocal movements. Moreover, signals from multiple human bodies at the same distance will not overlap in the heatmap, and independent spatial windows for the lips and throat can be accurately segmented subsequently.

[0058] Step 22-2: Call the rotation mode of the CORDIC algorithm to perform rotation operations on the M sets of data streams composed of the column elements of the inverse matrix of the covariance matrix of the target distance to the warehouse and the guide vector of the current spatial angle. Then, sum the results of the rotation operation according to the real and imaginary parts of the data to obtain the first summation result in complex form.

[0059] The core function of the steering vector is to characterize the mapping relationship between the signal direction and the array response, which is the basis for DOA estimation. During the rotation operation, the complex number of each column of the inverse matrix is ​​taken as the complex number to be rotated, and the phase of the channel component corresponding to the current spatial angle steering vector is taken as the rotation phase. The rotation operation is completed by calling the rotation mode of the CORDIC algorithm.

[0060] Step 22-3: The result of the conjugate operation on the guide vector of the current spatial angle is used as the conjugate guide vector. Taking the phase of the conjugate guide vector of the current spatial angle as the rotation reference, CORDIC rotation and real / imaginary part accumulation are performed again on the first summation result to obtain the second summation result in complex form.

[0061] During implementation, the complex numbers of each channel of the first summation result are taken as the complex numbers to be rotated, and the phase of the corresponding channel component of the conjugate steering vector is taken as the rotation phase. The rotation mode of the algorithm is called to perform the rotation operation channel by channel.

[0062] Step 22-4: Use the modulus of the second summation result as the Capon beam cepstral value of the current spatial angle, and iterate through all spatial angles to form a set of Capon beam cepstral values ​​for the target range cell.

[0063] Step 22-5: Take the data in the Capon beam cepstral value set that are smaller than the cepstral values ​​of the left and right adjacent Capon beams as local minima, and obtain the spatial angle corresponding to the local minima.

[0064] Step 22-6: Assign a non-zero pitch angle to the spatial angle corresponding to the local minimum and record it as the effective spatial angle. Recalculate the Capon beam cepstral value of the effective spatial angle to obtain the final Capon beam cepstral values ​​for different spatial angles.

[0065] For example, using three data points as a sliding window for filtering, the data whose Capon beam cepstral value at the center of the sliding window is less than the Capon beam cepstral values ​​on the left and right sides is taken as the local minimum, corresponding to a maximum micro-motion power. For the boundary angles at both ends with only one neighboring point, the minimum value can be determined by comparing one side.

[0066] The above steps take into account the small difference in elevation angles. First, the Capon beam cepstral values ​​are calculated without considering the elevation angle. Then, local minima are selected and assigned non-zero elevation angles to the corresponding spatial angles, and the Capon beam cepstral values ​​are calculated again. Compared to directly considering the elevation and horizontal azimuth angles, this method significantly reduces the computational load of iterative calculations, improves computational efficiency, avoids a large number of floating-point complex multiplication operations, and ensures real-time processing capabilities for multi-person lip-throat recognition scenarios.

[0067] Step 23: Take the reciprocal of the Capon beam cepstral value of each spatial angle as the spatial micro-motion power of each spatial angle under the target range cell, and construct the power row vector corresponding to the target range cell for all spatial angles under the target range cell.

[0068] Step 24: The power row vectors corresponding to all target distance bins are stitched together along the distance dimension to form a joint heat map. Each cell of the joint heat map stores the spatial micro-motion power of the corresponding distance-azimuth position.

[0069] By following the steps described above, the hardware implementation defects of traditional Capon beamforming methods, which require a large number of division operations, are avoided. Parallel data processing significantly improves the efficiency of Capon beamforming, offering advantages such as high speed, high accuracy of calculation results, and low hardware resource consumption. It has high application value and prospects in radar signal processing and communication-related fields, ensuring the efficiency and accuracy of generating range-azimuth joint heatmaps.

[0070] This step involves independently constructing the covariance matrix and solving for its inverse for each range cell, decoupling the calculations between range cells to accommodate parallel processing of multiple human micro-motion target range cells. For each scanning angle, the Capon beam cepstral value is calculated independently, and the micro-motion power is obtained, preserving all azimuth dimension pixel information and stitching them together to form a joint heatmap. Specifically, when solving the inverse matrix, an improved Gauss-Jordan iterative method based on CORDIC is used to mitigate the divergence in inverse values ​​caused by ill-conditioned singularities of the covariance matrix in indoor multi-person, multipath, and coherent micro-motion scenarios, reducing spatial spectrum false peaks, improving angle estimation accuracy, and facilitating subsequent differentiation of the same human body's lip and throat micro-motion regions.

[0071] In a specific embodiment of the present invention, the constant false alarm rate (CFAR) detection in step 3 is used to filter out false spatial peaks caused by multipath reflections and environmental noise, significantly reducing false area locking caused by indoor multipath reflections, passing pedestrians, and random electromagnetic noise. Specifically, the CFAR detection is performed on the joint heatmap using a diamond shape, including: Step 31: Each pixel in the distance-orientation joint heatmap, which is traversed in the order of distance first and then orientation, is taken as the unit to be measured.

[0072] Step 32: Use the rhombus shape as the window geometry of the unit under test, and form a reference window area and a protective window area with the window geometry relative to the unit under test.

[0073] The protection window region is the set of all matrix elements within a diamond-shaped closed area centered on the unit under test (DUT). Elements within the protection window region do not participate in background clutter power statistics; they are used solely to isolate the DUT itself and nearby targets, preventing target signals from mixing into the reference sample and causing falsely high detection thresholds and missed detections. The reference window region is the set of all matrix elements within an annular diamond-shaped band centered on the DUT, between the outer diamond boundary and the inner protection window diamond boundary. All elements within the reference window region serve as reference samples for estimating background clutter power.

[0074] In implementation, the rhombus shape is used as the window geometry of the unit under test, and a reference window area and a protective window area with the window geometry are formed relative to the unit under test, including: Step 32-1: For a single unit under test, the rhombus shape is used as the window geometry of the unit under test. The position index of each unit under test in the distance-azimuth joint heatmap is obtained. The boundary of the distance-azimuth joint heatmap is expanded in combination with the given protection window width and reference window width to prevent the window from going out of bounds, resulting in an expanded distance-azimuth joint heatmap. The position index includes the index of the distance and azimuth dimensions, and the reference window width is greater than the protection window width.

[0075] The necessary condition for expansion is that the number of cells from the measured cell to any boundary of the distance-orientation joint heatmap is less than the width of the reference window. If this is the case, the reference window area corresponding to the measured cell cannot be fully accommodated on the original heatmap, resulting in window overflow. Expansion employs a mirror extension method, mirroring the existing cells on the inner side using the heatmap boundary as the axis of symmetry. The expansion amount is limited by the width of the reference window, expanding the boundary of the heatmap with a number of cells equal to the width of the reference window. This method preserves the power variation trend of the heatmap edge region, avoids detection failures caused by window incompleteness at the boundary, and ensures that micro-movement pixels of the lips and throat at the edge of the effective face distance range can also be effectively detected. After expansion, the measured cell in the expanded distance-orientation joint heatmap satisfies the following: the value obtained by subtracting the reference window width from the expanded position index is greater than or equal to 0, and the value obtained by adding the reference window width to the expanded position index is less than or equal to the maximum index of the corresponding dimension of the expanded heatmap. This ensures that the complete window geometry, centered on the measured unit and with the reference window width as the maximum offset, is fully contained within the expanded distance-angle joint heatmap.

[0076] Step 32-2: Using the position index of the measured unit in the expanded distance-azimuth combined heat map as the center point, and combined with the protection window width, select the cells that meet the Manhattan distance determination condition of the diamond shape from the expanded distance-azimuth combined heat map to form the protection window area.

[0077] Step 32-3: Using the position index of the measured unit in the expanded distance-orientation joint heatmap as the center point, and combining it with the width of the reference window, obtain the candidate cell set corresponding to the outer rhombus. Remove the cells that have been included in the protection window area from the candidate cell set of the outer rhombus, and the remaining cells together constitute the reference window area.

[0078] In implementation, each measured unit has a corresponding protected window area and a reference window area. If the measured unit falls on or near the edge of the heatmap, this type of window will exceed the boundary. If it is directly truncated, only the cells existing inside the image are retained as reference units, which will lead to a reduction in the samples on which constant false alarm rate (CFAR) detection depends, statistical distortion of noise, high false alarm rate at the edge, and significant missed detection. This application ensures that the protected window and reference window of the measured unit at all positions in the image have consistent window geometry by expanding the boundary, thus ensuring the accuracy of detection. It can be understood that the measured unit is located at the geometric center of the protected window area, and the reference window area and the protected window area together form a complete rhombus window, but the two have no intersection, that is, all elements of the protected window area do not belong to the reference window area.

[0079] Before implementation, the distance-azimuth joint heatmap can be cropped sequentially according to distance and azimuth. For example, first, the distance value corresponding to each column in the image is obtained, and it is determined whether the distance value is within the set target distance range. If not, the corresponding spatial micro-motion power is set to noise power. Then, the azimuth in the image is obtained; if the azimuth is not within the set azimuth range, the corresponding spatial micro-motion power is set to noise power. The above steps only replace pixel values, not delete them, to avoid disrupting the CFAR sliding window coordinate mapping.

[0080] Step 33: Calculate the constant false alarm rate (CFAR) detection threshold of the unit under test (TUD) based on the spatial micro-motion power of the reference window area and the protection window area. The calculation of the CFR threshold includes the following sub-steps: Step 33-1: For each unit under test, based on the position index of the cells contained in the reference window area and the protection window area of ​​the unit under test, filter out the reference cells that belong only to the reference window area, remove the cells that fall into the protection window area, and take the number of reference cells as the number of reference samples for the unit under test.

[0081] For example, the widths of the reference window area and the protection window area are defined as follows: , The total number of elements used to construct the tested cell, the guard window area, and the reference window area is then determined. for Total number of elements that construct the protected window area for The total number of elements in all construction reference window areas The difference between the two is, .

[0082] Step 33-2: Obtain the micro-motion power value corresponding to each reference cell and sort them in order according to the power magnitude. Use the micro-motion power value at the preset position after sorting as the background clutter power value of the reference window area of ​​the tested unit. This method has stronger anti-clutter interference capability in indoor multi-person and multi-target coexistence scenarios and is suitable for lip-throat sensing scenarios with multiple speakers.

[0083] In practice, the preset position is generally greater than half of the total number of samples to counteract the power of another nearby target mixed in within the reference window. This method can suppress the occlusion effect caused by nearby targets such as the lips and throat mixed in within the reference window, and is more suitable for the detection scenario of two weakly adjacent, slightly moving targets in this application.

[0084] Step 33-3: The product of the obtained background clutter power value and the threshold factor is used as the constant false alarm detection threshold of the unit under test.

[0085] For example, the result of adding the background clutter power value to the detection threshold constant can also be used as the constant false alarm detection threshold, with the factors and constants involved depending on empirical settings.

[0086] Step 34: If the spatial micro-motion power of the unit under test is greater than the constant false alarm detection threshold, then the constant false alarm detection result of the unit under test is recorded as the presence of a target; otherwise, the constant false alarm detection result of the unit under test is recorded as the absence of a target.

[0087] Understandably, the energy of micro-movement targets such as the human lips and throat diffuses in a diamond shape along the Manhattan distance on the joint heatmap. If a rectangular window is used as the window geometry of the measured unit, it will introduce a large number of low-correlation background pixels, resulting in high false alarm rates and high computational cost. This application replaces the traditional rectangular shape with a diamond shape. The diamond window selects reference units following the energy diffusion pattern of the target. The reference units have a higher correlation with the clutter environment of the measured unit, reducing the probability of false alarms. At the same time, it reduces the number of invalid reference units, reduces the computational cost, and ensures real-time processing. In addition, the inner diamond-shaped protection window area shields the target energy around the measured unit, preventing the leakage of the target's own micro-movement energy into the reference sample.

[0088] In a specific embodiment of the present invention, step 4 involves clustering the effective spatial cells of the joint heatmap into connected components and dividing multiple groups of spatial windows in conjunction with human physiological constraints, including the following sub-steps: First, spatial cells in the distance-azimuth joint heatmap whose spatial micro-motion power is greater than the constant false alarm detection threshold are taken as valid spatial cells. Then, the spatially adjacent valid spatial cells are aggregated into multiple spatial connectivity clusters using a 4-neighbor or 8-neighbor search method, which serve as candidate head spatial clusters.

[0089] Specifically, the candidate head spatial clusters should not include clusters with excessively small areas; that is, small clusters causing false alarms due to multipath propagation and noise should be filtered out. The filtering process can also incorporate the ratio of spatial micro-motion power to the constant false alarm detection threshold, using this ratio as the confidence level of target presence, and retaining clusters with high confidence levels.

[0090] Then, the centroid of each candidate head spatial cluster is obtained, and the candidate center positions of the lips and throat are obtained by combining the angular offset range of the lips and throat relative to the centroid, respectively, thus obtaining the effective candidate head spatial clusters.

[0091] Among them, the distance between the candidate center positions of the lips and the candidate center positions of the throat in the effective candidate head middle cluster is within the physiologically permissible range.

[0092] Finally, for the effective candidate head spatial clusters, non-overlapping lip spatial windows and throat spatial windows are defined with the lip candidate center position and throat candidate center position of the candidate head middle cluster as the center, respectively; wherein, the lip spatial window and throat spatial window are both contained in the effective candidate head spatial clusters.

[0093] Understandably, this windowing method targets the same heatmap, avoiding crosstalk between lip and throat echoes.

[0094] In a specific embodiment of the present invention, step 4 involves calculating the phase difference of the signals in each group of windows frame by frame to obtain a first micro-Doppler signal reflecting the opening and closing motion of the target lips, and a second micro-Doppler signal reflecting the vibration of the target vocal cords, including: After capturing the frame-by-frame echo signals of each spatial cell within the lip and throat spatial windows, coherent accumulation is performed on the frame-by-frame echo signals within the lip and throat spatial windows respectively to obtain the composite echo signals of the lip and throat spatial windows. Coherent accumulation is a fundamental concept in radar signal processing. Its core purpose is to improve the signal-to-noise ratio (SNR) of target detection by preserving and accumulating phase information, thereby increasing the detection range and accuracy of the radar. This technique is particularly suitable for radar, sonar, and other applications that require the detection of weak signals from noisy backgrounds, enhancing the signal strength of weak vibrations from the lips and throat, and improving phase extraction performance under low SNR conditions.

[0095] After extracting the phase of the composite echo signal from the lip space window and the throat space window, phase unwrapping is performed separately. Phase folding is eliminated by phase unwrapping, resulting in two unwrapped phase sequences, denoted as the unwrapped phase sequence of the lip space window and the unwrapped phase sequence of the throat space window. The two unwound phase sequences are subjected to time-domain difference analysis to convert the phase changes into micro-motion velocity information, thereby obtaining the time-domain difference signal of the lip spatial window and the time-domain difference signal of the throat spatial window. The result of the difference is proportional to the radial micro-velocity of the target, characterizing the vibration change of the lips.

[0096] The time-domain difference signal of the lip spatial window is filtered by a bandpass filter to obtain the first micro-Doppler signal, and the time-domain difference signal of the throat spatial window is filtered by a high-pass filter to obtain the second micro-Doppler signal.

[0097] The lower cutoff frequency of the bandpass filter removes ultra-low frequency disturbances caused by overall head shaking and body swaying, while the upper cutoff frequency preserves the effective frequency band corresponding to lip opening and closing and rapid lip vibration during speech, filtering out high-frequency noise from the radar system. The resulting signal is the lip micro-Doppler signal. A preset high-pass filter is then used to filter the time-domain differential signal from the second window; this filters out low-frequency disturbances introduced by head pitching, swallowing, and torso shaking, while retaining the high-frequency, minute vibration components corresponding to vocal cord phonation. The resulting signal is the laryngeal micro-Doppler signal.

[0098] It's important to understand that for lip micro-Doppler signals, directly using a low-pass filter, while filtering out ultra-low frequency interference from significant head shaking, will suppress the effective high-frequency components of rapid lip movements during speech, resulting in the loss of lip micro-movement characteristics. Using a band-pass filter, however, suppresses ultra-low frequency interference from significant head shaking while fully preserving the effective frequency band of lip vocalization. Furthermore, the two micro-Doppler signals are time-matched based on the timestamps of the radar hardware acquisition frames, correcting the naturally occurring micro-time misalignment of lip and throat movements.

[0099] Secondly, such as Figure 2 As shown, this application provides a voice content recognition and identity recognition system based on a single millimeter-wave radar, including: The target distance chamber acquisition unit is used to perform distance-dimensional Fourier transform on the preprocessed multi-echo signals to obtain a two-dimensional heat map including distance and time dimensions, and to filter out multiple target distance chambers that match human micro-movements based on the phase of each distance chamber in the two-dimensional heat map. The joint heatmap construction unit is used to perform two-level DOA estimation on the preprocessed multi-echo signals based on the target distance bin and according to the Capon algorithm to obtain multiple target azimuth angles, and construct a joint heatmap including distance and azimuth dimensions; wherein, each spatial cell in the joint heatmap stores the spatial micro-motion power of the spatial angle formed by the target distance bin and the target azimuth angle; The constant false alarm detection unit is used to perform constant false alarm detection on the joint heat map. The spatial cells in the joint heat map whose spatial micro-motion power is greater than the constant false alarm detection threshold are taken as the valid spatial cells of the joint heat map. The window geometry of the tested cell in the constant false alarm detection is a rhombus shape. The micro-Doppler signal separation unit is used to perform connected component clustering on the effective spatial cells of the joint heatmap and divide multiple groups of spatial windows in combination with human physiological constraints. Each group of spatial windows includes a lip spatial window and a larynx spatial window. The phase difference of the signals in each group of windows is calculated frame by frame to obtain the first micro-Doppler signal reflecting the opening and closing of the target's lips and the second micro-Doppler signal reflecting the vibration of the target's vocal cords. The Doppler time-frequency diagram generation unit is used to perform short-time Fourier transform on the time-synchronized first micro-Doppler signal and the second micro-Doppler signal respectively to obtain the first micro-Doppler time-frequency diagram and the second micro-Doppler time-frequency diagram with time as the horizontal axis and Doppler frequency as the vertical axis.

[0100] This application provides a computer-readable storage medium for storing a computer program used in the method described.

[0101] This application provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the method.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for speech content recognition and identity recognition based on single millimeter-wave radar, characterized in that, Includes the following steps: The preprocessed multi-echo signals are subjected to distance-dimensional Fourier transform to obtain a two-dimensional heat map including distance and time dimensions. Based on the phase of each distance cell in the two-dimensional heat map, multiple target distance cells that match human micro-movements are selected. Based on the target range cell, the preprocessed multi-echo signals are subjected to two-stage DOA estimation according to the Capon algorithm, and the spatial micro-motion power of each spatial angle is calculated to construct a joint heatmap including the range dimension and the azimuth dimension. In the joint heatmap, each spatial cell stores the spatial micro-motion power of each spatial angle. The two-stage DOA estimation means that DOA estimation is first performed on multiple spatial angles with zero pitch angle to obtain candidate spatial angles, and then DOA estimation is performed on candidate spatial angles with non-zero pitch angle to obtain effective spatial angles. Constant false alarm rate (CFAR) detection is performed on the joint heatmap. Spatial cells in the joint heatmap whose spatial micro-motion power is greater than the CFAR detection threshold are taken as valid spatial cells of the joint heatmap. The window geometry of the tested cell in the CFAR detection is a rhombus shape. Connected component clustering is performed on the effective spatial cells of the joint heatmap, and multiple groups of spatial windows are divided in combination with human physiological constraints. Each group of spatial windows includes a lip spatial window and a throat spatial window. The phase difference of the signal in each group of windows is calculated frame by frame to obtain the first micro-Doppler signal reflecting the opening and closing of the target's lips, and the second micro-Doppler signal reflecting the vibration of the target's vocal cords. Short-time Fourier transforms were performed on the first and second micro-Doppler signals after time synchronization to obtain the first and second micro-Doppler time-frequency diagrams with time on the horizontal axis and Doppler frequency on the vertical axis.

2. The speech content recognition and identity recognition method based on single millimeter-wave radar as described in claim 1, characterized in that, Based on the target range bin, the preprocessed multi-echo signals are subjected to two-stage DOA estimation using the Capon algorithm to obtain multiple target azimuth angles, and a joint heatmap including range and azimuth dimensions is constructed, including: The M-order covariance matrix of the target range cell is calculated based on the preprocessed M echo signals with different chirps under a single target range cell, and the inverse matrix of the covariance matrix of each target range cell is solved by the improved Gauss-Jordan iterative method based on the CORDIC algorithm; where M is the number of receiving antennas of a single millimeter-wave radar. The steering vectors of each spatial angle of a single target range cell are obtained, and the two-level DOA estimation is performed by combining the inverse of the covariance matrix of the target range cell to obtain the Capon beam cepstral values ​​of the target range cell at different spatial angles. The reciprocal of the Capon beam cepstral value at each spatial angle is taken as the spatial micro-motion power at each spatial angle under the target range cell, and the spatial micro-motion power corresponding to all spatial angles under the target range cell is used to form the power row vector corresponding to the target range cell. The power row vectors corresponding to all target distance bins are stitched together along the distance dimension to form a joint heatmap.

3. The speech content recognition and identity recognition method based on single millimeter-wave radar as described in claim 2, characterized in that, The method of solving the inverse matrix of the covariance matrix of the target distance bin using the improved Gauss-Jordan iterative method based on the CORDIC algorithm includes: The first row and first column of the M-order covariance matrix of the target distance to the bin are used as the pivot element, and the rotation angle of the pivot element is calculated using the vector pattern of the CORDIC algorithm. The identity matrix is ​​used as the current inverse matrix of the covariance matrix. The covariance matrix and the current inverse matrix are grouped by row, with the number of groups equal to the number of rows in the matrix. Each group includes one row of complex data of the covariance matrix and one row of complex data of the current inverse matrix. Using the opposite of the rotation angle of the principal component and each set of data as input, the rotation mode of the CORDIC algorithm is called to rotate the real and imaginary parts of each set of data synchronously, and the covariance matrix and the current inverse matrix are updated with the row dimension coordinates after rotation. Based on the principal component and its rotation angle, calculate the rotation angle of each element in the first column of the covariance matrix. Then, call the rotation mode of the CORDIC algorithm to rotate each covariance matrix and the current inverse matrix in the column dimension. Update each covariance matrix and the current inverse matrix with the rotated column dimension coordinates. Circularly shift each column of the covariance matrix and the current inverse matrix to the left by one column, and circularly shift each row of the covariance matrix and the current inverse matrix to the up by one row; The above steps are executed M times iteratively. After the iteration is completed, the rotation angles of the diagonal elements in the covariance matrix are obtained. The rotation angles are inverted and the rotation mode of the CORDIC algorithm is called to calculate the compensated rotation coordinates of the current inverse matrix to replace the original coordinates in the current inverse matrix. The final inverse matrix is ​​then output.

4. The speech content recognition and identity recognition method based on single millimeter-wave radar as described in claim 3, characterized in that, The process of obtaining the steering vectors at various spatial angles of a single target range cell, and then performing two-stage DOA estimation using the inverse of the covariance matrix of the target range cell to obtain the Capon beam cepstral values ​​at different spatial angles under the target range cell includes: For a single target distance chamber, multiple preset spatial angle guidance vectors are constructed, wherein the pitch angle of the spatial angle is zero; The inverse matrix of the covariance matrix of the target distance to the warehouse is decomposed into M columns of data streams. The rotation mode of the CORDIC algorithm is called to rotate the M columns of data streams and the guide vector of the current spatial angle. The results of the rotation operation are accumulated and summed according to the real part and the imaginary part of the data to obtain the first summation result in complex form. The result of the conjugate operation on the guide vector of the current spatial angle is used as the conjugate guide vector. Taking the phase of the conjugate guide vector of the current spatial angle as the rotation reference, CORDIC rotation and real and imaginary part accumulation are performed again on the first summation result to obtain the second summation result in complex form. The modulus of the second summation result is used as the Capon beam cepstral value of the current spatial angle, and the set of Capon beam cepstral values ​​of the target range cell is formed by traversing all spatial angles. The data in the Capon beam cepstral value set that are smaller than the cepstral values ​​of the left and right adjacent Capon beams are taken as local minimum values, and the spatial angles corresponding to the local minimum values ​​are taken as candidate spatial angles. The angle after assigning a non-zero pitch angle to the candidate spatial angle is recorded as the effective spatial angle. The Capon beam cepstral value of the effective spatial angle is recalculated, thus obtaining the Capon beam cepstral value for different spatial angles.

5. The speech content recognition and identity recognition method based on single millimeter-wave radar as described in claim 1, characterized in that, The constant false alarm rate (CFAR) detection of the joint heatmap includes: Traverse all pixels of the joint heatmap in the order of distance dimension first, then orientation dimension, and take each pixel obtained as the measured unit. The rhombus shape is used as the window geometry of the unit under test, and a reference window area and a protective window area with window geometry are formed relative to the unit under test. The constant false alarm detection threshold of the unit under test is calculated based on the spatial micro-motion power within the reference window area and the protection window area. If the spatial micro-motion power of the tested unit is greater than the constant false alarm detection threshold, the constant false alarm detection result of the tested unit is recorded as the existence of a target and marked as a valid spatial cell; otherwise, the constant false alarm detection result of the tested unit is recorded as the absence of a target.

6. The speech content recognition and identity recognition method based on single millimeter-wave radar as described in claim 5, characterized in that, The method of using a rhombus shape as the window geometry of the unit under test, and forming a reference window area and a protective window area with the window geometry relative to the unit under test, includes: The location index of a single unit under test in the joint heatmap is obtained. The boundary of the joint heatmap is expanded by combining the given protection window width and reference window width to prevent the window from going out of bounds, resulting in an expanded range-azimuth joint heatmap. The location index includes the indexes of the range dimension and the azimuth dimension, and the reference window width is greater than the protection window width. Using the position index of the measured unit within the expanded joint heat map as the center point, and combining the protection window width, cells that meet the Manhattan distance determination condition of the diamond shape are selected from the expanded joint heat map to form the protection window area; Using the position index of the measured unit within the expanded joint heatmap as the center point, and combining it with the width of the reference window, a set of candidate cells corresponding to the outer rhombus is obtained. Cells that have been included in the protection window area are removed from this set of candidate cells of the outer rhombus, and the remaining cells together constitute the reference window area.

7. The speech content recognition and identity recognition method based on single millimeter-wave radar as described in claim 6, characterized in that, The step of calculating the constant false alarm detection threshold of the unit under test based on the spatial micro-motion power within the reference window area and the protection window area includes: For each unit under test, the number of samples in the reference cell within the reference window area of ​​that unit under test is taken as the number of reference samples for that unit under test. The micro-motion power values ​​corresponding to each reference cell are obtained and sorted in order according to the power magnitude. The micro-motion power value at the preset position after sorting is used as the constant false alarm detection threshold of the tested unit.

8. The speech content recognition and identity recognition method based on single millimeter-wave radar as described in claim 6, characterized in that, The process of clustering the effective spatial cells of the joint heatmap into connected components and dividing them into multiple groups of spatial windows based on human physiological constraints includes: Each spatially adjacent valid spatial cell is aggregated into multiple spatially connected clusters, which serve as candidate head spatial clusters. The centroid of each candidate head spatial cluster is obtained. The candidate center positions of the lips and throat are obtained by tilting the centroid upward and downward, respectively, based on the angular offset range of the lips and throat relative to the centroid, thus obtaining multiple effective candidate head spatial clusters. Among them, the physical distance between the candidate center positions of the lips and throat of the effective candidate head middle cluster is within the physiologically permissible range. For a single valid candidate head spatial cluster, non-overlapping lip spatial windows and throat spatial windows are defined with the lip candidate center position and throat candidate center position of the corresponding candidate head middle cluster as the center; wherein, both the lip spatial window and the throat spatial window are contained within the valid candidate head spatial cluster.

9. The speech content recognition and identity recognition method based on single millimeter-wave radar as described in claim 8, characterized in that, The step-by-frame calculation of the phase difference of the signals in each window yields a first micro-Doppler signal reflecting the opening and closing motion of the target's lips, and a second micro-Doppler signal reflecting the vibration of the target's vocal cords, including: After capturing the frame-by-frame echo signals of each spatial cell within the lip and throat spatial windows of the same valid candidate head spatial cluster, coherent accumulation is performed on the frame-by-frame echo signals within the lip and throat spatial windows to obtain the composite echo signal of the lip and throat spatial windows of the valid candidate head spatial cluster. After extracting the phase of the composite echo signal from the lip space window and the throat space window, phase unwrapping is performed to obtain two unwrapped phase sequences, which are denoted as the unwrapped phase sequence of the lip space window and the unwrapped phase sequence of the throat space window of the effective candidate head space cluster. The two unwound phase sequences are subjected to time-domain difference to obtain the time-domain difference signal of the lip space window and the time-domain difference signal of the throat space window of the effective candidate head space cluster. The time-domain difference signal of the lip spatial window is filtered by a bandpass filter to obtain the first micro-Doppler signal, and the time-domain difference signal of the throat spatial window is filtered by a high-pass filter to obtain the second micro-Doppler signal.

10. A voice content recognition and identity recognition system based on single millimeter-wave radar, characterized in that, include: The target distance chamber acquisition unit is used to perform distance-dimensional Fourier transform on the preprocessed multi-echo signals to obtain a two-dimensional heat map including distance and time dimensions, and to filter out multiple target distance chambers that match human micro-movements based on the phase of each distance chamber in the two-dimensional heat map. The joint heatmap construction unit is used to construct a joint heatmap including distance and azimuth dimensions by performing two-stage DOA estimation on the preprocessed multi-echo signals according to the Capon algorithm based on the target distance bin and calculating the spatial micro-motion power at each spatial angle. Each spatial cell in the joint heatmap stores the spatial micro-motion power at each spatial angle. The two-stage DOA estimation means that DOA estimation is first performed on multiple spatial angles with zero pitch angle to obtain candidate spatial angles, and then DOA estimation is performed on candidate spatial angles with non-zero pitch angle to obtain effective spatial angles. The constant false alarm detection unit is used to perform constant false alarm detection on the joint heat map. The spatial cells in the joint heat map whose spatial micro-motion power is greater than the constant false alarm detection threshold are taken as the valid spatial cells of the joint heat map. The window geometry of the tested cell in the constant false alarm detection is a rhombus shape. The micro-Doppler signal separation unit is used to perform connected component clustering on the effective spatial cells of the joint heatmap and divide multiple groups of spatial windows in combination with human physiological constraints. Each group of spatial windows includes a lip spatial window and a larynx spatial window. The phase difference of the signals in each group of windows is calculated frame by frame to obtain the first micro-Doppler signal reflecting the opening and closing of the target's lips and the second micro-Doppler signal reflecting the vibration of the target's vocal cords. The Doppler time-frequency diagram generation unit is used to perform short-time Fourier transform on the time-synchronized first micro-Doppler signal and the second micro-Doppler signal respectively to obtain the first micro-Doppler time-frequency diagram and the second micro-Doppler time-frequency diagram with time as the horizontal axis and Doppler frequency as the vertical axis.

Images