A non-line-of-sight path speech recognition method and system based on millimeter-wave radar

Abstract

This invention provides a non-line-of-sight (NLS) speech recognition method and system based on millimeter-wave radar, comprising: acquiring the position of a target human body; distinguishing human body movement trajectories based on the target human body position using a preset maximum likelihood estimation algorithm; determining static objects within a set range based on the human body movement trajectories; extracting millimeter-wave signals emitted by the millimeter-wave radar and reflected by the static objects based on the static objects; extracting common frequency components of the millimeter-wave signals using a preset adaptive beamforming algorithm and a fine-grained bandpass filter; extracting vibration signals from the millimeter-wave signals; superimposing the vibration signals based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum; and performing speech recognition on the time-frequency spectrum using a preset recurrent neural network model to parse the meaning of the speech. This invention overcomes the inaccuracies of existing NLS speech recognition methods, achieving accurate speech recognition using millimeter-wave radar in NLS scenarios.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a non-line-of-sight path speech recognition method and system based on millimeter-wave radar. Background Technology

[0002] Millimeter-wave sensing technology is an important technique for sensing objects by collecting and analyzing reflected millimeter-wave signals. Millimeter waves, as a sensing medium, have received widespread attention in recent years due to their short wavelength and high spatial resolution. Applications range from micro-motion measurement, motion and activity sensing, material identification, environmental sensing, thermal imaging, and ferromagnetic tags.

[0003] Utilizing ultra-high frequency (UHF) technology, millimeter-wave sensing can achieve millimeter-level and even sub-millimeter-level displacement measurements. mTrack achieves millimeter-level tracking accuracy using a signal phase-based model. mmVib introduces a multi-signal merging model to achieve sub-millimeter-level vibration measurement errors. These works demonstrate the capability of millimeter-wave sensing to measure minute displacements, which can be applied to speech sensing.

[0004] Millimeter-wave sensing holds immense potential for supporting intelligent human-computer interaction, such as speech recognition. Traditional solutions use microphones to collect sound and then analyze the human voice contained within it. Their performance degrades significantly in noisy environments. Due to the ultra-high frequency and high spatial resolution of millimeter-wave signals, speech-related vibration signals can be extracted from the near-laryngeal region within the radar's field of view. Speech recognition can be achieved by analyzing these vibration signals. WaveEar transmits millimeter-wave signals to the near-laryngeal region to sense sound vibrations and recover the sound. Vocalprint utilizes millimeter-wave sensing to preserve fine-grained speech biometric attributes and achieve speech authentication. RadioMic implements a training-free method for robust sound detection and high-fidelity sound recovery based on minute vibrations of the sound source. Wavoice achieves accurate speech recognition by fusing multimodal signals (millimeter-wave and audio signals). All of these methods sense sound by analyzing vibration signals from the sound source, which requires a line-of-sight path between the sound source (e.g., a person's throat) and the radar. These methods fail to perform well when the line-of-sight path disappears with changes in a person's position and posture. This problem also exists in other millimeter-wave-based human perception applications. Whether locating and tracking the position and posture of a human body, or sensing vital signs such as respiration and cardiac signals, line-of-sight sensing is essential. How to achieve millimeter-wave sensing in non-line-of-sight scenarios remains an unsolved problem. Summary of the Invention

[0005] This invention provides a non-line-of-sight (NLS) speech recognition method and system based on millimeter-wave radar, which solves the defects of inaccurate existing NLS speech recognition and realizes accurate speech recognition through millimeter-wave radar in NLS paths.

[0006] This invention provides a non-line-of-sight path speech recognition method based on millimeter-wave radar, comprising:

[0007] The location of the target human body is obtained, and the human body movement trajectory is distinguished based on the location of the target human body using a preset maximum likelihood estimation algorithm. Static objects are determined within a set range based on the human body movement trajectory.

[0008] Based on the static object, the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object is extracted, and the common frequency component of the millimeter-wave signal is extracted by a preset adaptive beamforming algorithm and a fine-grained bandpass filter.

[0009] Vibration signals are extracted from the millimeter-wave signals, and the vibration lines are superimposed based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum.

[0010] The time-frequency spectrum is used to perform speech recognition and extract the meaning of the speech using a pre-defined recurrent neural network model.

[0011] The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm.

[0012] According to the present invention, a non-line-of-sight path speech recognition method based on millimeter-wave radar is provided, wherein the position of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm, specifically including:

[0013] By scanning the environment with millimeter-wave radar, the signal strength of all locations within a set range is obtained, and a range angle spectrum is generated;

[0014] Based on the range angle spectrum, a preset constant false alarm rate (CFAR) algorithm is used to detect targets in environmental noise and generate CFAR results.

[0015] The location of the target human body is obtained by performing clustering operations on the constant false alarm rate results obtained by the preset clustering algorithm.

[0016] According to the present invention, a non-line-of-sight path speech recognition method based on millimeter-wave radar is provided, wherein the steps of acquiring the position of the target human body, distinguishing the human body movement trajectory based on the position of the target human body using a preset maximum likelihood estimation algorithm, and determining static objects within a set range based on the human body movement trajectory specifically include:

[0017] Based on the location information of the target human body, calculate the Jaccard similarity coefficient between each object in the scan results of two adjacent frames of millimeter-wave radar;

[0018] The similarity coefficient of Jaccard is used to measure the similarity of cluster positions in two adjacent scan results, and a bipartite graph is constructed to obtain trajectory information.

[0019] Based on the trajectory information, the trajectory with the largest variance is calculated using a preset Kuhn-Mankers algorithm and identified as the human movement trajectory.

[0020] Based on the human body's movement trajectory, static objects within a set range of the current human body's location are identified, and multiple objects with high signal reflection intensity are selected as reflectors to extract the reflected signals.

[0021] According to the present invention, a non-line-of-sight path speech recognition method based on millimeter-wave radar is provided. The method involves extracting millimeter-wave signals emitted by the millimeter-wave radar and reflected by the static object, and extracting the common frequency components of the millimeter-wave signals using a preset adaptive beamforming algorithm and a fine-grained bandpass filter. Specifically, the method includes:

[0022] Extract millimeter-wave signals emitted by the millimeter-wave radar and reflected by the reflectors from multiple selected reflectors;

[0023] The common component of the reflected millimeter wave signal is extracted using a preset adaptive beamforming algorithm;

[0024] By using a pre-set fine-grained bandpass filter, the signals in the relevant frequency bands of all reflected millimeter-wave signals are preserved, and the common frequency components are determined.

[0025] According to the present invention, a non-line-of-sight path speech recognition method based on millimeter-wave radar extracts vibration signals from the millimeter-wave signals and superimposes the vibration signals based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum, specifically including:

[0026] Vibration signals are extracted from the millimeter-wave signals and superimposed into an enhanced signal.

[0027] The amplification factor of other signals is calculated based on the most recent vibration signal, and the amplified signal that is most similar to the reference and the corresponding factor is selected as the final superimposed signal;

[0028] The superimposed signal is normalized and subjected to short-time Fourier transform to obtain a time-frequency spectrum.

[0029] According to the present invention, a non-line-of-sight path speech recognition method based on millimeter-wave radar is provided, wherein the speech recognition is performed on the time-frequency spectrum using a preset recurrent neural network model to parse the speech meaning, specifically including:

[0030] The recurrent neural network model includes: a feature encoding network, a label encoding network, and a joint network;

[0031] The feature encoding network extracts global and local correlation features from the time-frequency spectrum. The input is processed by a convolutional acquisition layer using a preset feature encoder. The global correlation is learned and local correlation is captured by multiple conformation blocks. The context of each position is encoded into a higher-level representation.

[0032] The label encoding network encodes a higher-level representation based on the same context, converts previously predicted non-blank labels into vector representations, and adds a masking operation to the attention score of the label encoding network to generate label codes;

[0033] The joint network linearly adds the outputs of the feature encoding network and the label encoding network to calculate the probability distribution on the vocabulary of the sentence segment and output the speech meaning.

[0034] The present invention also provides a non-line-of-sight path speech recognition system based on millimeter-wave radar, comprising:

[0035] The object detection module is used to obtain the position of the target human body, distinguish the human body movement trajectory based on the position of the target human body using a preset maximum likelihood estimation algorithm, and determine static objects within a set range based on the human body movement trajectory.

[0036] The common frequency component extraction module is used to extract the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object based on the static object, and to extract the common frequency component of the millimeter-wave signal through a preset adaptive beamforming algorithm and a fine-grained bandpass filter.

[0037] The signal superposition module is used to extract vibration signals from the millimeter-wave signals and superimpose the vibration signals based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum.

[0038] The speech recognition module is used to perform speech recognition on the time-frequency spectrum using a preset recurrent neural network model, and to parse out the meaning of the speech.

[0039] The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the non-line-of-sight path speech recognition method based on millimeter-wave radar as described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the non-line-of-sight path speech recognition method based on millimeter-wave radar as described above.

[0042] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the non-line-of-sight path speech recognition method based on millimeter-wave radar as described above.

[0043] This invention provides a non-line-of-sight path speech recognition method and system based on millimeter-wave radar. By performing surrounding object detection, common frequency component extraction, signal superposition, and speech recognition, a speech recognition model based on millimeter-wave radar is established to recognize speech in the non-line-of-sight range. It only collects vibration signals of surrounding objects and does not require locating the person's throat, thus broadening the application scenarios and enabling accurate non-line-of-sight speech recognition. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is one of the flowcharts of a non-line-of-sight path speech recognition method based on millimeter-wave radar provided by the present invention;

[0046] Figure 2 This is the second flowchart of a non-line-of-sight path speech recognition method based on millimeter-wave radar provided by the present invention;

[0047] Figure 3 This is the third flowchart of a non-line-of-sight path speech recognition method based on millimeter-wave radar provided by the present invention;

[0048] Figure 4 This is the fourth flowchart of a non-line-of-sight path speech recognition method based on millimeter-wave radar provided by the present invention;

[0049] Figure 5 This is a schematic diagram of the module connection of a non-line-of-sight path speech recognition system based on millimeter-wave radar provided by the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention;

[0051] Figure 7 This is a diagram of the recurrent neural network model architecture provided by the present invention.

[0052] Figure label:

[0053] 110: Object detection module; 120: Common frequency component extraction module; 130: Signal superposition module; 140: Speech recognition module;

[0054] 610: Processor; 620: Communication interface; 630: Memory; 640: Communication bus. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] The following is combined with Figures 1-4 The present invention describes a non-line-of-sight path speech recognition method based on millimeter-wave radar, comprising:

[0057] S100: Obtain the position of the target human body, distinguish the human body movement trajectory based on the position of the target human body using a preset maximum likelihood estimation algorithm, and determine the static object within a set range based on the human body movement trajectory.

[0058] S200. Based on the static object, extract the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object, and extract the common frequency component of the millimeter-wave signal through a preset adaptive beamforming algorithm and fine-grained bandpass filter.

[0059] S300. Extract the vibration signal from the millimeter wave signal, and superimpose the vibration lines based on the common frequency components of the millimeter wave signal to generate a time-frequency spectrum.

[0060] S400. The time-frequency spectrum is subjected to speech recognition using a preset recurrent neural network model to parse out the meaning of the speech.

[0061] The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm.

[0062] The specific scheme for extracting vibration signals using millimeter-wave radar in this invention is as follows:

[0063] Millimeter-wave radars typically transmit frequency-modulated continuous wave (FMCW) signals periodically to measure target distance. The frequency difference between the transmitted and received signals corresponds to the signal propagation time and can be used to calculate the propagation distance. By mixing the transmitted and received signals, a beat frequency signal s(t) can be obtained as shown below:

[0064] s(t)=αexp[j4π(f c +Kt)R(t) / c]

[0065] Where α represents the path loss, f c K and are the chirp (coded pulse technique) start frequency and slope of the frequency-modulated continuous wave signal, respectively. To separate the Rx signal components reflected from different ranges, a range-fast Fourier transform (FFT) operation is performed on the samples of s(t) within the chirp to separate the reflected signals from different distances. Then, at a specific distance, these samples can form a new reflected signal S(t). The reflected signal can be calculated as:

[0066]

[0067] Where R(t) represents the distance between the radar and the target. When the object vibrates, the distance R(t) can be rewritten as R(t) = R0 + x(t), where R0 represents the radar distance to the object and x(t) represents the displacement of the object due to vibration.

[0068] The signal S(t) is represented as an arc in the IQ domain. The phase change of the arc represents the change in radar target range, which can be obtained as follows:

[0069]

[0070] Where φ n Let x(n) represent the phase of the nth sample, and let x(n) represent the change in radar target distance.

[0071] When a human speaks, the vocal cords vibrate first, generating airflow. This vibration signal can be represented as a superposition of a series of sinusoidal signals:

[0072]

[0073] This airflow passes through the vocal tract, which consists of the soft palate, tongue, nasal cavity, and oral cavity. The vocal tract acts as a complex filter to correct the signal's spectrum and produce the final sound. Sound from the oral cavity can be represented as:

[0074] s1(t)=h1*v1(t)

[0075] Where h1 represents the filtering process, and * represents the convolution operation. Because the vibrations of surrounding objects are directly excited by the sound signal, when the sound signal reaches the target, it can be represented as:

[0076] s2(t)=h2*s1(t)

[0077] Where h2 represents the air channel response of the sound signal. Since sound at different frequencies attenuates differently when propagating in the air channel, h2 can be regarded as a composite filter acting on the sound.

[0078] Sound around an object causes the object to vibrate. Considering that the object's material, shape, and other properties affect the vibration signal excited by sound, the object's vibration signal can be represented as:

[0079] v2(t)=h3*(s2(t)+n1(t))=h3*s2(t)+h3*n1(t)

[0080] Where h3 represents the conversion process from sound to vibration signal, and n1 represents the ambient noise around the object.

[0081] When the millimeter-wave radar senses this vibration signal, the received millimeter-wave signal can be represented as:

[0082] m(t) = g(v²(t)) + n²(t)

[0083] Where g() is described in formula (1), and n2(t) represents random electromagnetic noise in signal processing.

[0084] Based on the above model, the relationship between the speech signal s1(t) and the millimeter-wave signal m(t) can be obtained:

[0085] m(t)=g(h3*(h2*s1(t))+h3*n1(t))+n2(t)

[0086] There is a strong correlation between speech signals and millimeter-wave signals. Speech signals can be analyzed using millimeter-wave signals. Since g(), representing signal processing, is constant, three factors affect the quality of the received signal: the spectral distortion h3*h2 between the speech and vibration signals, the ambient noise n1(t) around the object, and the electromagnetic noise n2(t) in millimeter-wave signal processing. These factors collectively affect signal quality. Considering noisy environments, signal quality degradation is more severe and must be addressed.

[0087] The location of the target human body was obtained through millimeter-wave radar scanning and calculation using a constant false alarm rate (CFAR) algorithm and a clustering algorithm, specifically including:

[0088] By scanning the environment with millimeter-wave radar, the signal strength of all locations within a set range is obtained, and a range angle spectrum is generated;

[0089] Based on the range angle spectrum, a preset constant false alarm rate (CFAR) algorithm is used to detect targets in environmental noise and generate CFAR results.

[0090] The location of the target human body is obtained by performing clustering operations on the constant false alarm rate results obtained by the preset clustering algorithm.

[0091] In this invention, the environment is periodically scanned to obtain the signal strength, i.e., the range-angle spectrum, at all locations within the field of view. This can be obtained by applying the classic Range FFT and receiver beamforming algorithms. Then, a constant false alarm rate (CFAR) algorithm is applied to the range-angle spectrum to detect these range-angle regions containing objects. CFAR is a standard adaptive algorithm used to detect targets in ambient noise. After estimating the noise level by convolving a CFAR window (constant false alarm rate detection window) with the signal strength, regions with energy higher than the noise level are preserved and considered as regions containing objects. According to experimental results, the detection results are best when both the guard unit and training unit values ​​are set to 2.

[0092] After obtaining the constant false alarm rate (CFAR) results, the DBSCAN clustering algorithm is applied to cluster the CFAR results. DBSCAN is a classic clustering algorithm that does not assume the number or shape of clusters, making it suitable for this invention. Each cluster center represents an object in the field of view, which can be represented as:

[0093] O j : <D j A j S j >

[0094] Among them O j D represents the j-th object. j A j and S j These are the distance between the j-th object and the radar, the angle between the j-th object and the radar, and the intensity of the reflected signal, respectively.

[0095] The process of obtaining the location of the target human body, distinguishing the human body's movement trajectory based on the target human body's location using a preset maximum likelihood estimation algorithm, and determining static objects within a set range based on the human body's movement trajectory specifically includes:

[0096] S101. Based on the location information of the target human body, calculate the Jaccard similarity coefficient between each object in the scan results of two adjacent frames of millimeter-wave radar.

[0097] S102. The similarity of clustering positions in two adjacent scan results is measured by the Jaccard similarity coefficient, and a bipartite graph is constructed to obtain trajectory information.

[0098] S103. Based on the trajectory information, the trajectory with the largest variance is calculated using the preset Kuhn-Mankers algorithm as the human movement trajectory;

[0099] S104. Based on the human body's movement trajectory, determine the static objects within a set range of the current human body's location, and select multiple objects with high signal reflection intensity as reflectors to extract the reflected signals.

[0100] Considering that a person's trajectory has more variations than that of a stationary object, these variations can be used to determine the person's position. Specifically, the scan results P of two adjacent frames are first calculated. i and P i+1 Jaccard similarity coefficient between each pair of objects in the table:

[0101]

[0102] Where C i，j P represents the result of the i-th scan. i The j-th cluster in N i and N i+1 They are P i and P i+1 The number of clustering results in the sample set. The Jaccard similarity coefficient is a statistical metric used to measure the similarity and diversity of sample sets. Here, it is used to measure the similarity of clustering locations in two adjacent scan results. A bipartite graph is then constructed to obtain the trajectory. The two vertex sets of the bipartite graph are the clusters of the two scan results, and the weight of the edge between the vertices is J. j，k This modeling transforms the tracking problem into an optimal matching problem. The classic Kuhn-Munkres algorithm is used to solve this problem, finding the match with the largest sum of weights. Such a match maximizes the positional similarity of the matching clusters. The clusters of matches in the optimal matching problem are then considered as trajectories of the same object. The trajectory with the largest variance is then selected as the person's trajectory.

[0103] After tracking a person's position, an attempt is made to locate static objects within a specific range (e.g., 1m) around that position. The three objects with the highest reflected signal intensity are selected as reflectors for further signal extraction. Considering the limited speed of human movement, the environmental detection update frequency is set to 0.5s. This method effectively handles the dynamic nature of the human body.

[0104] Based on the extraction of millimeter-wave signals emitted by millimeter-wave radar and reflected by the static object, the common frequency components of the millimeter-wave signals are extracted using a preset adaptive beamforming algorithm and a fine-grained bandpass filter. Specifically, this includes:

[0105] S201. Extract the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the reflector from the selected multiple reflectors;

[0106] S202. Extract the common component of the reflected millimeter wave signal using a preset adaptive beamforming algorithm;

[0107] S203. By using a preset fine-grained bandpass filter, the signals in the relevant frequency bands of all reflected millimeter-wave signals are retained, and the fair frequency components are determined.

[0108] In this invention, after selecting a reflector, the reflected signal is extracted from these static objects using a beamforming algorithm at the receiver. However, these reflected signals cannot be directly used to measure the reflector's vibration. For example, in preliminary studies, the vibration amplitude is approximately 2-5 μm, and the corresponding phase change in the reflected signal is approximately 0.003-0.008 rad. This tiny phase change caused by weak vibration is easily masked by electromagnetic noise. If the reflected signal is observed in the IQ domain, the arc shape formed by these reflected signals can easily transform into a clumped shape under the influence of electromagnetic noise. If the vibration signal is extracted directly, the corresponding arc cannot be identified, and the calculated phase change will be completely incorrect. On the other hand, since it is desirable to operate in noisy environments, the reflector is also affected by environmental noise. If the vibration signal is extracted directly, the vibration signal will contain vibration components caused by environmental noise.

[0109] To address this issue, the common frequency components of multiple reflected signals are calculated, which can be used to improve the signal-to-noise ratio of the reflected signals. Preliminary studies indicate that the range within which sound can excite vibrations in surrounding objects is limited; therefore, environmental noise typically only affects a portion of the reflectors. Considering that the vibrations of all reflectors contain sound information, the common components can help resist the effects of environmental noise. Simultaneously, this information can filter out the effects of electromagnetic noise.

[0110] A general common component extraction algorithm was designed based on the MVDR algorithm (MVDR is an adaptive beamforming algorithm based on the maximum signal-to-noise ratio criterion). This algorithm is used for receiver beamforming tasks to analyze whether signals from different directions are correlated. Its idea and structure naturally meet the requirements for extracting common components. It is modified to analyze whether signals are correlated at different frequencies. In the original MVDR algorithm, there is a known unitary matrix U. Each column vector of U represents a spatial direction. When U is transformed by the Fourier matrix F = [f0 f1 ... f...], the common component extraction algorithm is derived from the common component extraction algorithm. K-1 During the replacement, the functions of the MVDR and generalized MVDR spectra in the spatial domain are replaced by functions in the frequency domain. Similar to the original calculation process, the two signals x are calculated in the k-th frequency unit. i (n) and x j The cross spectrum of (n) is:

[0111]

[0112] in

[0113]

[0114] x H Let x represent the conjugate transpose of x. In the k-th frequency unit, the correlation between two signals x1(n) and x2(n) is:

[0115]

[0116] A time window length similar to that of some speech recognition tasks (e.g., 20 milliseconds) is chosen, and the MVDR algorithm is applied to each pair of reflected signals within each time window. Common components are then determined by comparing the correlation of each pair to an empirical threshold. Fine-grained bandpass filters are then applied to all reflected signals to preserve signals in these correlated frequency bands.

[0117] Vibration signals are extracted from the millimeter-wave signals, and the vibration lines are superimposed based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum, specifically including:

[0118] S301. Extract the vibration signal from the millimeter-wave signal and superimpose the vibration signal into an enhanced signal;

[0119] S302. Calculate the amplification factor of other signals based on the most recent vibration signal, and select the amplified signal that is most similar to the reference and the corresponding factor as the final superimposed signal;

[0120] S303. Normalize and perform short-time Fourier transform on the superimposed signal to obtain a time-frequency spectrum.

[0121] In this invention, the vibration of each reflector can be obtained by filtering electromagnetic noise. To further resist the influence of environmental noise, the signals of each reflector are superimposed to obtain the final time-frequency diagram used for speech recognition.

[0122] First, the vibration signal is extracted from the filtered reflected signal. The IQ samples of the reflected signal form an arc-shaped trajectory, and the phase change of the sampling points on the arc represents the displacement of the vibration signal. To further combat the low signal-to-noise ratio, a circle fitting algorithm is performed on these IQ sampling points. Let P = {p n}, p n ∈R 2 Let represent the IQ sampling points. The circle fitting problem can be described as calculating the center c and radius r of the circle to minimize the sum of the distances between the sampling points and the circle:

[0123]

[0124] Several algorithms can solve this nonlinear least squares optimization problem. This invention uses the classic Levenberg-Marquardt algorithm because of its versatility and effectiveness.

[0125] Once the center and radius of the circle are determined, the phase φ of the sampling points on the circle can be determined. n And it can be converted into a vibration signal x(n) according to equation (2).

[0126] To further combat the effects of environmental noise, these vibration signals are superimposed into an enhanced signal. Considering that different types of objects have different amplitudes under the same acoustic excitation, the amplification factor of other signals is calculated using the most recent vibration signal as a reference. The amplified signal most similar to the reference and its corresponding factor is selected.

[0127]

[0128] Where v1 represents the most recent vibration signal, v i and λ i (i = 2, 3) represents the i-th vibration signal and its amplification factor, and xcorr represents the cross-correlation operation. Superimposed signal v s It can be represented as:

[0129]

[0130] After obtaining the superimposed signal, it is normalized and subjected to short-time Fourier transform to obtain a time-spectrum, which can be used as input for further speech recognition.

[0131] The time-frequency spectrum is processed by a pre-defined recurrent neural network model to perform speech recognition and parse the speech meaning, specifically including:

[0132] The recurrent neural network model includes: a feature encoding network, a label encoding network, and a joint network;

[0133] The feature encoding network extracts global and local correlation features from the time-frequency spectrum. The input is processed by a convolutional acquisition layer using a preset feature encoder. The global correlation is learned and local correlation is captured by multiple conformation blocks. The context of each position is encoded into a higher-level representation.

[0134] The label encoding network encodes a higher-level representation based on the same context, converts previously predicted non-blank labels into vector representations, and adds a masking operation to the attention score of the label encoding network to generate label codes;

[0135] The joint network linearly adds the outputs of the feature encoding network and the label encoding network to calculate the probability distribution on the vocabulary of the sentence segment and output the speech meaning.

[0136] The present invention obtains the time-frequency spectrum of the enhanced vibration signal caused by the user's voice, and the ultimate task is to identify the semantic information contained in such vibration signals.

[0137] Due to sound attenuation in the air channel and energy loss during sound vibration conversion, vibration signals exhibit significant spectral distortion compared to speech signals. This signal distortion renders commonly used speech information feature extraction techniques, such as MFCC and wavelet transform, unusable directly. Therefore, this invention designs an end-to-end network to extract speech-related features and decode them into semantic information.

[0138] like Figure 7 As shown, this invention employs a recurrent neural network (RNN-T) framework for speech recognition. Because RNN-T can resolve the misalignment between vibration signals and semantic information, this model is small and highly accurate. The recognition framework comprises three modules: a feature encoding network, a label encoding network, and a joint network. The feature encoding network extracts hidden features from the input sequence, the label encoding network calculates the corresponding predictive codes, and to preserve historical prediction outputs, the joint network linearly adds the outputs of the two encoders to calculate the probability distribution over the sentence segment vocabulary.

[0139] Feature Encoding Network: Extracts two types of feature information from the distorted spectrum: global correlation and local correlation. The former contains the contextual relationships of the sound, while the latter represents a specific letter. The feature encoder first processes the input using a convolutional subsampling layer, which shortens the time series length and fuses contextual information. Then, multiple conformational blocks are applied to learn global correlation and effectively capture local correlation.

[0140] The construct combines self-attention and convolution operations to learn global interactions and capture local relevance, respectively. It comprises two feedforward (FFN) modules, a multi-head self-attention (MHSA) module, and a convolutional module. The FFN module performs feature transformation, enhancing the model's representational power. The MHSA module integrates a relative sinusoidal positional encoding scheme to capture the internal structure and representation of the sentence, as well as the semantics and relevance at different positions, thus enabling it to withstand the effects of variable input lengths. The convolutional module encodes the context of each position into a higher-level representation.

[0141] Label Encoder and Joint Network: Considering that speech information depends on context, a label encoder module is introduced to embed previous outputs. First, the embedding layer converts previously predicted non-whitespace labels into vector representations. Then, several linear layers project the embedding vectors, followed by a self-attention layer. Simultaneously, a masking operation is added to the attention score to access only past states and ensure causality.

[0142] For simplicity and efficiency, the joint network uses only a single fully connected feedforward neural network with a single hidden layer and tanh (hyperbolic tangent) as the activation function. The outputs of the multi-channel audio encoder and the tag encoder are concatenated as the input to the joint network.

[0143] In the specific operation, the number N of consistent blocks CR The embedding dimension is set to 512, and the size of the hidden states in the feedforward sublayer is 1024. The Adam optimizer (Adaptive Moment Estimator) is used. The Adam algorithm, a first-order gradient optimization algorithm for a stochastic objective function, is based on adaptive low-order moment estimation. The network is trained with β1 = 0.9, β2 = 0.98, and an adaptive learning rate scheduling ∈ 10. -9 .

[0144] For regularization, dropout is applied to each remaining unit of the conformation (i.e., the output of each module) before it is added to the module input. The probability of using dropout is P. drop =0.1. Variational noise is introduced into the model as regularization. An L2 regularization module with weights of 1e-6 is added to all trainable weights in the network.

[0145] In a specific example, this is implemented based on the commercial millimeter-wave radar Texas Instruments IWR1642. The radar board has two TX antennas and four RX antennas. An FMCW signal is transmitted from 77 GHz with a bandwidth of 4.0 GHz via one TX antenna, and all RX antennas receive reflected signals. The ramp-end time and idle time are set to 80 µs and 20 µs, respectively. In this configuration, the slow-time sampling rate can be calculated as follows: The frequency slope of the FMCW signal is 49.97 MHz / µs, and the radar's ADC sampling rate is 3430 kHz. Therefore, the radar's maximum detection range is... Considering the radar's field of view is approximately 120°, the raw radar data is acquired by the TI DCA1000EVM data acquisition board, ensuring high-speed real-time transmission.

[0146] refer to Figure 5 The present invention also discloses a non-line-of-sight path speech recognition system based on millimeter-wave radar, comprising:

[0147] The object detection module 110 is used to obtain the position of the target human body, distinguish the human body movement trajectory according to the position of the target human body through a preset maximum likelihood estimation algorithm, and determine static objects within a set range based on the human body movement trajectory.

[0148] The common frequency component extraction module 120 is used to extract the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object based on the static object, and to extract the common frequency component of the millimeter-wave signal through a preset adaptive beamforming algorithm and a fine-grained bandpass filter.

[0149] The signal superposition module 130 is used to extract vibration signals from the millimeter wave signals and superimpose the vibration signals based on the common frequency components of the millimeter wave signals to generate a time-frequency spectrum.

[0150] The speech recognition module 140 is used to perform speech recognition on the time-frequency spectrum using a preset recurrent neural network model and parse out the meaning of the speech.

[0151] The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm.

[0152] The object detection module 110 scans the environment with millimeter-wave radar to obtain the signal strength of all locations within a set range and generates a range angle spectrum.

[0153] Based on the range angle spectrum, a preset constant false alarm rate (CFAR) algorithm is used to detect targets in environmental noise and generate CFAR results.

[0154] The location of the target human body is obtained by performing clustering operations on the constant false alarm rate results obtained by the preset clustering algorithm.

[0155] The similarity coefficient of Jaccard is used to measure the similarity of cluster positions in two adjacent scan results, and a bipartite graph is constructed to obtain trajectory information.

[0156] Based on the trajectory information, the trajectory with the largest variance is calculated using a preset Kuhn-Mankers algorithm and identified as the human movement trajectory.

[0157] Based on the human body's movement trajectory, static objects within a set range of the current human body's location are identified, and multiple objects with high signal reflection intensity are selected as reflectors to extract the reflected signals.

[0158] The common frequency component extraction module 120 extracts millimeter-wave signals emitted by the millimeter-wave radar and reflected by the reflectors from a selection of multiple reflectors;

[0159] The common component of the reflected millimeter wave signal is extracted using a preset adaptive beamforming algorithm;

[0160] By using a pre-set fine-grained bandpass filter, the signals in the relevant frequency bands of all reflected millimeter-wave signals are preserved, and the fair frequency components are determined.

[0161] The signal superposition module 130 extracts the vibration signal from the millimeter-wave signal and superimposes the vibration signal into an enhanced signal.

[0162] The amplification factor of other signals is calculated based on the most recent vibration signal, and the amplified signal that is most similar to the reference and the corresponding factor is selected as the final superimposed signal;

[0163] The superimposed signal is normalized and subjected to short-time Fourier transform to obtain a time-frequency spectrum.

[0164] The speech recognition module 140 extracts global and local correlation feature information from the time-frequency spectrum through the feature encoding network, processes the input using a convolutional acquisition layer with a preset feature encoder, and learns global correlation to capture local correlation through multiple conformation blocks, encoding the context of each position into a higher-level representation.

[0165] The label encoding network encodes a higher-level representation based on the same context, converts previously predicted non-blank labels into vector representations, and adds a masking operation to the attention score of the label encoding network to generate label codes;

[0166] The joint network linearly adds the outputs of the feature encoding network and the label encoding network to calculate the probability distribution on the vocabulary of the sentence segment and output the speech meaning.

[0167] This invention provides a non-line-of-sight path speech recognition system based on millimeter-wave radar. By performing surrounding object detection, common frequency component extraction, signal superposition, and speech recognition, a speech recognition model based on millimeter-wave radar is established to recognize speech within the non-line-of-sight range. It only collects vibration signals from surrounding objects and does not require locating the person's throat, thus broadening the application scenarios and enabling accurate non-line-of-sight speech recognition.

[0168] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a non-line-of-sight path speech recognition method based on millimeter-wave radar. This method includes: acquiring the position of a target human body; distinguishing the human body's movement trajectory based on the target human body's position using a preset maximum likelihood estimation algorithm; and determining a static object within a set range based on the human body's movement trajectory.

[0169] Based on the static object, the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object is extracted, and the common frequency component of the millimeter-wave signal is extracted by a preset adaptive beamforming algorithm and a fine-grained bandpass filter.

[0170] Vibration signals are extracted from the millimeter-wave signals, and the vibration lines are superimposed based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum.

[0171] The time-frequency spectrum is used to perform speech recognition and extract the meaning of the speech using a pre-defined recurrent neural network model.

[0172] The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm.

[0173] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute a non-line-of-sight path speech recognition method based on millimeter-wave radar provided by the above methods, the method including: obtaining the position of a target human body, distinguishing human body movement trajectories based on the position of the target human body using a preset maximum likelihood estimation algorithm, and determining static objects within a set range based on the human body movement trajectory;

[0175] Based on the static object, the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object is extracted, and the common frequency component of the millimeter-wave signal is extracted by a preset adaptive beamforming algorithm and a fine-grained bandpass filter.

[0176] Vibration signals are extracted from the millimeter-wave signals, and the vibration lines are superimposed based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum.

[0177] The time-frequency spectrum is used to perform speech recognition and extract the meaning of the speech using a pre-defined recurrent neural network model.

[0178] The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm.

[0179] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a non-line-of-sight path speech recognition method based on millimeter-wave radar provided by the above methods. The method includes: acquiring the position of a target human body, distinguishing human body movement trajectories based on the position of the target human body using a preset maximum likelihood estimation algorithm, and determining static objects within a set range based on the human body movement trajectories.

[0180] Based on the static object, the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object is extracted, and the common frequency component of the millimeter-wave signal is extracted by a preset adaptive beamforming algorithm and a fine-grained bandpass filter.

[0181] Vibration signals are extracted from the millimeter-wave signals, and the vibration lines are superimposed based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum.

[0182] The time-frequency spectrum is used to perform speech recognition and extract the meaning of the speech using a pre-defined recurrent neural network model.

[0183] The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0185] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-line-of-sight path speech recognition method based on millimeter-wave radar, characterized in that, include: The location of the target human body is obtained, and the human body movement trajectory is distinguished based on the location of the target human body using a preset maximum likelihood estimation algorithm. Static objects are determined within a set range based on the human body movement trajectory. Based on the static object, the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object is extracted, and the common frequency component of the millimeter-wave signal is extracted by a preset adaptive beamforming algorithm and a fine-grained bandpass filter. Vibration signals are extracted from the millimeter-wave signals, and the vibration lines are superimposed based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum. The time-frequency spectrum is used to perform speech recognition and extract the meaning of the speech using a pre-defined recurrent neural network model. The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm. Specifically, obtaining the location of the target human body, distinguishing human movement trajectories based on the target human body's location using a preset maximum likelihood estimation algorithm, and determining static objects within a set range based on the human movement trajectories include: Based on the location information of the target human body, calculate the Jaccard similarity coefficient between each object in the scan results of two adjacent frames of millimeter-wave radar; The similarity coefficient of Jaccard is used to measure the similarity of cluster positions in two adjacent scan results, and a bipartite graph is constructed to obtain trajectory information. Based on the trajectory information, the trajectory with the largest variance is calculated using a preset Kuhn-Mankers algorithm and identified as the human movement trajectory. Based on the human movement trajectory, static objects within a set range of the current human location are identified, and multiple static objects with high signal reflection intensity are selected as reflectors to extract the reflected signals.

2. The non-line-of-sight path speech recognition method based on millimeter-wave radar according to claim 1, characterized in that, The location of the target human body was obtained through millimeter-wave radar scanning and calculation using a constant false alarm rate algorithm and a clustering algorithm, specifically including: By scanning the environment with millimeter-wave radar, the signal strength of all locations within a set range is obtained, and a range angle spectrum is generated; Based on the range angle spectrum, a preset constant false alarm rate (CFAR) algorithm is used to detect targets in environmental noise and generate CFAR results. The target human body location is obtained by performing clustering operations on the constant false alarm rate results using a preset clustering algorithm.

3. The non-line-of-sight path speech recognition method based on millimeter-wave radar according to claim 1, characterized in that, The step of extracting millimeter-wave signals emitted by millimeter-wave radar and reflected by static objects based on the static object, and extracting the common frequency components of the millimeter-wave signals through a preset adaptive beamforming algorithm and fine-grained bandpass filter, specifically includes: Extract millimeter-wave signals emitted by the millimeter-wave radar and reflected by the reflectors from a selection of multiple reflectors; The common component of the reflected millimeter wave signal is extracted using a preset adaptive beamforming algorithm; By using a pre-set fine-grained bandpass filter, the signals in the relevant frequency bands of all reflected millimeter-wave signals are preserved, and the common frequency components are determined.

4. The non-line-of-sight path speech recognition method based on millimeter-wave radar according to claim 1, characterized in that, Vibration signals are extracted from the millimeter-wave signals, and the vibration signals are superimposed based on the common frequency components of the millimeter-wave signals to generate a time-frequency spectrum, specifically including: Extract vibration signals from the millimeter-wave signals; The amplification factor of other signals is calculated based on the most recent vibration signal, and the amplified signal that is most similar to the reference and the corresponding factor is selected as the final superimposed signal; The final superimposed signal is normalized and subjected to short-time Fourier transform to obtain a time-frequency spectrum.

5. The non-line-of-sight path speech recognition method based on millimeter-wave radar according to claim 1, characterized in that, The step of performing speech recognition on the time-frequency spectrum using a preset recurrent neural network model to parse the speech meaning specifically includes: The recurrent neural network model includes: a feature encoding network, a label encoding network, and a joint network; The feature encoding network extracts global and local correlation features from the time-frequency spectrum. The input is processed by a convolutional acquisition layer using a preset feature encoder. The global correlation is learned and local correlation is captured by multiple conformation blocks. The context of each position is encoded into a higher-level representation. The label encoding network encodes a higher-level representation based on the same context, converts previously predicted non-blank labels into vector representations, and adds a masking operation to the attention score of the label encoding network to generate label codes; The joint network linearly adds the outputs of the feature encoding network and the label encoding network to calculate the probability distribution on the vocabulary of the sentence segment and output the speech meaning.

6. A non-line-of-sight path speech recognition system based on millimeter-wave radar, characterized in that, include: The object detection module is used to obtain the position of the target human body, distinguish the human body movement trajectory based on the position of the target human body using a preset maximum likelihood estimation algorithm, and determine static objects within a set range based on the human body movement trajectory. The common frequency component extraction module is used to extract the millimeter-wave signal emitted by the millimeter-wave radar and reflected by the static object based on the static object, and to extract the common frequency component of the millimeter-wave signal through a preset adaptive beamforming algorithm and a fine-grained bandpass filter. The signal superposition module is used to extract vibration signals from the millimeter wave signals and superimpose the vibration signals based on the common frequency components of the millimeter wave signals to generate a time-frequency spectrum. The speech recognition module is used to perform speech recognition on the time-frequency spectrum using a preset recurrent neural network model, and to parse out the meaning of the speech. The location of the target human body is obtained by scanning with millimeter-wave radar and performing calculations using a constant false alarm rate algorithm and a clustering algorithm. Specifically, obtaining the location of the target human body, distinguishing human movement trajectories based on the target human body's location using a preset maximum likelihood estimation algorithm, and determining static objects within a set range based on the human movement trajectories include: Based on the location information of the target human body, calculate the Jaccard similarity coefficient between each object in the scan results of two adjacent frames of millimeter-wave radar; The similarity coefficient of Jaccard is used to measure the similarity of cluster positions in two adjacent scan results, and a bipartite graph is constructed to obtain trajectory information. Based on the trajectory information, the trajectory with the largest variance is calculated using a preset Kuhn-Mankers algorithm and identified as the human movement trajectory. Based on the human movement trajectory, static objects within a set range of the current human location are identified, and multiple static objects with high signal reflection intensity are selected as reflectors to extract the reflected signals.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the non-line-of-sight path speech recognition method based on millimeter-wave radar as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the non-line-of-sight path speech recognition method based on millimeter-wave radar as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the non-line-of-sight path speech recognition method based on millimeter-wave radar as described in any one of claims 1 to 5.

Images