Speech synthesis method and system based on speech radar and deep learning

Abstract

The present invention discloses a speech synthesis method and system based on speech radar and deep learning, the method comprising: obtaining vocal cord vibration echo signals by radar; obtaining fundamental waves by mode decomposition of radar echo signals, and then performing center clipping on the fundamental waves as speech excitation sources; training a resonance peak estimation model by a deep learning method using a video of the lip area of ​​a speaker when speaking; obtaining resonance peak parameters by using a video of the speaker's pronunciation as input; and finally performing speech synthesis based on the radar echo excitation source model and resonance peak parameters. The method of the present invention can obtain a speech excitation source that retains the continuous variation characteristics of the fundamental frequency and a more accurate resonance peak parameter estimation, and can obtain a more natural synthesized speech through a speech synthesizer.

Description

Technical Field

[0001] The present invention belongs to the field of radar technology, and in particular to a speech synthesis method and system based on speech radar and deep learning. Background Art

[0002] Speech synthesis has always been one of the hottest research topics in the field of speech. Bioradar has been proven to be useful for acquiring and synthesizing speech signals, but the quality of speech still needs to be improved. The quality of synthesized speech is related to the selection of the synthesis model and the accuracy of the parameters.

[0003] The accuracy of the speech excitation source is closely related to the quality of synthesized speech. Most existing studies use fundamental frequency to generate pulses to generate the excitation source. The excitation source generated by this method has a strong regularity and loses the characteristic of continuous change of fundamental frequency during the vocalization process. Therefore, the speech synthesized using this method sounds mechanical. More parameters are needed to obtain more natural speech.

[0004] In recent years, with the popularization of artificial intelligence technology, deep learning technology has been widely used in the field of image processing and has achieved some results in the field of speech recognition. However, there has been no research on applying deep learning technology to video processing and obtaining relevant features of speech signals. The existing technology obtains the resonance peak frequency by fitting empirical formulas with image features, but the process is cumbersome and the results have large errors. Summary of the invention

[0005] The purpose of the present invention is to provide a speech synthesis method and system based on speech radar and deep learning to address the problems existing in the above-mentioned prior art.

[0006] The technical solution to achieve the purpose of the present invention is: a speech synthesis method based on speech radar and deep learning, comprising the following steps:

[0007] Step 1, obtaining a vocal cord vibration echo signal from a radar, and obtaining a speech synthesis excitation source based on the echo signal;

[0008] Step 2, extracting lip area video and corresponding audio from the video of the speaker's pronunciation, selecting M consecutive video frames from the speaker's mouth opening to closing as a package input, and extracting the resonance peak parameters of the corresponding audio, inputting multiple videos and using the corresponding resonance peak parameters as labels, and using a deep learning method to train the mapping relationship between lip movement features and resonance peaks to obtain a resonance peak parameter estimation model;

[0009] Step 3, using the lip area video information of the speaker being tested as the input of the formant parameter estimation model to obtain N groups of formant parameters of the speaker's speech;

[0010] Step 4, performing speech synthesis on the speech synthesis excitation source obtained in step 1 and the N groups of formant parameters obtained in step 3 to obtain a highly natural synthesized speech signal.

[0011] Furthermore, the step 1 of obtaining the voice excitation source according to the radar echo signal specifically includes:

[0012] Step 1-1, the voice radar sends a sinusoidal continuous wave, and the receiving antenna receives an echo signal containing vocal cord vibration information;

[0013] Step 1-2, high-pass filtering the radar echo signal to remove the low-frequency DC component and body motion information to obtain a signal S;

[0014] Step 1-3, use the empirical wavelet transform algorithm to decompose the signal S, and the kth mode f after decomposition k (t) is expressed as:

[0015]

[0016] In the formula, is the coefficient of the empirical wavelet transform of the kth mode, ψ k (t) is the k-th mode empirical wavelet function. According to the synthesis requirements, the first mode f is selected. 1 (t), obtain the fundamental wave X = f 1 (t);

[0017] Step 1-4, according to the synthesis requirements, divide the fundamental wave into m frames, perform center clipping on the signal X(i) of the i-th frame, and obtain the speech synthesis excitation source; wherein the formula of center clipping is:

[0018]

[0019] Where C is the clipping level.

[0020] Furthermore, the deep learning method in step 2 adopts an improved 3D convolutional neural network, which includes a point-like convolution filter of size 1×1×1 set in sequence, followed by three parallel convolution kernels of size 1×3×3, a convolution kernel of size 3×1×1 and a convolution kernel of size 3×3×3, and finally a fully connected layer.

[0021] Furthermore, the specific process of step 2 includes:

[0022] Step 2-1, recording the voice video, and then performing lip area segmentation to obtain the lip area video;

[0023] Step 2-2, extracting the speech signal corresponding to the speaker's pronunciation recorded by the microphone in the video, and extracting N groups of time-varying formant parameters, each group including n formant parameters;

[0024] Step 2-3, construct a resonance peak parameter estimation model based on the improved 3D convolutional neural network, specifically:

[0025] Based on step 2-1, multiple lip area video data are obtained as input of the 3D convolutional neural network, and N groups of resonance peaks are used as outputs of the corresponding lip area videos in the 3D convolutional neural network;

[0026] The lip area video data is first convolved with a point-wise convolution filter of size 1×1×1, and then divided into three groups. The spatial features are extracted by convolution with a convolution kernel of size 1×3×3, the temporal features are extracted by convolution with a convolution kernel of size 3×1×1, and the spatiotemporal features are extracted by convolution with a convolution kernel of size 3×3×3. Then, the spatiotemporal features of the lip area sounding video are fused by channel shuffling, and finally, N groups of resonance peaks are output through a fully connected layer.

[0027] Step 2-4, training the resonance peak parameter estimation model, specifically: the neural network uses the Adam optimizer, uses the poly learning rate strategy to update the learning rate, and uses the RMSE function as the loss function. The losses of different labels are integrated and the final loss function is calculated as: L = (L 1 +L 2 +…+L N ) / N, where L is the comprehensive loss of the model, L 1 , L 2 ,…,L N They are the losses of the labels corresponding to N groups of resonance peaks.

[0028] A speech synthesis system based on speech radar and deep learning, the system comprising:

[0029] The first module is used to obtain a vocal cord vibration echo signal from a radar and obtain a speech synthesis excitation source based on the echo signal;

[0030] The second module is used to extract the lip area video and the corresponding audio from the video of the speaker's pronunciation, select M consecutive video frames from the speaker's mouth opening to closing as a package input, and extract the resonance peak parameters of the corresponding audio, input multiple videos, and use the corresponding resonance peak parameters as labels, and use the deep learning method to train the mapping relationship between lip movement features and resonance peaks to obtain a resonance peak parameter estimation model;

[0031] The third module is used to use the lip area video information of the speaker being tested as the input of the formant parameter estimation model to obtain N groups of formant parameters of the speaker's voice;

[0032] The fourth module is used to perform speech synthesis on the speech synthesis excitation source obtained by the first module and the N groups of formant parameters obtained by the third module to obtain a highly natural synthesized speech signal.

[0033] Compared with the prior art, the present invention has the following significant advantages: 1) The present invention is a speech synthesis for the personalized characteristics of the pronunciation subject, and the length, timbre and other characteristics of the speech change with the pronunciation subject, which has greater flexibility; 2) The fundamental frequency component is extracted from the radar signal and directly used as the excitation source after clipping processing, which retains the continuity of the fundamental frequency change, makes the speech synthesis sound smoother, and reduces the impact of errors in the fundamental frequency extraction process on the speech quality; 3) The mapping relationship between lip movement characteristics and resonance peaks is trained by a deep learning method, which improves the accuracy and robustness of resonance peak frequency estimation. The model generated by using a large amount of data can reduce the problems of false peaks, resonance peak merging and the like that often occur in traditional resonance peak frequency estimation methods; 4) Compared with the speech restored by using the fundamental frequency to generate the excitation source in the art, the speech synthesized by the present invention has higher fluency and naturalness.

[0034] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The present invention is a flow chart of a high-quality speech synthesis method based on a radar echo excitation source model.

[0036] Figure 2 The present invention is a flow chart of the resonance peak parameter estimation model trained by the lip area video during pronunciation.

[0037] Figure 3 This is a network structure diagram for training the resonance peak parameter model of the present invention.

[0038] Figure 4 A flow chart of obtaining resonance peak parameters for the resonance peak parameter estimation model of the present invention.

[0039] Figure 5 Schematic diagram of an excitation source model based on radar echo in an embodiment, wherein FIG. (a) is a frame of fundamental wave waveform, and FIG. (b) is the excitation source waveform after center clipping.

[0040] Figure 6 This is a comparison diagram of the formant parameters of the speech “E” calculated by the formant parameter estimation model in one embodiment and the formant parameters obtained by the microphone audio.

[0041] Figure 7 The figures show the synthesized speech waveform of the synthesized speech "E" and the original speech waveform synchronously recorded by a microphone in one embodiment, wherein Figure (a) is the synthesized speech waveform and Figure (b) is the original speech waveform recorded by the microphone. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] In addition, if there are descriptions involving "first", "second", etc. in the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0044] Combination Figure 1 , provides a speech synthesis method based on speech radar and deep learning, the method comprising the following steps:

[0045] Step 1: obtain the vocal cord vibration echo signal from the radar, and obtain the speech synthesis excitation source according to the echo signal; the specific process includes:

[0046] Step 1-1, the voice radar sends a sinusoidal continuous wave, and the receiving antenna receives an echo signal containing vocal cord vibration information;

[0047] Step 1-2, high-pass filtering the radar echo signal to remove the low-frequency DC component and body motion information to obtain a signal S;

[0048] Step 1-3, use the empirical wavelet transform algorithm to decompose the signal S, and the kth mode f after decomposition k (t) is expressed as:

[0049]

[0050] In the formula, is the coefficient of the empirical wavelet transform of the kth mode, ψ k (t) is the k-th mode empirical wavelet function. According to the synthesis requirements, the first mode f is selected. 1 (t), obtain the fundamental wave X = f 1 (t);

[0051] Step 1-4, according to the synthesis requirements, divide the fundamental wave into m frames, perform center clipping on the signal X(i) of the i-th frame, and obtain the speech synthesis excitation source; wherein the formula of center clipping is:

[0052]

[0053] Where C is the clipping level.

[0054] Step 2, extracting lip area video and corresponding audio from the video of the speaker's pronunciation, selecting M consecutive video frames from the speaker's mouth opening to closing as a package input, and extracting the resonance peak parameters of the corresponding audio, inputting multiple videos and using the corresponding resonance peak parameters as labels, and using a deep learning method to train the mapping relationship between lip movement features and resonance peaks to obtain a resonance peak parameter estimation model;

[0055] Here, the deep learning method adopts an improved 3D convolutional neural network, which includes a point-wise convolution filter of size 1×1×1 set in sequence, followed by three parallel convolution kernels of size 1×3×3, a convolution kernel of size 3×1×1 and a convolution kernel of size 3×3×3, and finally a fully connected layer.

[0056] Combination Figure 2 , the step specifically includes:

[0057] Step 2-1, recording the sound video, and then performing lip area segmentation to obtain the lip area video; the specific process includes:

[0058] Step 2-1-1, record the video of the speaker pronouncing, and use the dlib face recognition module to obtain 20 lip contour feature points;

[0059] Step 2-1-2, segment the lip area according to the lip contour feature points to obtain a lip area image of size H×W, where H is the height of the image and W is the width of the image;

[0060] Step 2-1-3, select M consecutive frames of images from the speaker's opening to closing mouth to form lip area video data of size M×H×W.

[0061] Step 2-2, extracting the speech signal corresponding to the speaker's pronunciation recorded by the microphone in the video, and extracting N groups of time-varying formant parameters, each group including n formant parameters;

[0062] Step 2-3, build a resonance peak parameter estimation model based on the improved 3D convolutional neural network, combined with Figure 3 and Figure 4 , specifically:

[0063] Based on step 2-1, multiple lip area video data are obtained as input of the 3D convolutional neural network, and N groups of resonance peaks are used as outputs of the corresponding lip area videos in the 3D convolutional neural network;

[0064] The lip area video data is first convolved with a point-wise convolution filter of size 1×1×1, and then divided into three groups. The spatial features are extracted by convolution with a convolution kernel of size 1×3×3, the temporal features are extracted by convolution with a convolution kernel of size 3×1×1, and the spatiotemporal features are extracted by convolution with a convolution kernel of size 3×3×3. Then, the spatiotemporal features of the lip area sounding video are fused by channel shuffling, and finally, N groups of resonance peaks are output through a fully connected layer.

[0065] Step 2-4, training the resonance peak parameter estimation model, specifically: the neural network uses the Adam optimizer, uses the poly learning rate strategy to update the learning rate, and uses the RMSE function as the loss function. The losses of different labels are integrated and the final loss function is calculated as: L = (L 1 +L 2 +…+L N ) / N, where L is the comprehensive loss of the model, L 1 , L 2 ,…,L N They are the losses of the labels corresponding to N groups of resonance peaks.

[0066] Step 3, using the lip area video information of the speaker being tested as the input of the formant parameter estimation model to obtain N groups of formant parameters of the speaker's speech;

[0067] Step 4, performing speech synthesis on the speech synthesis excitation source obtained in step 1 and the N groups of formant parameters obtained in step 3 to obtain a highly natural synthesized speech signal.

[0068] The present invention proposes a speech synthesis system based on speech radar and deep learning, the system comprising:

[0069] The first module is used to obtain a vocal cord vibration echo signal from a radar and obtain a speech synthesis excitation source according to the echo signal; the module includes the following steps executed in sequence:

[0070] A first signal acquisition unit is used for sending a sine continuous wave by a voice radar, and a receiving antenna receives an echo signal containing vocal cord vibration information;

[0071] A second signal acquisition unit is used to perform high-pass filtering on the radar echo signal to remove low-frequency DC components and body motion information to obtain a signal S;

[0072] The mode decomposition unit is used to decompose the signal S using the empirical wavelet transform algorithm. The kth mode f after decomposition k (t) is expressed as:

[0073]

[0074] In the formula, is the coefficient of the empirical wavelet transform of the kth mode, ψ k (t) is the k-th mode empirical wavelet function. According to the synthesis requirements, the first mode f is selected. 1 (t), obtain the fundamental wave X = f 1 (t);

[0075] The speech synthesis excitation source acquisition unit is used to divide the fundamental wave into m frames according to the synthesis requirements, perform center clipping on the signal X(i) of the i-th frame, and obtain the speech synthesis excitation source; wherein the formula of center clipping is:

[0076]

[0077] Where C is the clipping level.

[0078] The second module is used to extract the lip area video and the corresponding audio from the video of the speaker's pronunciation, select M consecutive frames of video images from the speaker's mouth opening to closing as a package input, and extract the resonance peak parameters of the corresponding audio, input multiple videos, and use the corresponding resonance peak parameters as labels, and use the deep learning method to train the mapping relationship between lip movement features and resonance peaks to obtain a resonance peak parameter estimation model; this module includes the following steps executed in sequence:

[0079] The lip area video acquisition unit is used to record the sound video and then perform lip area segmentation to obtain the lip area video; the unit includes the following steps executed in sequence:

[0080] The feature point extraction subunit is used to record the video of the speaker's pronunciation and obtain the lip contour feature points using the dlib face recognition module;

[0081] The lip area segmentation subunit is used to perform lip area segmentation according to the lip contour feature points to obtain a lip area image of size H×W, where H is the height of the image and W is the width of the image;

[0082] The lip area video selection subunit is used to select M consecutive frames of images from the speaker's opening to closing of the mouth to form lip area video data of size M×H×W.

[0083] A formant parameter extraction unit is used to extract a speech signal corresponding to the speaker's pronunciation recorded by a microphone in the video, and extract N groups of time-varying formant parameters, each group including n formant parameters;

[0084] The model building unit is used to build a resonance peak parameter estimation model based on the improved 3D convolutional neural network, specifically:

[0085] A plurality of lip area video data are obtained based on a lip area video acquisition unit as input of a 3D convolutional neural network, and N groups of resonance peak parameters are output of corresponding lip area videos in the 3D convolutional neural network;

[0086] The lip area video data is first convolved with a point-wise convolution filter of size 1×1×1, and then divided into three groups. The spatial features are extracted by convolution with a convolution kernel of size 1×3×3, the temporal features are extracted by convolution with a convolution kernel of size 3×1×1, and the spatiotemporal features are extracted by convolution with a convolution kernel of size 3×3×3. Then, the spatiotemporal features of the lip area sounding video are fused by channel shuffling, and finally, N groups of resonance peak parameters are output through a fully connected layer.

[0087] The model training unit is used to train the resonance peak parameter estimation model. Specifically, the Adam optimizer is used for the neural network, the poly learning rate strategy is used to update the learning rate, and the loss function uses the RMSE function to integrate the losses of different labels. The final loss function is calculated as: L = (L 1 +L 2 +…+L N ) / N, where L is the comprehensive loss of the model, L 1 , L 2 ,…,L N They are the losses of the labels corresponding to N groups of resonance peaks.

[0088] The third module is used to use the lip area video information of the speaker being tested as the input of the formant parameter estimation model to obtain N groups of formant parameters of the speaker's voice;

[0089] The fourth module is used to perform speech synthesis on the speech synthesis excitation source obtained by the first module and the N groups of formant parameters obtained by the third module to obtain a highly natural synthesized speech signal.

[0090] For the specific limitations of the speech synthesis system based on voice radar and deep learning, please refer to the limitations of the speech synthesis method based on voice radar and deep learning above, which will not be repeated here. Each module in the above-mentioned speech synthesis system based on voice radar and deep learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0091] The present invention is further described in detail below with reference to the embodiments.

[0092] Example

[0093] This embodiment is an adult man pronouncing the English letter "I". When the speaker pronouncing "I", the radar receives the echo signal containing the vocal cord vibration information. The voice radar sends a continuous sine wave to the speaker, and the echo signal is received by the receiving antenna. The radar signal is subjected to high-pass filtering and then empirical wavelet transformation. The first mode component is selected for windowing and framing, and then the center clipping is performed as the speech synthesis excitation source. Figure 5 Figure 2 is a comparison diagram before and after center clipping, where Figure (a) is a schematic diagram of the fundamental waveform, and Figure (b) is the excitation source waveform after center clipping.

[0094] A large number of different pronunciation video clips of multiple speakers are used as input, and the formant parameters extracted from the speech signal synchronously obtained by the microphone are used as labels. The deep learning method is used for training to obtain a model that estimates 4 sets of formant parameters through lip movement. The lip area video of the speaker to be synthesized is used as input to obtain 4 sets of formant parameters corresponding to the speaker's voice, such as Figure 6 As shown in the figure, it can be seen that the method of the present invention can estimate the formant frequency more accurately and avoid the problem of formant merging. Finally, the radar echo is used as the excitation source and the four groups of formant parameters estimated by the video are used for speech synthesis to obtain the synthesized speech of "I", as shown in FIG. Figure 7 As shown in the figure, it can be seen that the method of the present invention can be used to better restore the voice signal from the radar echo signal.

[0095] The method of the present invention relies on the radar echo excitation source model and the formant parameter estimation model based on deep learning to achieve high-quality speech synthesis. The method extracts the fundamental frequency component from the radar signal and directly uses it as the excitation source after clipping, which retains the continuity of the fundamental frequency change, making the speech synthesis sound smoother. At the same time, the formant parameter estimation model is obtained through the deep learning method, which improves the accuracy and robustness of the formant estimation.

[0096] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A speech synthesis method based on speech radar and deep learning, characterized in that: The method comprises the following steps: Step 1, obtaining a vocal cord vibration echo signal from a radar, and obtaining a speech synthesis excitation source based on the echo signal; Step 2, extracting lip area video and corresponding audio from the video of the speaker's pronunciation, selecting M consecutive video frames from the speaker's mouth opening to closing as a package input, and extracting the resonance peak parameters of the corresponding audio, inputting multiple videos and using the corresponding resonance peak parameters as labels, and using a deep learning method to train the mapping relationship between lip movement features and resonance peaks to obtain a resonance peak parameter estimation model; Step 3, using the lip area video information of the speaker being tested as the input of the formant parameter estimation model to obtain N groups of formant parameters of the speaker's speech; Step 4, performing speech synthesis on the speech synthesis excitation source obtained in step 1 and the N groups of formant parameters obtained in step 3 to obtain a highly natural synthesized speech signal; The step 1 of obtaining the voice excitation source according to the radar echo signal includes: Step 1-1, the voice radar sends a sinusoidal continuous wave, and the receiving antenna receives an echo signal containing vocal cord vibration information; Step 1-2, high-pass filtering the radar echo signal to remove the low-frequency DC component and body motion information to obtain a signal S; Step 1-3, use the empirical wavelet transform algorithm to decompose the signal S, and the kth mode f after decomposition k (t) is expressed as: In the formula, is the coefficient of the empirical wavelet transform of the kth mode, ψ k (t) is the k-th mode empirical wavelet subfunction. According to the synthesis requirements, the first mode f1(t) is selected to obtain the fundamental wave X=f1(t); Step 1-4, according to the synthesis requirements, divide the fundamental wave into m frames, perform center clipping on the signal X(i) of the i-th frame, and obtain the speech synthesis excitation source; wherein the formula of center clipping is: Where C is the clipping level.

2. The speech synthesis method based on speech radar and deep learning according to claim 1, characterized in that: In step 2, the deep learning method adopts an improved 3D convolutional neural network, which includes a point-wise convolution filter of size 1×1×1 set in sequence, followed by three parallel convolution kernels of size 1×3×3, a convolution kernel of size 3×1×1, and a convolution kernel of size 3×3×3, and finally a fully connected layer.

3. The speech synthesis method based on speech radar and deep learning according to claim 2, characterized in that: The specific process of step 2 includes: Step 2-1, recording the voice video, and then performing lip area segmentation to obtain the lip area video; Step 2-2, extracting the speech signal corresponding to the speaker's pronunciation recorded by the microphone in the video, and extracting N groups of time-varying formant parameters, each group including n formant parameters; Step 2-3, construct a resonance peak parameter estimation model based on the improved 3D convolutional neural network, specifically: Based on step 2-1, multiple lip area video data are obtained as input of the 3D convolutional neural network, and N groups of resonance peak parameters are used as outputs of the corresponding lip area videos in the 3D convolutional neural network; The lip area video data is first convolved with a point-wise convolution filter of size 1×1×1, and then divided into three groups. The spatial features are extracted by convolution with a convolution kernel of size 1×3×3, the temporal features are extracted by convolution with a convolution kernel of size 3×1×1, and the spatiotemporal features are extracted by convolution with a convolution kernel of size 3×3×3. Then, the spatiotemporal features of the lip area sounding video are fused by channel shuffling, and finally, N groups of resonance peak parameters are output through a fully connected layer. Step 2-4, training the resonance peak parameter estimation model, specifically: the neural network uses the Adam optimizer, the poly learning rate strategy is used to update the learning rate, the loss function uses the RMSE function, and the losses of different labels are integrated to calculate the final loss function: L = (L1 + L2 + ... + L N ) / N, where L is the comprehensive loss of the model, L1, L2, ..., L N They are the losses of the labels corresponding to N groups of resonance peaks.

4. The speech synthesis method based on speech radar and deep learning according to claim 3 is characterized in that: The specific process of step 2-1 includes: Step 2-1-1, record the video of the speaker pronouncing, and use the dlib face recognition module to obtain the lip contour feature points; Step 2-1-2, segment the lip area according to the lip contour feature points to obtain a lip area image of size H×W, where H is the height of the image and W is the width of the image; Step 2-1-3, select M consecutive frames of images from the speaker's opening to closing mouth to form lip area video data of size M×H×W.

5. A speech synthesis system based on speech radar and deep learning that implements the method described in any one of claims 1 to 4, characterized in that: The system comprises: The first module is used to obtain a vocal cord vibration echo signal from a radar and obtain a speech synthesis excitation source based on the echo signal; The second module is used to extract the lip area video and the corresponding audio from the video of the speaker's pronunciation, select M consecutive video frames from the speaker's mouth opening to closing as a package input, and extract the resonance peak parameters of the corresponding audio, input multiple videos, and use the corresponding resonance peak parameters as labels, and use the deep learning method to train the mapping relationship between lip movement features and resonance peaks to obtain a resonance peak parameter estimation model; The third module is used to use the lip area video information of the speaker being tested as the input of the formant parameter estimation model to obtain N groups of formant parameters of the speaker's speech; The fourth module is used to perform speech synthesis on the speech synthesis excitation source obtained by the first module and the N groups of formant parameters obtained by the third module to obtain a highly natural synthesized speech signal.

6. The speech synthesis system based on speech radar and deep learning according to claim 5, characterized in that: The first module includes the following steps executed in sequence: A first signal acquisition unit is used for sending a sine continuous wave by a voice radar, and a receiving antenna receives an echo signal containing vocal cord vibration information; A second signal acquisition unit is used to perform high-pass filtering on the radar echo signal to remove low-frequency DC components and body motion information to obtain a signal S; The mode decomposition unit is used to decompose the signal S using the empirical wavelet transform algorithm. The kth mode f after decomposition k (t) is expressed as: In the formula, is the coefficient of the empirical wavelet transform of the kth mode, ψ k (t) is the k-th mode empirical wavelet subfunction. According to the synthesis requirements, the first mode f1(t) is selected to obtain the fundamental wave X=f1(t); The speech synthesis excitation source acquisition unit is used to divide the fundamental wave into m frames according to the synthesis requirements, perform center clipping on the signal X(i) of the i-th frame, and obtain the speech synthesis excitation source; wherein the formula of center clipping is: Where C is the clipping level.

7. The speech synthesis system based on speech radar and deep learning according to claim 6, characterized in that: The deep learning method adopts an improved 3D convolutional neural network, which includes a point-like convolution filter of size 1×1×1 set in sequence, followed by three parallel convolution kernels of size 1×3×3, a convolution kernel of size 3×1×1 and a convolution kernel of size 3×3×3, and finally a fully connected layer.

8. The speech synthesis system based on speech radar and deep learning according to claim 7, characterized in that: The second module includes the following steps: A lip area video acquisition unit is used to record the sound video and then perform lip area segmentation to obtain the lip area video; A formant parameter extraction unit is used to extract a speech signal corresponding to the speaker's pronunciation recorded by a microphone in the video, and extract N groups of time-varying formant parameters, each group including n formant parameters; The model building unit is used to build a resonance peak parameter estimation model based on the improved 3D convolutional neural network, specifically: A plurality of lip area video data are obtained based on a lip area video acquisition unit as input of a 3D convolutional neural network, and N groups of resonance peak parameters are output of corresponding lip area videos in the 3D convolutional neural network; The lip area video data is first convolved with a point-wise convolution filter of size 1×1×1, and then divided into three groups. The spatial features are extracted by convolution with a convolution kernel of size 1×3×3, the temporal features are extracted by convolution with a convolution kernel of size 3×1×1, and the spatiotemporal features are extracted by convolution with a convolution kernel of size 3×3×3. Then, the spatiotemporal features of the lip area sounding video are fused by channel shuffling, and finally, N groups of resonance peak parameters are output through a fully connected layer. The model training unit is used to train the resonance peak parameter estimation model. Specifically, the Adam optimizer is used for the neural network, the poly learning rate strategy is used to update the learning rate, the RMSE function is used for the loss function, and the losses of different labels are integrated to calculate the final loss function: L = (L1 + L2 + ... + L N ) / N, where L is the comprehensive loss of the model, L1, L2, ..., L N They are the losses of the labels corresponding to N groups of resonance peaks.

9. The speech synthesis system based on speech radar and deep learning according to claim 8, characterized in that: The lip area video acquisition unit includes the following steps executed in sequence: The feature point extraction subunit is used to record the video of the speaker's pronunciation and obtain the lip contour feature points using the dlib face recognition module; The lip area segmentation subunit is used to perform lip area segmentation according to the lip contour feature points to obtain a lip area image of size H×W, where H is the height of the image and W is the width of the image; The lip area video selection subunit is used to select M consecutive frames of images from the speaker's opening to closing of the mouth to form lip area video data of size M×H×W.

Images