Deep learning-based mobile phone end face action voice recovery method and system

Abstract

The invention provides a deep learning-based mobile phone end face action voice recovery method and system, and relates to the technical field of voice processing, and the method comprises the steps: calling a mobile phone acceleration sensor to synchronously collect face acceleration and voice data in real time, and transmitting the face acceleration and voice data to a computer; receiving, aligning and storing data at the computer end, and denoising to enhance the signal-to-noise ratio; constructing a model, firstly converting the acceleration signal into a voice signal with a sampling rate of 4khz by a U-Net model, and then generating a voice signal with a sampling rate of 16khz by an SR-model super-division model; and inputting the processed face acceleration signal into the trained deep learning model, generating voice and outputting the voice as required. The method effectively solves the existing problems, realizes the recovery from the facial action to the voice, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a method and system for restoring speech using facial movements on a mobile phone based on deep learning. Background Art

[0002] Accelerometers, as sensors that can detect the motion of objects in three-dimensional space, have been widely used in modern smartphones. They can capture various physical changes of the phone, such as stillness, tilt, shaking, etc., and convert these changes into electrical signals for further processing by the processor. Based on these acceleration data, smartphone applications can achieve a variety of functions such as automatic screen rotation, step counting, and game control. These data mainly come from the built-in accelerometer of the phone, which was mainly used to judge the status of the phone and the user's posture when using the phone in the early stage of phone design.

[0003] Due to the high integration requirements of smartphones, the sampling rate, response speed and other parameters of acceleration sensors have been limited in the past, resulting in relatively low performance. However, with the continuous advancement of sensor design, manufacturing and integration technology, the performance of accelerometers built into mobile phones has been significantly improved. At present, its sampling rate can reach 400 Hz to 500 Hz. However, this sampling rate is still not high enough for some advanced applications, and there is a lot of noise in the acquisition process. Therefore, more complex programs or software based on acceleration signals have not yet been popularized in current smartphones.

[0004] In the field of image processing, the IDE (Iterative Deep Enhancement) super-resolution model is a method that gradually improves image resolution through iterative enhancement. Compared with the traditional single upsampling method, the IDE model can gradually improve the resolution and quality of the image through multiple iterations. However, this model mainly focuses on the processing and training of two-dimensional image data, and the efficiency of feature learning for one-dimensional time series data has not been fully verified.

[0005] On the other hand, the U-Net model has been widely used in image segmentation tasks, and it can utilize shallow features extracted previously in the image restoration process. However, it is worth noting that the U-Net model has a large amount of computation when processing data. Similarly, the computational cost and memory consumption of the super-resolution model (SR-model) will increase with the number of iterations during the conversion of images from low to high resolution. Although these two models have achieved remarkable results in the field of image processing, their application effect and efficiency in one-dimensional time series data processing remain to be explored.

[0006] In summary, although acceleration sensors and image processing technologies have made significant progress in their respective fields, there is currently no technical solution specifically designed to generate high-sampling rate voice signals based on low-sampling rate acceleration signals. Especially on smartphone platforms, how to effectively use the facial motion acceleration signals collected by acceleration sensors and convert them into clear voice signals is still a technical problem that needs to be solved urgently. Summary of the invention

[0007] To this end, an embodiment of the present invention provides a method and system for restoring voice using facial movements on a mobile phone based on deep learning, which is used to solve the problem in the prior art that the built-in accelerometer of the mobile phone cannot generate a high-sampling rate voice signal from a low-sampling rate facial acceleration signal due to limited sampling rate, high collection noise and lack of an adaptation model.

[0008] In order to solve the above problems, an embodiment of the present invention provides a method for restoring voice from facial movements on a mobile phone based on deep learning, the method comprising: S1: Call the mobile phone acceleration sensor and set the sampling rate to collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer; S2: Receive the signal on the computer and display it in real time. Use the timestamp to align the acceleration signal and the voice signal. Then save the aligned signal pair locally. Design a suitable filter algorithm to remove noise from the acceleration signal and enhance the data signal-to-noise ratio. S3: construct a deep learning model based on the U-Net model and the SR-model super-resolution model, wherein the deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model; S4: The facial acceleration signal of the user speaking silently after data processing and noise removal is input into the trained deep learning model to generate the corresponding voice signal and transmit it according to the user's requirements.

[0009] Preferably, the deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through a U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through an SR-model super-resolution model, specifically including: The input end of the U-Net model is the processed acceleration signal. The left contraction path uses a 3x3 convolution kernel with a step size of 1. Two convolution layers are used in each layer. The convolution kernel is initialized using a random normal distribution with a standard deviation of 0.01. The convolution layer is followed by a ReLU activation function, and a maximum pooling operation is performed after every two convolution layers. The pooling window is 2x2 to gradually reduce the spatial resolution and extract deep-level features; the right expansion path is upsampled by transposed convolution with an upsampling factor of 2, and a bilinear interpolation method is used. At the same time, the feature map in the contraction path is connected to the corresponding layer in the expansion path in combination with a jump connection. The weight of the jump connection is adaptively adjusted according to the importance of the feature during the training process to make the data size on the left and right sides equal. The output is a speech signal with a sampling rate of 4kHz, and the loss function between the predicted output and the actual label is Mel cepstral distortion and signal-to-noise ratio distortion; The generated speech signal with a sampling rate of 4k is then put into a lightweight SR-model super-resolution model. This super-resolution model reduces the number of parameters by reducing the number of internal convolutional layers and adjusting the dimension of the feature map. It also meets the requirement that the input and output are one-dimensional data. It adopts an iterative resolution improvement strategy. Each iteration gradually increases the resolution of the speech signal by adjusting the convolution kernel parameters to generate the final speech signal with a sampling rate of 16kHz. The loss function of the super-resolution model is also Mel cepstral distortion and signal-to-noise ratio distortion. During the training process, a joint training strategy is designed to train the U-Net model and the SR-model super-resolution model at the same time. The multi-task learning method is used to coordinately optimize the two models through Mel-cepstral distortion and signal-to-noise ratio distortion. In the selection of the optimization algorithm, the Adam optimization algorithm is used for both the U-Net model and the SR-model super-resolution model. The initial value of the learning rate is set to 0.001, and the learning rate decays to 0.9 times the original value every 10 epochs. Regarding the construction of the loss function, the Mel-cepstral distortion and the signal-to-noise ratio distortion are calculated as the targets in multi-task learning. For speech signals with different sampling rates, their corresponding losses are calculated respectively. For speech signals with a sampling rate of 4kHz, the Mel-cepstral distortion is calculated using Mel-cepstral distortion. 4k Indicates that the signal-to-noise ratio distortion is expressed by S 4k Indicates that the overall loss is 4k By formula Loss 4k = 5×M 4k + 1×S 4k To calculate, for a speech signal with a sampling rate of 16kHz, the Mel cepstrum distortion is recorded as M 16k , the signal-to-noise ratio distortion is recorded as S 16k , overall loss Loss16k The calculation formula is Loss 16k = 5×M 16k + 1×S 16k ; In order to comprehensively consider the loss of speech signals with different sampling rates, the overall loss Loss of deep learning model training is defined, which is the overall loss Loss of speech signals with a sampling rate of 4kHz 4k and the overall loss of the 16kHz sampling rate speech signal 16k The specific formula is Loss = w1×Loss. 4k + w2×Loss 16k , where w1 and w2 are adjustable weight coefficients.

[0010] Preferably, the Mel cepstral distortion calculation process is: first pre-emphasize the speech signal with a pre-emphasis coefficient of 0.97, then obtain the spectrum through short-time Fourier transform, and then convert the linear frequency into a Mel frequency scale. The conversion formula of the Mel frequency scale is: Mel (f) = 2595 × log10 (1 + f / 700), and calculate the mean square error between the predicted speech and the real speech on the Mel spectrum as the Mel cepstral distortion, where Mel (f) represents the Mel frequency scale and f represents the Hertz frequency.

[0011] Preferably, the signal-to-noise ratio distortion is calculated as follows: based on signal power and noise power, the signal power is calculated by the average value of the square of the speech signal amplitude, the noise power is estimated by the average value of the square of the noise signal amplitude, and the logarithm of the ratio of the two is used as the signal-to-noise ratio distortion Mel loss. The calculation formula is SNR loss = 10 × log10 (Psignal / Pnoise), where Psignal represents signal power and Pnoise represents noise power.

[0012] Preferably, the method of removing noise in the acceleration signal by designing a suitable filter algorithm is: A denoising algorithm based on wavelet transform is used to remove noise from acceleration signals. In the wavelet decomposition process, the Daubechies wavelet basis function is selected, and the decomposition layer number is 4. The wavelet coefficients are thresholded to remove coefficients less than the threshold. The threshold is determined by the soft threshold method, and the threshold value is adaptively adjusted according to the noise variance.

[0013] The embodiment of the present invention also provides a mobile phone end facial action voice restoration system based on deep learning, which is used to implement the mobile phone end facial action voice restoration method based on deep learning, and specifically includes: The data acquisition module is used to call the mobile phone acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer; The data processing module is used to receive the signal on the computer side and display it in real time. It uses the timestamp to align the acceleration signal and the voice signal in time, and then saves the aligned signal pair locally. It also removes the noise in the acceleration signal by designing a suitable filter algorithm to enhance the data signal-to-noise ratio. A model building and training module is used to build a deep learning model based on a U-Net model and an SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 khz through the U-Net model, and then converts the speech signal with a sampling rate of 4 khz to a speech signal with a sampling rate of 16 khz through the SR-model super-resolution model. The speech generation module is used to input the facial acceleration signal of the user speaking silently after data processing to remove noise into the trained deep learning model, generate the corresponding speech signal and transmit it according to the user's requirements.

[0014] Preferably, the U-Net model in the model construction and training module includes a skip connection optimization unit, which calculates the correlation between the corresponding feature maps in the contraction path and the expansion path, and dynamically adjusts the weight of the skip connection according to the correlation to optimize the feature fusion effect.

[0015] Preferably, the SR-model super-resolution model in the model construction and training module includes a resolution enhancement unit, which adjusts the frequency response characteristics of the convolution kernel according to the spectral characteristics of the speech signal during each iteration to gradually improve the resolution of the speech signal.

[0016] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned deep learning-based mobile phone facial movement voice recovery method.

[0017] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes a number of instructions for enabling a computer device to execute the above-mentioned deep learning-based mobile phone facial movement voice recovery method.

[0018] It can be seen from the above technical solutions that the present invention has the following beneficial effects: (1) Effective data processing and signal conversion: Reasonable acquisition and preprocessing processes solve the problem of low sampling rate and high noise of mobile phone accelerometers, and enhance the data signal-to-noise ratio. The unique deep learning model realizes the conversion of low sampling rate facial acceleration signals to high sampling rate voice signals, making up for the defects of existing technologies.

[0019] (2) Excellent model performance: The combination and optimization of U-Net and SR-model super-resolution models, including parameter setting, lightweight design and training strategy adjustment, make its adaptability and accuracy in facial movement speech recovery scenarios far exceed traditional models, effectively improving the overall performance of the model.

[0020] (3) The system is practical and stable: It is easy to operate. Users only need to hold the phone close to their face and speak silently to achieve voice recovery and transmission. It can also monitor the phone status in real time and automatically adjust to ensure stable operation. It can be widely used in a variety of silent voice transmission scenarios, expanding the functions of mobile phones. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 A flowchart of a method for restoring voice using facial movements on a mobile phone based on deep learning provided in an embodiment; Figure 2 It is a structural diagram of a deep learning model in an embodiment; Figure 3 Schematic diagram of the structure of the U-Net model in the embodiment; Figure 4 A block diagram of a deep learning-based mobile phone facial movement voice recovery system provided in an embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Embodiment 1

[0023] In order to solve the problem that the built-in accelerometer of the mobile phone in the prior art cannot generate a high-sampling-rate voice signal from a low-sampling-rate facial acceleration signal due to limited sampling rate, high collection noise and lack of adaptation model. Figure 1 As shown, the embodiment of the present invention proposes a method for restoring voice from facial movements on a mobile phone based on deep learning, the method comprising: S1: Call the mobile phone acceleration sensor and set the sampling rate to collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer; S2: Receive the signal on the computer and display it in real time. Use the timestamp to align the acceleration signal and the voice signal. Then save the aligned signal pair locally. Design a suitable filter algorithm to remove noise from the acceleration signal and enhance the data signal-to-noise ratio. S3: construct a deep learning model based on the U-Net model and the SR-model super-resolution model, wherein the deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model; S4: The facial acceleration signal of the user speaking silently after data processing and noise removal is input into the trained deep learning model to generate the corresponding voice signal and transmit it according to the user's requirements.

[0024] It can be seen from the above technical scheme that the present invention proposes a method for restoring voice from facial movements on a mobile phone based on deep learning. First, by calling the mobile phone acceleration sensor to set a suitable sampling rate, the facial acceleration and voice signals are synchronously collected and transmitted to the computer to ensure the comprehensiveness and timeliness of data acquisition. On the computer side, the signal is stored after accurate alignment using the timestamp, and the acceleration signal noise is removed using a specially designed filter algorithm, which effectively improves the data signal-to-noise ratio and lays a good foundation for subsequent processing. Then, a deep learning architecture integrating the U-Net model and the SR-model super-resolution model is constructed. The U-Net model can convert the processed acceleration signal into a 4kHz voice signal, fully extract features, and the SR-model super-resolution model further increases it to a 16kHz voice signal to achieve high-resolution voice output. The advantages of this scheme are significant. It successfully overcomes the problems of low sampling rate, high noise and no adaptation model of the built-in accelerometer of the mobile phone. It can efficiently generate high-sampling rate voice signals from low-sampling rate facial acceleration signals, providing a reliable technical means for silent voice transmission, and has great application potential in special communication scenarios.

[0025] In this embodiment, a mobile phone software program is firstly written to call the mobile phone acceleration sensor and set the sampling rate, collect the user's facial acceleration signal and voice signal in real time and synchronously, and transmit the data to the computer through the socket communication principle.

[0026] Specifically, develop a mobile phone application, use the sensor interface provided by the mobile phone operating system, write code to call the built-in accelerometer of the mobile phone, and set an appropriate sampling rate (such as 400hz). During the acquisition process, start the voice signal acquisition function at the same time to ensure the real-time synchronous acquisition of facial acceleration data and voice signals. By establishing a stable socket communication connection, the collected data is transmitted to the computer in real time. For example, in the Android system, use the SensorManager class to obtain the accelerometer instance, and use the AudioRecord class to collect voice signals, and then use the Socket class to establish a TCP connection to send the data to the computer.

[0027] In this embodiment, the signal is received on the computer and displayed in real time. The acceleration signal and the voice signal are aligned in time through timestamps, and then the aligned signal pairs are saved locally. The noise in the acceleration signal is removed by designing a suitable filter algorithm to enhance the data signal-to-noise ratio.

[0028] Specifically, on the computer side, Matlab is used to write the receiving end server program. After receiving the data from the mobile phone, the data is first parsed and stored, and then the acceleration signal and the voice signal are accurately aligned in time according to the timestamp information in the data. In view of the noise problem in the acceleration signal, a suitable filter algorithm is designed and implemented. For example, a denoising algorithm based on wavelet transform is used to remove the noise in the acceleration signal. In the wavelet decomposition process, the denoising algorithm based on wavelet transform selects the Daubechies wavelet basis function, and the decomposition layer number is 4. By thresholding the wavelet coefficients, the coefficients less than the threshold are removed. The threshold is determined by the soft threshold method, and the threshold size is adaptively adjusted according to the noise variance.

[0029] In this embodiment, a deep learning model based on a U-Net model and an SR-model super-resolution model is constructed. The deep learning model first restores the processed input acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model. Figure 2 shown.

[0030] Furthermore, the U-Net model is Figure 3As shown in the figure, the input of the U-Net model is the processed acceleration signal. The left contraction path uses a 3x3 convolution kernel with a step size of 1. Two convolution layers are used in each layer. The convolution kernel is initialized with a random normal distribution with a standard deviation of 0.01. The convolution layer is followed by a ReLU activation function, and a maximum pooling operation is performed after every two convolution layers. The pooling window is 2x2 to gradually reduce the spatial resolution and extract deep features. The right expansion path is upsampled by transposed convolution with an upsampling factor of 2. Bilinear interpolation is used. At the same time, the feature map in the contraction path is connected to the corresponding layer in the expansion path by combining jump connections. The weights of the jump connections are adaptively adjusted according to the importance of the features during the training process to make the data size on the left and right sides equal. The output is a speech signal with a sampling rate of 4kHz. The loss function between the predicted output and the actual label is Mel cepstral distortion and signal-to-noise ratio distortion.

[0031] Furthermore, the generated speech signal with a sampling rate of 4k is put into a lightweight SR-model super-resolution model. This super-resolution model reduces the number of parameters by reducing the number of internal convolutional layers and adjusting the dimension of the feature map. It also meets the requirement that the input and output are one-dimensional data. It adopts an iterative resolution improvement strategy. Each iteration gradually increases the resolution of the speech signal by adjusting the convolution kernel parameters to generate the final speech signal with a sampling rate of 16kHz. The loss function of the super-resolution model is also Mel cepstral distortion and signal-to-noise ratio distortion.

[0032] During the training process, the present invention designs a joint training strategy to simultaneously train the U-Net model and the SR-model super-resolution model, wherein a multi-task learning approach is adopted to collaboratively optimize the two models through two target values.

[0033] Specifically, in terms of the selection of optimization algorithms, both the U-Net model and the SR-model super-resolution model use the Adam optimization algorithm, which has many advantages and can adaptively adjust the learning rate based on the first-order moment estimation and second-order moment estimation of the gradient, which helps the model converge more efficiently.

[0034] The initial value of the learning rate is set to 0.001. In order to ensure the stability and effectiveness of the training process, the learning rate will be dynamically adjusted according to certain rules, that is, after every 10 epochs, the learning rate will decay to 0.9 times the current value. This adjustment method allows the model to quickly explore the parameter space in the early stage of training, and optimize the parameters more finely in the later stage to avoid excessive oscillation near the optimal solution, thereby improving the generalization ability of the model.

[0035] Regarding the construction of the loss function, the present invention focuses on the two indicators of Mel-cepstral distortion and signal-to-noise ratio distortion, and calculates them as the objectives in multi-task learning. For speech signals with different sampling rates, the corresponding losses are calculated respectively. For a speech signal with a sampling rate of 4kHz, its Mel-cepstral distortion is expressed as M 4k Indicates that the signal-to-noise ratio distortion is expressed by S 4k Indicates that the overall loss is 4k By the formula Loss4k = 5×M 4k + 1×S 4k This formula shows that the emphasis on Mel cepstral distortion is relatively high at this sampling rate. For a speech signal with a sampling rate of 16kHz, the Mel cepstral distortion is recorded as M 16k , the signal-to-noise ratio distortion is recorded as S 16k , overall loss Loss 16k The calculation formula is Loss 16k = 5×M 16k +1×S 16k , which also reflects the emphasis on Mel's cepstral distortion.

[0036] Finally, in order to comprehensively consider the loss of speech signals with different sampling rates, the present invention defines the overall loss Loss of deep learning model training, which is the overall loss Loss of speech signals with a sampling rate of 4kHz 4k and the overall loss of the 16kHz sampling rate speech signal 16k The specific formula is Loss = w1×Loss. 4k +w2×Loss 16k , where w1 and w2 are adjustable weight coefficients. By adjusting the values ​​of w1 and w2, different importance can be flexibly assigned to the loss of speech signals with different sampling rates according to specific training requirements, so that the model can develop in the desired performance optimization direction during the training process, thereby improving the performance of the overall model in related tasks, such as better processing speech signals with different sampling rates in tasks such as recovering speech from facial movements, and improving the quality and accuracy of speech recovery.

[0037] Furthermore, the above-mentioned Mel cepstrum distortion calculation process is: first pre-emphasize the speech signal with a pre-emphasis coefficient of 0.97, then obtain the spectrum through short-time Fourier transform, and then convert the linear frequency into the Mel frequency scale. The conversion formula of the Mel frequency scale is: Mel (f) = 2595 × log10 (1 + f / 700), and calculate the mean square error between the predicted speech and the real speech on the Mel spectrum as the Mel cepstrum distortion, where Mel (f) represents the Mel frequency scale and f represents the Hertz frequency.

[0038] The signal-to-noise ratio distortion is calculated based on signal power and noise power. The signal power is calculated by the average value of the square of the speech signal amplitude, and the noise power is estimated by the average value of the square of the noise signal amplitude. The logarithm of the ratio of the two is used as the signal-to-noise ratio distortion Mel loss. The calculation formula is SNR loss = 10 × log10 (Psignal / Pnoise), where Psignal represents signal power and Pnoise represents noise power.

[0039] In this embodiment, in actual use, the user opens the mobile phone application, puts the phone close to the face, and starts to speak silently. The mobile phone software collects the facial acceleration signal in real time, and after denoising, inputs it into the trained deep learning model. The model quickly generates the corresponding voice signal based on the learned mapping relationship. The user can pre-set the transmission method and target device of the voice signal in the mobile phone software. For example, if the Bluetooth headset is paired, choose to transmit the voice signal to the Bluetooth headset via Bluetooth to achieve the output of silent voice. Embodiment 2

[0040] like Figure 4 As shown, the present invention provides a mobile phone facial action voice restoration system based on deep learning, which is used to implement the mobile phone facial action voice restoration method based on deep learning in the above embodiment 1, and specifically includes: The data acquisition module is used to call the mobile phone acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer; The data processing module is used to receive the signal on the computer side and display it in real time. It uses the timestamp to align the acceleration signal and the voice signal in time, and then saves the aligned signal pair locally. It also removes the noise in the acceleration signal by designing a suitable filter algorithm to enhance the data signal-to-noise ratio. A model building and training module is used to build a deep learning model based on a U-Net model and an SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 khz through the U-Net model, and then converts the speech signal with a sampling rate of 4 khz to a speech signal with a sampling rate of 16 khz through the SR-model super-resolution model. The speech generation module is used to input the facial acceleration signal of the user speaking silently after data processing to remove noise into the trained deep learning model, generate the corresponding speech signal and transmit it according to the user's requirements.

[0041] Furthermore, the U-Net model in the model construction and training module includes a skip connection optimization unit, which calculates the correlation between the corresponding feature maps in the contraction path and the expansion path, and dynamically adjusts the weight of the skip connection according to the size of the correlation to optimize the feature fusion effect.

[0042] Furthermore, the SR-model super-resolution model in the model construction and training module includes a resolution enhancement unit, which adjusts the frequency response characteristics of the convolution kernel according to the spectral characteristics of the speech signal during each iteration to gradually improve the resolution of the speech signal.

[0043] A mobile phone facial movement recovery voice system based on deep learning in this embodiment is used to implement the aforementioned mobile phone facial movement recovery voice method based on deep learning. Therefore, the specific implementation method of the mobile phone facial movement recovery voice system based on deep learning can be seen in the embodiment part of the mobile phone facial movement recovery voice method based on deep learning in the previous text. For example, the data acquisition module 100, the data processing module 200, the model building and training module 300, and the voice generation module 400 are respectively used to implement steps S1, S2, S3, and S4 in the aforementioned mobile phone facial movement recovery voice method based on deep learning. Therefore, its specific implementation method can refer to the description of the corresponding each part of the embodiment. In order to avoid redundancy, it will not be repeated here. Embodiment 3

[0044] An embodiment of the present invention provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned deep learning-based mobile phone facial movement voice recovery method. Embodiment 4

[0045] An embodiment of the present invention provides a computer storage medium, which stores a computer software product. The computer software product includes a number of instructions for enabling a computer device to execute the above-mentioned method for restoring voice through facial movements on a mobile phone based on deep learning.

[0046] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0047] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0048] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0049] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A method for restoring speech from facial movements on a mobile phone based on deep learning, characterized in that: include: S1: Call the mobile phone acceleration sensor and set the sampling rate to collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer; S2: Receive the signal on the computer and display it in real time. Use the timestamp to align the acceleration signal and the voice signal. Then save the aligned signal pair locally. Design a suitable filter algorithm to remove noise from the acceleration signal and enhance the data signal-to-noise ratio. S3: construct a deep learning model based on the U-Net model and the SR-model super-resolution model, wherein the deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model; S4: The facial acceleration signal of the user speaking silently after data processing and noise removal is input into the trained deep learning model to generate the corresponding voice signal and transmit it according to the user's requirements.

2. The method for restoring voice from facial movements on a mobile phone based on deep learning according to claim 1, characterized in that: The deep learning model first restores the processed input acceleration signal to a speech signal with a sampling rate of 4 kHz through a U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through an SR-model super-resolution model, specifically including: The input end of the U-Net model is the processed acceleration signal. The left contraction path uses a 3x3 convolution kernel with a step size of 1. Two convolution layers are used in each layer. The convolution kernel is initialized using a random normal distribution with a standard deviation of 0.

01. The convolution layer is followed by a ReLU activation function, and a maximum pooling operation is performed after every two convolution layers. The pooling window is 2x2 to gradually reduce the spatial resolution and extract deep-level features; the right expansion path is upsampled by transposed convolution with an upsampling factor of 2, and a bilinear interpolation method is used. At the same time, the feature map in the contraction path is connected to the corresponding layer in the expansion path in combination with a jump connection. The weight of the jump connection is adaptively adjusted according to the importance of the feature during the training process to make the data size on the left and right sides equal. The output is a speech signal with a sampling rate of 4kHz, and the loss function between the predicted output and the actual label is Mel cepstral distortion and signal-to-noise ratio distortion; The generated speech signal with a sampling rate of 4k is then put into a lightweight SR-model super-resolution model. This super-resolution model reduces the number of parameters by reducing the number of internal convolutional layers and adjusting the dimension of the feature map. It also meets the requirement that the input and output are one-dimensional data. It adopts an iterative resolution improvement strategy. Each iteration gradually increases the resolution of the speech signal by adjusting the convolution kernel parameters to generate the final speech signal with a sampling rate of 16kHz. The loss function of the super-resolution model is also Mel cepstral distortion and signal-to-noise ratio distortion. During the training process, a joint training strategy is designed to train the U-Net model and the SR-model super-resolution model at the same time. The multi-task learning method is used to coordinately optimize the two models through Mel-cepstral distortion and signal-to-noise ratio distortion. In the selection of the optimization algorithm, the Adam optimization algorithm is used for both the U-Net model and the SR-model super-resolution model. The initial value of the learning rate is set to 0.001, and the learning rate decays to 0.9 times the original value every 10 epochs. Regarding the construction of the loss function, the Mel-cepstral distortion and the signal-to-noise ratio distortion are calculated as the targets in multi-task learning. For speech signals with different sampling rates, their corresponding losses are calculated respectively. For speech signals with a sampling rate of 4kHz, the Mel-cepstral distortion is calculated using Mel-cepstral distortion. 4k Indicates that the signal-to-noise ratio distortion is expressed by S 4k Indicates that the overall loss is 4k Through the formula Loss 4k = 5×M 4k + 1×S 4k To calculate, for a speech signal with a sampling rate of 16kHz, the Mel cepstrum distortion is recorded as M 16k , the signal-to-noise ratio distortion is recorded as S 16k , overall loss Loss 16k The calculation formula is Loss 16k = 5×M 16k + 1×S 16k ; In order to comprehensively consider the loss of speech signals with different sampling rates, the overall loss Loss of deep learning model training is defined, which is the overall loss Loss of speech signals with a sampling rate of 4kHz 4k and the overall loss of the 16kHz sampling rate speech signal 16k The specific formula is Loss = w1×Loss. 4k + w2×Loss 16k , where w1 and w2 are adjustable weight coefficients.

3. The method for restoring voice from facial movements on a mobile phone based on deep learning according to claim 2, characterized in that: The Mel cepstral distortion calculation process is as follows: firstly, the speech signal is pre-emphasized with a pre-emphasis coefficient of 0.97, then a spectrum is obtained by short-time Fourier transform, and then the linear frequency is converted into a Mel frequency scale. The conversion formula of the Mel frequency scale is: Mel (f) = 2595 × log10 (1 + f / 700), and the mean square error between the predicted speech and the real speech on the Mel spectrum is calculated as the Mel cepstral distortion, where Mel (f) represents the Mel frequency scale and f represents the Hertz frequency.

4. The method for restoring voice from facial movements on a mobile phone based on deep learning according to claim 2, characterized in that: The signal-to-noise ratio distortion is calculated as follows: based on signal power and noise power, the signal power is calculated by the average value of the square of the speech signal amplitude, and the noise power is estimated by the average value of the square of the noise signal amplitude. The logarithm of the ratio of the two is used as the signal-to-noise ratio distortion Mel loss, and the calculation formula is SNR loss = 10×log10 (P signal / P noise ), where P signal Represents the signal power, P noise Represents the noise power.

5. The method for restoring voice from facial movements on a mobile phone based on deep learning according to claim 1, characterized in that: The method for removing noise in the acceleration signal by designing a suitable filter algorithm is: A denoising algorithm based on wavelet transform is used to remove noise from acceleration signals. In the wavelet decomposition process, the Daubechies wavelet basis function is selected, and the decomposition layer number is 4. The wavelet coefficients are thresholded to remove coefficients less than the threshold. The threshold is determined by the soft threshold method, and the threshold value is adaptively adjusted according to the noise variance.

6. A mobile phone facial action voice recovery system based on deep learning, characterized in that: The system is used to implement the method for restoring voice based on facial movements on a mobile phone based on deep learning according to any one of claims 1 to 5, and specifically comprises: The data acquisition module is used to call the mobile phone acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer; The data processing module is used to receive the signal on the computer side and display it in real time. It uses the timestamp to align the acceleration signal and the voice signal in time, and then saves the aligned signal pair locally. It also removes the noise in the acceleration signal by designing a suitable filter algorithm to enhance the data signal-to-noise ratio. A model building and training module is used to build a deep learning model based on a U-Net model and an SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 khz through the U-Net model, and then converts the speech signal with a sampling rate of 4 khz to a speech signal with a sampling rate of 16 khz through the SR-model super-resolution model. The speech generation module is used to input the facial acceleration signal of the user speaking silently after data processing to remove noise into the trained deep learning model, generate the corresponding speech signal and transmit it according to the user's requirements.

7. The deep learning-based mobile phone facial action voice recovery system according to claim 6, characterized in that: The U-Net model in the model construction and training module includes a skip connection optimization unit, which calculates the correlation between the corresponding feature maps in the contraction path and the expansion path, and dynamically adjusts the weight of the skip connection according to the correlation to optimize the feature fusion effect.

8. The deep learning-based mobile phone facial action voice recovery system according to claim 6, characterized in that: The SR-model super-resolution model in the model construction and training module includes a resolution enhancement unit, which adjusts the frequency response characteristics of the convolution kernel according to the spectral characteristics of the speech signal during each iteration to gradually improve the resolution of the speech signal.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the deep learning-based mobile phone facial action voice recovery method as described in any one of claims 1 to 5.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, which includes a number of instructions for enabling a computer device to execute the deep learning-based mobile phone facial action voice restoration method as described in any one of claims 1 to 5.

Images