Method and system for restoring voice from facial movements on mobile phones based on deep learning

Abstract

This invention provides a method and system for restoring speech from facial movements on a mobile phone based on deep learning, relating to the field of speech processing technology. The method involves using a mobile phone's accelerometer to synchronously collect facial acceleration and speech data in real time and transmit them to a computer; receiving, aligning, storing, and denoising the data on the computer to enhance the signal-to-noise ratio; constructing a model that first converts the acceleration signal into a speech signal with a 4kHz sampling rate using a U-Net model, and then generates a speech signal with a 16kHz sampling rate using an SR-model super-resolution model; and inputting the processed facial acceleration signal into a trained deep learning model to generate speech that is transmitted on demand. This invention effectively solves existing problems, achieving speech restoration from facial movements, and has promising application prospects.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a method and system for restoring speech using facial movements on a mobile phone based on deep learning. Background Art

[0002] Accelerometers, sensors that can detect the motion of objects in three-dimensional space, are widely used in modern smartphones. They capture various physical changes in the phone, such as when it's stationary, tilted, or shaken, and convert these changes into electrical signals for further processing by the processor. Based on this acceleration data, smartphone applications can implement functions such as automatic screen rotation, step counting, and game control. This data primarily comes from the phone's built-in accelerometer, which, in its early stages of design, was primarily used to determine the phone's state and the user's posture while using it.

[0003] Due to the high level of integration required by smartphones, parameters such as the sampling rate and response speed of accelerometers were previously limited, resulting in relatively low performance. However, with the continuous advancement of sensor design, manufacturing, and integration technologies, the performance of accelerometers built into mobile phones has significantly improved. Currently, sampling rates can reach 400 Hz to 500 Hz. However, this rate is still not high enough for some advanced applications, and the acquisition process is subject to significant noise. As a result, more complex programs or software based on acceleration signals are not yet widely available in smartphones.

[0004] In the field of image processing, the IDE (Iterative Deep Enhancement) super-resolution model is a method that gradually improves image resolution through iterative enhancement. Compared to traditional single-shot upsampling methods, the IDE model can gradually improve image resolution and quality through multiple iterations. However, this model primarily focuses on processing and training two-dimensional image data, and its efficiency in feature learning for one-dimensional time series data has not been fully verified.

[0005] Meanwhile, the U-Net model has been widely used in image segmentation tasks, leveraging previously extracted shallow features during image restoration. However, it's worth noting that the U-Net model is computationally expensive when processing data. Similarly, when converting images from low-resolution to high-resolution, the computational cost and memory consumption of super-resolution models (SR models) increase with the number of iterations. While both models have achieved significant results in image processing, their effectiveness and efficiency in processing one-dimensional time series data remain underdeveloped.

[0006] In summary, despite significant progress in accelerometers and image processing technologies, there is currently no solution specifically designed to generate high-sampling-rate speech signals from low-sampling-rate acceleration signals. On smartphones in particular, effectively utilizing facial motion acceleration signals captured by accelerometers and converting them into clear speech signals remains a pressing technical challenge. Summary of the Invention

[0007] To this end, embodiments of the present invention provide a method and system for recovering speech from facial movements on a mobile phone based on deep learning, which is used to solve the problem in the prior art that the built-in accelerometer of the mobile phone cannot generate a high-sampling-rate speech signal from a low-sampling-rate facial acceleration signal due to limited sampling rate, high acquisition noise and lack of an adaptation model.

[0008] In order to solve the above problems, an embodiment of the present invention provides a method for restoring voice from facial movements on a mobile phone based on deep learning, the method comprising:

[0009] S1: Call the mobile phone's accelerometer and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time, and transmit the signal to the computer;

[0010] S2: Receive the signal on the computer and display it in real time. Use timestamps to align the acceleration signal and voice signal. Then, save the aligned signal pair locally. Design a suitable filter algorithm to remove noise from the acceleration signal and enhance the data signal-to-noise ratio.

[0011] S3: Construct a deep learning model based on the U-Net model and the SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model.

[0012] S4: The facial acceleration signal of the user speaking silently after data processing and noise removal is input into the trained deep learning model to generate the corresponding voice signal and transmit it according to the user's requirements.

[0013] Preferably, the deep learning model first restores the processed input acceleration signal to a speech signal with a sampling rate of 4 kHz through a U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through an SR-model super-resolution model, specifically including:

[0014] The input end of the U-Net model is the processed acceleration signal. The left contraction path uses a 3x3 convolution kernel with a step size of 1. Two convolution layers are used in each layer. The convolution kernel is initialized using a random normal distribution with a standard deviation of 0.01. The convolution layer is followed by a ReLU activation function, and a maximum pooling operation is performed after every two convolution layers. The pooling window is 2x2 to gradually reduce the spatial resolution and extract deep features; the right expansion path is upsampled by transposed convolution with an upsampling factor of 2. Bilinear interpolation is used, and the feature map in the contraction path is connected to the corresponding layer in the expansion path in combination with a jump connection. The weight of the jump connection is adaptively adjusted according to the feature importance during the training process to make the data size on the left and right sides equal. The output is a speech signal with a sampling rate of 4kHz. The loss function between the predicted output and the actual label is Mel-frequency cepstral distortion and signal-to-noise ratio distortion.

[0015] The generated 4kHz sampling rate speech signal is then fed into a lightweight SR super-resolution model. This model reduces the number of parameters by reducing the number of internal convolutional layers and adjusting the feature map dimensions. Adapting to the requirement of one-dimensional input and output data, it employs an iterative resolution enhancement strategy, gradually increasing the speech signal resolution by adjusting the convolution kernel parameters at each iteration to generate a final 16kHz sampling rate speech signal. The super-resolution model's loss functions are also based on Mel-frequency cepstral distortion and signal-to-noise ratio distortion.

[0016] During the training process, a joint training strategy is designed to train the U-Net model and the SR-model super-resolution model at the same time. The multi-task learning method is adopted to coordinately optimize the two models through Mel-cepstral distortion and signal-to-noise ratio distortion. In the selection of optimization algorithm, the Adam optimization algorithm is adopted for both the U-Net model and the SR-model super-resolution model. The initial value of the learning rate is set to 0.001, and the learning rate decays to 0.9 times of the original value every 10 epochs. Regarding the construction of the loss function, the Mel-cepstral distortion and the signal-to-noise ratio distortion are calculated as the targets in multi-task learning. For speech signals with different sampling rates, their corresponding losses are calculated respectively. For speech signals with a sampling rate of 4kHz, its Mel-cepstral distortion is calculated using M 4k Indicates that the signal-to-noise ratio distortion is expressed as S 4k Indicates that the overall loss is 4k Through the formula Loss 4k = 5×M 4k + 1×S 4k To calculate, for a speech signal with a sampling rate of 16kHz, the Mel cepstral distortion is recorded as M 16k , the signal-to-noise ratio distortion is recorded as S16k , overall loss Loss 16k The calculation formula is Loss 16k = 5×M 16k + 1×S 16k In order to comprehensively consider the loss of speech signals with different sampling rates, the overall loss Loss of deep learning model training is defined, which is the overall loss Loss of speech signals with a sampling rate of 4kHz. 4k and the overall loss of the 16kHz sampling rate speech signal Loss 16k The specific formula is Loss = w1×Loss 4k + w2×Loss 16k , where w1 and w2 are adjustable weight coefficients.

[0017] Preferably, the Mel-cepstral distortion calculation process is: first pre-emphasize the speech signal with a pre-emphasis coefficient of 0.97, then obtain the spectrum through short-time Fourier transform, and then convert the linear frequency into the Mel frequency scale. The conversion formula of the Mel frequency scale is: Mel (f) = 2595 × log10 (1 + f / 700), and calculate the mean square error between the predicted speech and the real speech on the Mel spectrum as the Mel-cepstral distortion, where Mel (f) represents the Mel frequency scale and f represents the Hertz frequency.

[0018] Preferably, the signal-to-noise ratio distortion is calculated as follows: based on signal power and noise power, the signal power is calculated by the average value of the square of the speech signal amplitude, and the noise power is estimated by the average value of the square of the noise signal amplitude. The logarithm of the ratio of the two is used as the signal-to-noise ratio distortion Mel loss, and the calculation formula is SNR loss = 10 × log10 (Psignal / Pnoise), where Psignal represents signal power and Pnoise represents noise power.

[0019] Preferably, the method of removing noise from the acceleration signal by designing a suitable filter algorithm is:

[0020] A wavelet transform-based denoising algorithm is used to remove noise from the acceleration signal. During the wavelet decomposition process, the Daubechies wavelet basis function is selected, and the decomposition level is 4. The wavelet coefficients are thresholded to remove coefficients less than the threshold. The threshold is determined using a soft threshold method, and the threshold value is adaptively adjusted according to the noise variance.

[0021] The embodiment of the present invention further provides a mobile phone end facial action voice restoration system based on deep learning, which is used to implement the above-mentioned mobile phone end facial action voice restoration method based on deep learning, specifically comprising:

[0022] The data acquisition module is used to call the mobile phone's acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer;

[0023] The data processing module is used to receive signals on the computer and display them in real time. It uses timestamps to align the acceleration signal and voice signal in time, then saves the aligned signal pairs locally. It also removes noise from the acceleration signal by designing a suitable filter algorithm to enhance the data signal-to-noise ratio.

[0024] A model building and training module is used to build a deep learning model based on the U-Net model and the SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model;

[0025] The speech generation module is used to input the facial acceleration signal of the user speaking silently after data processing to remove noise into the trained deep learning model, generate the corresponding speech signal and transmit it according to the user's requirements.

[0026] Preferably, the U-Net model in the model construction and training module includes a skip connection optimization unit, which calculates the correlation between the corresponding feature maps in the contraction path and the expansion path, and dynamically adjusts the weight of the skip connection according to the correlation to optimize the feature fusion effect.

[0027] Preferably, the SR-model super-resolution model in the model construction and training module includes a resolution enhancement unit, which adjusts the frequency response characteristics of the convolution kernel according to the spectral characteristics of the speech signal during each iteration to gradually improve the resolution of the speech signal.

[0028] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned deep learning-based mobile phone facial movement voice recovery method.

[0029] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned deep learning-based mobile phone facial movement voice recovery method.

[0030] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0031] (1) Effective data processing and signal conversion: A reasonable acquisition and preprocessing process solves the problem of low sampling rate and high noise of mobile phone accelerometers, and enhances the data signal-to-noise ratio. A unique deep learning model realizes the conversion of low sampling rate facial acceleration signals to high sampling rate voice signals, which makes up for the shortcomings of existing technologies.

[0032] (2) Excellent model performance: The combination and optimization of U-Net and SR-model super-resolution models, including parameter setting, lightweight design and training strategy adjustment, make its adaptability and accuracy in facial movement speech recovery scenarios far exceed traditional models, effectively improving the overall performance of the model.

[0033] (3) The system is practical and stable: It is easy to operate. Users only need to hold the phone close to their face and speak silently to achieve voice recovery and transmission. It can also monitor the phone status in real time and automatically adjust to ensure stable operation. It can be widely used in various silent voice transmission scenarios, expanding the functions of mobile phones. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. Those skilled in the art can derive other drawings based on these drawings without inventive effort. Among them:

[0035] Figure 1 This is a flowchart of a method for restoring voice using facial movements on a mobile phone based on deep learning provided in an embodiment;

[0036] Figure 2 A schematic diagram of the structure of a deep learning model in an embodiment;

[0037] Figure 3 Schematic diagram of the structure of the U-Net model in the embodiment;

[0038] Figure 4 This is a block diagram of a deep learning-based mobile phone facial movement voice recovery system provided in an embodiment. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention. Example 1

[0040] In order to solve the problem that the existing mobile phone built-in accelerometer cannot generate high sampling rate voice signal from low sampling rate facial acceleration signal due to limited sampling rate, high sampling noise and lack of adaptation model. Figure 1 As shown, an embodiment of the present invention proposes a method for restoring voice from facial movements on a mobile phone based on deep learning, the method comprising:

[0041] S1: Call the mobile phone's accelerometer and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time, and transmit the signal to the computer;

[0042] S2: Receive the signal on the computer and display it in real time. Use timestamps to align the acceleration signal and voice signal. Then, save the aligned signal pair locally. Design a suitable filter algorithm to remove noise from the acceleration signal and enhance the data signal-to-noise ratio.

[0043] S3: Construct a deep learning model based on the U-Net model and the SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model.

[0044] S4: The facial acceleration signal of the user speaking silently after data processing and noise removal is input into the trained deep learning model to generate the corresponding voice signal and transmit it according to the user's requirements.

[0045] As can be seen from the above technical solution, the present invention proposes a deep learning-based method for recovering speech from facial movements on mobile phones. First, by using the mobile phone's accelerometer to set an appropriate sampling rate, facial acceleration and speech signals are synchronously collected and transmitted to a computer, ensuring comprehensive and timely data acquisition. On the computer, the signals are precisely aligned using timestamps and stored, and a specially designed filter algorithm is applied to remove noise from the acceleration signal, effectively improving the data's signal-to-noise ratio and laying a good foundation for subsequent processing. Next, a deep learning architecture is constructed that integrates a U-Net model with an SR super-resolution model. The U-Net model converts the processed acceleration signal into a 4kHz speech signal to fully extract features, and the SR super-resolution model further upscales it to a 16kHz speech signal, achieving high-resolution speech output. This solution has significant advantages, successfully overcoming the challenges of low sampling rate, high noise, and the lack of an adaptive model in mobile phones' built-in accelerometers. It can efficiently generate high-sampled-rate speech signals from low-sampled-rate facial acceleration signals, providing a reliable technical means for transmitting silent speech and possessing great potential for application in specialized communication scenarios.

[0046] In this embodiment, a mobile phone software program is first written to call the mobile phone acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time, and transmit the data to the computer through the socket communication principle.

[0047] Specifically, develop a mobile app, utilize the sensor interface provided by the phone's operating system, write code to access the phone's built-in accelerometer, and set an appropriate sampling rate (e.g., 400 Hz). During the acquisition process, simultaneously enable the voice signal acquisition function to ensure real-time and synchronous collection of facial acceleration data and voice signals. Establish a stable socket communication connection to transmit the collected data to the computer in real time. For example, on Android, use the SensorManager class to obtain an accelerometer instance, use the AudioRecord class to collect voice signals, and then use the Socket class to establish a TCP connection to send the data to the computer.

[0048] In this embodiment, the signal is received on the computer and displayed in real time. The acceleration signal and the voice signal are temporally aligned using timestamps. The aligned signal pairs are then saved locally. A suitable filter algorithm is designed to remove noise from the acceleration signal and enhance the data signal-to-noise ratio.

[0049] Specifically, on the computer side, a receiving server program was written using MATLAB. After receiving data from the mobile phone, the data was first parsed and stored. Then, based on the timestamp information in the data, the acceleration signal and voice signal were accurately aligned in time. To address the noise issue in the acceleration signal, a suitable filter algorithm was designed and implemented. For example, a wavelet transform-based denoising algorithm was used to remove noise from the acceleration signal. During the wavelet decomposition process, this wavelet transform-based denoising algorithm selected the Daubechies wavelet basis function with a decomposition layer of four. The wavelet coefficients were thresholded to remove coefficients below the threshold. The threshold was determined using a soft thresholding method, and the threshold value was adaptively adjusted based on the noise variance.

[0050] In this embodiment, a deep learning model based on the U-Net model and the SR-model super-resolution model is constructed. The deep learning model first restores the processed input acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model. Figure 2 shown.

[0051] Furthermore, the U-Net model is Figure 3 As shown in the figure, the U-Net model input is the processed acceleration signal. The contracting path on the left uses a 3x3 convolution kernel with a stride of 1. Two convolution layers are used in each layer. The convolution kernels are initialized using a random normal distribution with a standard deviation of 0.01. The convolution layers are followed by a ReLU activation function, and after every two convolution layers, a maximum pooling operation is performed with a 2x2 pooling window to gradually reduce the spatial resolution and extract deep features. The dilation path on the right is upsampled by a transposed convolution with an upsampling factor of 2 and bilinear interpolation. Skip connections are used to connect the feature maps in the contracting path with the corresponding layers in the dilation path. The weights of the skip connections are adaptively adjusted during training based on feature importance to ensure that the data sizes on the left and right sides are equal. The output is a speech signal with a sampling rate of 4kHz. The loss function between the predicted output and the actual label is the Mel-frequency cepstral distortion and the signal-to-noise ratio distortion.

[0052] The generated 4kHz sampling rate speech signal is then fed into a lightweight SR super-resolution model. This model reduces the number of parameters by reducing the number of internal convolutional layers and adjusting the feature map dimensions. Adapting to the requirement of one-dimensional input and output data, it employs an iterative resolution enhancement strategy, gradually increasing the speech signal resolution by adjusting the convolution kernel parameters at each iteration, ultimately generating a speech signal with a 16kHz sampling rate. The super-resolution model's loss functions are also based on Mel-frequency cepstral distortion and signal-to-noise ratio distortion.

[0053] During the training process, the present invention designs a joint training strategy to simultaneously train the U-Net model and the SR-model super-resolution model. In this approach, a multi-task learning approach is adopted to collaboratively optimize the two models through two objective values.

[0054] Specifically, the Adam optimization algorithm was used for both the U-Net model and the SR-model super-resolution model. It offers numerous advantages, including the ability to adaptively adjust the learning rate based on the first- and second-order moment estimates of the gradient, helping the model converge more efficiently.

[0055] The learning rate is initially set to 0.001. To ensure the stability and effectiveness of the training process, the learning rate is dynamically adjusted according to a specific rule: after every 10 epochs, the learning rate decays to 0.9 times its current value. This adjustment allows the model to quickly explore the parameter space in the early stages of training, while more finely optimizing parameters in the later stages, avoiding excessive oscillation around the optimal solution and improving the model's generalization ability.

[0056] Regarding the construction of the loss function, this paper focuses on the two indicators of Mel-frequency cepstral distortion and signal-to-noise ratio distortion, and calculates them as the objectives in multi-task learning. For speech signals with different sampling rates, the corresponding losses are calculated respectively. For speech signals with a sampling rate of 4kHz, its Mel-frequency cepstral distortion is expressed as M 4k Indicates that the signal-to-noise ratio distortion is expressed as S 4k Indicates that the overall loss is 4k By the formula Loss4k = 5×M 4k + 1×S 4k This formula shows that the emphasis on Mel-frequency cepstral distortion is relatively high at this sampling rate. For a speech signal with a sampling rate of 16kHz, the Mel-frequency cepstral distortion is recorded as M 16k , the signal-to-noise ratio distortion is recorded as S 16k , overall loss Loss 16k The calculation formula is Loss16k = 5×M 16k +1×S 16k , which also reflects the emphasis on Mel cepstral distortion.

[0057] Finally, in order to comprehensively consider the loss of speech signals with different sampling rates, the present invention defines the overall loss Loss of deep learning model training, which is the overall loss Loss of 4kHz sampling rate speech signal 4k and the overall loss of the 16kHz sampling rate speech signal Loss 16k The specific formula is Loss = w1×Loss 4k +w2×Loss 16k Here, w1 and w2 are adjustable weight coefficients. By adjusting the values ​​of w1 and w2, we can flexibly assign different importance to the loss of speech signals with different sampling rates according to specific training requirements. This allows the model to develop towards the desired performance optimization during training, thereby improving the overall model performance in related tasks. For example, in tasks such as recovering speech from facial movements, it can better handle speech signals with different sampling rates, improving the quality and accuracy of speech recovery.

[0058] Furthermore, the above-mentioned Mel-cepstral distortion calculation process is as follows: first, the speech signal is pre-emphasized with a pre-emphasis coefficient of 0.97, and then a spectrum is obtained by short-time Fourier transform. Then, the linear frequency is converted to the Mel-frequency scale. The conversion formula of the Mel-frequency scale is: Mel (f) = 2595 × log10 (1 + f / 700). The mean square error between the predicted speech and the true speech on the Mel-frequency spectrum is calculated as the Mel-cepstral distortion, where Mel (f) represents the Mel-frequency scale and f represents the Hertz frequency.

[0059] The signal-to-noise ratio (SNR) distortion is calculated based on signal power and noise power. Signal power is calculated by averaging the squared amplitudes of the speech signal, while noise power is estimated by averaging the squared amplitudes of the noise signal. The logarithm of the ratio of the two is used as the SNR distortion (Mel loss). The calculation formula is SNR loss = 10 × log10 (Psignal / Pnoise), where Psignal represents signal power and Pnoise represents noise power.

[0060] In actual use, the user opens the mobile app, holds the phone close to their face, and begins speaking silently. The mobile software collects facial acceleration signals in real time, de-noises them, and then feeds them into a trained deep learning model. Based on the learned mapping, the model quickly generates the corresponding speech signal. The user can pre-set the voice signal transmission method and target device in the mobile software. For example, if a Bluetooth headset is paired, the user can choose to transmit the voice signal to the headset via Bluetooth, achieving silent speech output. Example 2

[0061] like Figure 4 As shown, the present invention provides a mobile phone facial action voice restoration system based on deep learning, which is used to implement the mobile phone facial action voice restoration method based on deep learning in the above embodiment 1, specifically comprising:

[0062] The data acquisition module is used to call the mobile phone's acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer;

[0063] The data processing module is used to receive signals on the computer and display them in real time. It uses timestamps to align the acceleration signal and voice signal in time, then saves the aligned signal pairs locally. It also removes noise from the acceleration signal by designing a suitable filter algorithm to enhance the data signal-to-noise ratio.

[0064] A model building and training module is used to build a deep learning model based on the U-Net model and the SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model;

[0065] The speech generation module is used to input the facial acceleration signal of the user speaking silently after data processing to remove noise into the trained deep learning model, generate the corresponding speech signal and transmit it according to the user's requirements.

[0066] Furthermore, the U-Net model in the model construction and training module includes a skip connection optimization unit, which calculates the correlation between the corresponding feature maps in the contraction path and the expansion path, and dynamically adjusts the weight of the skip connection according to the size of the correlation to optimize the feature fusion effect.

[0067] Furthermore, the SR-model super-resolution model in the model construction and training module includes a resolution enhancement unit, which adjusts the frequency response characteristics of the convolution kernel according to the spectral characteristics of the speech signal during each iteration to gradually improve the resolution of the speech signal.

[0068] A mobile phone facial movement recovery voice system based on deep learning in this embodiment is used to implement the aforementioned mobile phone facial movement recovery voice method based on deep learning. Therefore, the specific implementation method of the mobile phone facial movement recovery voice system based on deep learning can be seen in the embodiment part of the mobile phone facial movement recovery voice method based on deep learning above. For example, the data acquisition module 100, the data processing module 200, the model construction and training module 300, and the voice generation module 400 are respectively used to implement steps S1, S2, S3, and S4 in the above-mentioned mobile phone facial movement recovery voice method based on deep learning. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here. Example 3

[0069] An embodiment of the present invention provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned deep learning-based mobile phone facial movement voice recovery method. Example 4

[0070] An embodiment of the present invention provides a computer storage medium storing a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned deep learning-based mobile phone facial movement voice recovery method.

[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0074] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications derived therefrom remain within the scope of protection of the present invention.

Claims

1. A method for restoring voice from facial movements on a mobile phone based on deep learning, characterized in that: include: S1: Call the mobile phone's accelerometer and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time, and transmit the signal to the computer; S2: Receive the signal on the computer and display it in real time. Use timestamps to align the acceleration signal and voice signal. Then, save the aligned signal pair locally. Design a suitable filter algorithm to remove noise from the acceleration signal and enhance the data signal-to-noise ratio. S3: Build a deep learning model based on the U-Net model and the SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model. S4: In actual use, the user opens the mobile application, holds the phone close to the face, and starts speaking silently. The facial acceleration signal of the user speaking silently, which has been processed to remove noise, is input into the trained deep learning model to generate the corresponding voice signal and transmit it according to the user's requirements.

2. The method for restoring voice from facial movements on a mobile phone based on deep learning according to claim 1, characterized in that: The deep learning model first restores the processed input acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model, specifically including: The input end of the U-Net model is the processed acceleration signal. The left contraction path adopts a 3x3 convolution kernel with a step size of 1. Two convolution layers are used in each layer. The convolution kernel is initialized using a random normal distribution with a standard deviation of 0.

01. The convolution layer is followed by a ReLU activation function, and a maximum pooling operation is performed after each two convolution layers. The pooling window is 2x2 to gradually reduce the spatial resolution and extract deep features; the right expansion path is upsampled by transposed convolution with an upsampling factor of 2. Bilinear interpolation is used, and the feature map in the contraction path is connected to the corresponding layer in the expansion path in combination with a jump connection. The weight of the jump connection is adaptively adjusted according to the feature importance during the training process to make the data size on the left and right sides equal. The output is a speech signal with a sampling rate of 4kHz. The loss function between the predicted output and the actual label is Mel-frequency cepstral distortion and signal-to-noise ratio distortion. The generated 4kHz sampling rate speech signal is then fed into a lightweight SR-model super-resolution model. This model reduces the number of parameters by reducing the number of internal convolutional layers and adjusting the feature map dimensions. This model also meets the requirement of one-dimensional input and output data and adopts an iterative resolution improvement strategy. Each iteration gradually increases the speech signal resolution by adjusting the convolution kernel parameters to generate the final 16kHz sampling rate speech signal. The loss function of the super-resolution model is also Mel-frequency cepstral distortion and signal-to-noise ratio distortion. During the training process, a joint training strategy is designed to train the U-Net model and the SR-model super-resolution model at the same time. The multi-task learning method is used to coordinately optimize the two models through Mel-cepstral distortion and signal-to-noise ratio distortion. In terms of the selection of optimization algorithms, the Adam optimization algorithm is used for both the U-Net model and the SR-model super-resolution model. The initial value of the learning rate is set to 0.001, and the learning rate decays to 0.9 times the original value every 10 epochs. Regarding the construction of the loss function, the Mel-cepstral distortion and the signal-to-noise ratio distortion are calculated as the targets in multi-task learning. For speech signals with different sampling rates, their corresponding losses are calculated respectively. For speech signals with a sampling rate of 4kHz, its Mel-cepstral distortion is calculated using M 4k Indicates that the signal-to-noise ratio distortion is expressed as S 4k Indicates that the overall loss is 4k Through the formula Loss 4k =5×M 4k +1×S 4k To calculate, for a speech signal with a sampling rate of 16kHz, the Mel cepstral distortion is recorded as M 16k , the signal-to-noise ratio distortion is recorded as S 16k , overall loss Loss 16k The calculation formula is Loss 16k =5×M 16k +1×S 16k In order to comprehensively consider the loss of speech signals with different sampling rates, the overall loss Loss of deep learning model training is defined, which is the overall loss Loss of speech signals with a sampling rate of 4kHz. 4k and the overall loss of the 16kHz sampling rate speech signal 16k The specific formula is Loss = w1 × Loss 4k +w2×Loss 16k , where w1 and w2 are adjustable weight coefficients.

3. The method for restoring voice from facial movements on a mobile phone based on deep learning according to claim 2, characterized in that: The Mel-frequency cepstrum distortion calculation process is as follows: first, the speech signal is pre-emphasized with a pre-emphasis coefficient of 0.97, then the spectrum is obtained by short-time Fourier transform, and then the linear frequency is converted to the Mel frequency scale. The conversion formula of the Mel frequency scale is: Mel(f) = 2595× log10(1+f / 700) is used to calculate the mean square error between the predicted speech and the true speech on the Mel spectrum as the Mel cepstrum distortion, where Mel(f) represents the Mel frequency scale and f represents the Hertz frequency.

4. The method for restoring voice from facial movements on a mobile phone based on deep learning according to claim 2, characterized in that: The signal-to-noise ratio distortion is calculated as follows: based on signal power and noise power, the signal power is calculated by the average value of the square of the speech signal amplitude, and the noise power is estimated by the average value of the square of the noise signal amplitude. The logarithm of the ratio of the two is used as the signal-to-noise ratio distortion Melloss. The calculation formula is SNRloss=10×log10(P signal / P noise ), where P signal Indicates signal power, P noise Represents the noise power.

5. The method for restoring voice from facial movements on a mobile phone based on deep learning according to claim 1, characterized in that: The method for removing noise from the acceleration signal by designing a suitable filter algorithm is: A denoising algorithm based on wavelet transform is used to remove noise from the acceleration signal. In the wavelet decomposition process, the Daubechies wavelet basis function is selected, and the decomposition layer number is 4. The wavelet coefficients are thresholded to remove coefficients less than the threshold. The threshold is determined by the soft threshold method, and the threshold value is adaptively adjusted according to the noise variance.

6. A mobile phone facial action voice recovery system based on deep learning, characterized by: The system is used to implement the deep learning-based mobile phone facial action voice recovery method according to any one of claims 1 to 5, specifically comprising: The data acquisition module is used to call the mobile phone's acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration signal and voice signal in real time and transmit the signal to the computer; The data processing module is used to receive signals on the computer and display them in real time. It uses timestamps to align the acceleration signal and voice signal in time, then saves the aligned signal pairs locally. It also removes noise from the acceleration signal by designing a suitable filter algorithm to enhance the data signal-to-noise ratio. The model construction and training module is used to build a deep learning model based on the U-Net model and the SR-model super-resolution model. The deep learning model first restores the input processed acceleration signal to a speech signal with a sampling rate of 4 kHz through the U-Net model, and then converts the speech signal with a sampling rate of 4 kHz to a speech signal with a sampling rate of 16 kHz through the SR-model super-resolution model. The speech generation module is used to input the facial acceleration signal of the user speaking silently after data processing to remove noise into the trained deep learning model, generate the corresponding speech signal and transmit it according to the user's requirements.

7. The deep learning-based mobile phone facial action voice recovery system according to claim 6, characterized in that: The U-Net model in the model construction and training module includes a skip connection optimization unit, which calculates the correlation between the corresponding feature maps in the contraction path and the expansion path, and dynamically adjusts the weight of the skip connection according to the size of the correlation to optimize the feature fusion effect.

8. The deep learning-based mobile phone facial action voice recovery system according to claim 6, characterized in that: The SR-model super-resolution model in the model construction and training module includes a resolution improvement unit, which adjusts the frequency response characteristics of the convolution kernel according to the spectral characteristics of the speech signal during each iteration to gradually improve the resolution of the speech signal.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the deep learning-based mobile phone facial action voice recovery method described in any one of claims 1 to 5.

10. A computer storage medium, characterized in that The computer storage medium stores a computer software product, which includes several instructions for enabling a computer device to execute the deep learning-based mobile phone facial action voice recovery method described in any one of claims 1 to 5.

Images