Mobile phone end face action recovery voice system based on convolution and attention mechanism

Abstract

The invention provides a mobile phone end face action voice recovery system based on convolution and an attention mechanism, and relates to the technical field of voice processing, and the system comprises a data acquisition module, a data processing module, a model training module and a voice generation module. The data acquisition module calls a mobile phone acceleration sensor to synchronously acquire facial acceleration and voice data in real time and transmit the facial acceleration and voice data to the computer; the data processing module is used for receiving, aligning and storing data at a computer end and de-noising the data to enhance the signal-to-noise ratio; the model training module constructs a model, the acceleration data is converted into a Mel language spectrogram through a convolutional neural network attention model, and then voice is generated through a vocoder; and the voice generation module inputs the processed facial acceleration data into the trained model, generates voice and transmits the voice as required. The system effectively solves the existing problems, realizes the recovery from the facial action to the voice, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a mobile phone facial action speech recovery system based on a convolution plus attention mechanism. Background Art

[0002] In the tide of modern technology, smartphones have become one of the core devices in people's lives, and the development of their built-in accelerometer technology has also gone through several stages. In the early days, accelerometers were mainly used to realize basic functions of mobile phones, such as controlling screen rotation by judging the status of the phone, counting steps, and providing input for game operations. However, due to the highly integrated design requirements of mobile phones at that time, key performance of accelerometers such as sampling rate and response speed were limited, making it difficult to meet more complex application requirements.

[0003] With the continuous advancement of technology, the performance of built-in accelerometers in mobile phones has been improved, and the sampling rate can reach 400-500hz. However, in actual application scenarios, there are still many challenges. On the one hand, for applications that require high-precision acceleration signal analysis, the sampling rate is still insufficient. For example, in some scenarios that require high-precision capture of motion details, it is impossible to provide sufficiently accurate data support. On the other hand, the acceleration signal acquisition process is easily interfered by external factors. Gravity acceleration, human activities, and the vibration of the device itself will generate noise, which seriously affects the signal quality, thereby restricting the development of applications based on acceleration signals, making it difficult to run more complex related programs or software in mobile phones.

[0004] At the same time, deep learning technology has flourished, setting off a wave of technological change in many fields. Deep learning models have opened up new paths for data processing with their advanced network architectures, such as Transformer and its variants. Taking Vision Transformer (ViT) as an example, its innovative image segmentation processing method has achieved comparable or even better results than traditional convolutional neural networks (CNN) in the field of image recognition. Residual networks (ResNet) and their improved versions have promoted the rapid development of image recognition and other fields by solving the problem of deep network training. These deep learning models trained with large amounts of data play a key role in autonomous driving, face recognition, security detection and other fields, achieving a leap in intelligence in various fields.

[0005] When deep learning models are applied to speech signal processing, traditional convolutional neural networks (CNNs) introduce an attention mechanism to enhance the focus on important information, among which the channel attention mechanism can automatically assign channel weights and highlight important channel information. However, traditional CNNs still have obvious shortcomings when processing speech signals, especially in converting low-sampling rate signals (such as the 400hz signal of a mobile phone accelerometer) into high-sampling rate signals (such as the 22050hz signal required for high-quality audio). Its computing power and learning capabilities are not up to the task. Although the vocoder can convert digitized speech signals into audible sounds, it relies on accurate sound feature input, which makes the accurate conversion from low-sampling rate acceleration signals to high-sampling rate speech signals a difficult problem in current technology. There is a lack of deep learning models designed specifically for this purpose to achieve the mapping of low-sampling rate one-dimensional time series signals to high-sampling rate time series signals.

[0006] In summary, the current technology development faces many difficulties in optimizing the performance of smartphone accelerometers, applying deep learning models to speech signal processing, and converting low-sampling-rate to high-sampling-rate signals. Innovative technical solutions are urgently needed to break through these bottlenecks, so as to achieve efficient and accurate speech signal recovery and processing based on the built-in accelerometer of the mobile phone, and provide stronger technical support for future applications such as intelligent voice interaction. Summary of the invention

[0007] To this end, an embodiment of the present invention provides a mobile phone facial movement recovery speech system based on a convolution plus attention mechanism, which is used to solve the problems in the prior art that the sampling rate of the built-in accelerometer of the mobile phone is low and the collection noise is high, there is a lack of a deep learning model specifically based on the low sampling rate acceleration signal to generate a high sampling rate speech signal, and it is difficult to capture the one-dimensional time series signal characteristics and mapping relationships.

[0008] In order to solve the above problems, an embodiment of the present invention provides a mobile phone facial action voice recovery system based on a convolution plus attention mechanism, the system comprising: The data acquisition module is used to call the mobile phone acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration data and voice data in real time and transmit the data to the computer; The data processing module is located on the computer side and is used to receive data and display it in real time. It uses timestamps to align the acceleration data and voice data in time, then saves the aligned data pairs locally, and removes noise from the acceleration data by designing a suitable filter algorithm to enhance the data signal-to-noise ratio. A model training module is used to construct a facial action recovery speech model, wherein the facial action recovery speech model first recovers the input acceleration data into a Mel spectrogram corresponding to the speech data through a pre-established convolutional neural network attention model, and then puts the Mel spectrogram into a vocoder to recover the final speech data; The speech generation module is used to input the facial acceleration data of the user speaking silently after data processing to remove noise into the trained facial movement recovery speech model, generate corresponding speech data and transmit it according to user requirements.

[0009] Preferably, the structure of the convolutional neural network attention model is: The acceleration data is used as the input layer; the first layer is composed of 5 residual shrinkage block units, which are used as specific residual structural units to perform preliminary feature extraction and noise processing on the input acceleration data; the second layer, namely the position encoding layer, is used to add position information to the data; the third layer is a multi-head self-attention mechanism layer, which is used to pay attention to different parts of the input data in different representation subspaces in parallel, and effectively capture long-distance dependencies by calculating the correlation between different positions of the data, and then deeply extract features; the fourth layer is a feedforward network layer, which is used to perform nonlinear transformation on the data processed by the self-attention mechanism to enhance the feature extraction ability of the model; the fifth layer is a layer normalization layer, which is used to normalize the input, accelerate model training and prevent gradient disappearance or explosion problems; the sixth layer is composed of 3 channel attention convolution modules, which are used to adaptively weight different channel features, highlight important channel features and suppress unimportant channel features; the output layer finally generates a Mel spectrum map.

[0010] Preferably, the parameters of the soft threshold in the residual shrinkage block unit are obtained through learning, so as to perform soft threshold processing on each channel while extracting features through convolution, and output intermediate features that remove redundant data and suppress the influence of noise.

[0011] Preferably, the structure of the residual shrinkage block unit is: Input layer: The dimension of input data is ,in Indicates the number of channels, Indicates width; First layer: batch normalization, rectified linear unit and convolution operations, convolution kernel size , the dimensions of both input and output are ; Second layer: batch normalization, rectified linear unit and convolution operations are also performed, and the convolution kernel size , the dimensions of both input and output are ; Intermediate processing: Input The data passes through a The convolution operation obtains a The intermediate variable ; Feature processing branch: output of the second layer The data is processed by absolute value and global average pooling to obtain a The eigenvector of Then it goes through the fully connected layer operation, batch normalization, rectified linear unit, fully connected layer operation, and then through the Sigmoid function to get , and finally through get ,in represents the adaptive weight adjustment parameter, is an intermediate variable; Output layer: Output the second layer Data and intermediate variables Multiply, then multiply , and finally perform residual connection with the input data to obtain the final output, whose dimension is still .

[0012] Preferably, the structure of the channel attention convolution module is: First, the input dimension is The feature data is then processed by the first layer of one-dimensional convolution operation, and its output dimension is still ; Then the second layer of one-dimensional convolution operation is performed, and the output dimension is still ,in Indicates length, Indicates the number of channels; Then, after two layers of convolution, The data is globally pooled on the length scale, resulting in a dimension of The channel length sequence of Next, a series of operations are performed on the data obtained by global pooling: first, the first full connection operation is performed, and the input dimension is , the output dimension is ; Then perform a rectified linear unit operation on it, and the input and output dimensions are ; Then the second full connection operation is performed, and the input dimension is , the output dimension is ; Finally, the Sigmoid activation operation is performed, and the input and output dimensions are ,in Represents the feature dimension after the transformation of the intermediate fully connected layer; Get the Sigmoid activated After thresholding the data, apply it to the output of the second convolution layer The result is scaled and the output dimension is ; Finally, the scaled Data and original input The data is residually connected to obtain the final output, whose dimension is .

[0013] Preferably, the vocoder is HIFIGAN, which is used to convert the Mel-spectrogram into final speech data.

[0014] Preferably, the mean square error loss function (mse) is used during model training to determine the difference between the model prediction output and the actual label.

[0015] The embodiment of the present invention also provides a method for restoring speech from facial movements on a mobile phone based on a convolution plus attention mechanism. The method adopts the above-mentioned system for restoring speech from facial movements on a mobile phone based on a convolution plus attention mechanism, and comprises the following steps: S1: Call the mobile phone acceleration sensor and set the sampling rate to collect the user's facial acceleration data and voice data in real time and transmit the data to the computer; S2: Receive data on the computer and display it in real time. Use timestamps to align the acceleration data and voice data in time. Then save the aligned data pairs locally. Design a suitable filter algorithm to remove noise from the acceleration data and enhance the data signal-to-noise ratio. S3: constructing a facial action recovery speech model, wherein the facial action recovery speech model first recovers the input acceleration data into a Mel spectrogram of the corresponding speech data through a pre-established convolutional neural network attention model, and then puts the Mel spectrogram into a vocoder to recover the final speech data; S4: The facial acceleration data of the user speaking silently after data processing and noise removal is input into the trained facial movement recovery speech model to generate corresponding speech data and transmit it according to the user's requirements.

[0016] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned mobile phone facial action speech recovery method based on convolution and attention mechanism.

[0017] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes a number of instructions for enabling a computer device to execute the above-mentioned mobile phone facial action voice recovery method based on convolution and attention mechanism.

[0018] It can be seen from the above technical solutions that the present invention has the following beneficial effects: (1) The present invention realizes the real-time synchronous collection and transmission of mobile phone acceleration and voice data, ensures accurate data association, and solves the problem of asynchronous collection. The computer side uses a filter algorithm to effectively remove the noise of acceleration data, improve the signal-to-noise ratio, and provide reliable data for model training.

[0019] (2) Construct a facial action speech restoration model that integrates the convolutional neural network attention model and the vocoder to fill the gap in the low-sampling rate to high-sampling rate speech signal model and achieve effective mapping. With the help of the convolutional neural network attention model, the one-dimensional time series signal characteristics and mapping relationship are accurately captured, the existing technical difficulties are overcome, and the accuracy of speech restoration is improved.

[0020] (3) It has the function of converting the processed facial acceleration data into voice and transmitting it on demand, meeting the needs of scenarios such as silent communication and has practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 A block diagram of a mobile phone facial action voice recovery system based on a convolution plus attention mechanism provided in an embodiment; Figure 2 This is a schematic diagram of the architecture of the facial action speech recovery model in the embodiment; Figure 3 It is a structural diagram of a convolutional neural network attention (CNN_Attention) model in an embodiment; Figure 4 Schematic diagram of the structure of a residual shrinkage block unit (RSBU) in an embodiment; Figure 5 Schematic diagram of the structure of the channel attention convolution (SE Conv) module in the embodiment; Figure 6 The present invention is a flowchart of a method for restoring speech using facial movements on a mobile phone based on a convolution and attention mechanism provided in an embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Embodiment 1

[0023] In order to solve the problems of low sampling rate and high noise of built-in accelerometers in mobile phones in the prior art, lack of deep learning models specifically based on low sampling rate acceleration signals to generate high sampling rate voice signals, and difficulty in capturing one-dimensional time series signal features and mapping relationships. Figure 1 As shown, the embodiment of the present invention proposes a mobile phone facial action voice recovery system based on convolution plus attention mechanism, the system comprising: The data acquisition module is used to call the mobile phone acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration data and voice data in real time and transmit the data to the computer; The data processing module is located on the computer side and is used to receive data and display it in real time. It uses timestamps to align the acceleration data and voice data in time, then saves the aligned data pairs locally, and removes noise from the acceleration data by designing a suitable filter algorithm to enhance the data signal-to-noise ratio. The model training module is used to build a facial action recovery speech model. The facial action recovery speech model first recovers the input acceleration data into the Mel spectrogram of the corresponding speech data through the pre-established convolutional neural network attention model, and then puts the Mel spectrogram into the vocoder to recover the final speech data; The speech generation module is used to input the facial acceleration data of the user speaking silently after data processing to remove noise into the trained facial movement recovery speech model, generate corresponding speech data and transmit it according to user requirements.

[0024] It can be seen from the above technical solution that the present invention proposes a mobile phone facial action recovery voice system based on convolution plus attention mechanism. The data acquisition module synchronously collects facial acceleration and voice data in real time and transmits them to the computer. The data processing module performs data alignment, denoising and storage on the computer to enhance the signal-to-noise ratio. The facial action recovery voice model constructed by the model training module first converts the acceleration data into a Mel-speech spectrogram using a convolutional neural network attention model, and then generates voice through a vocoder, which solves the problem of lack of a special model and difficulty in capturing features and mapping relationships. The voice generation module can generate voice from the processed facial acceleration data and transmit it on demand. The system realizes efficient and accurate data acquisition and processing. The innovative model architecture can effectively convert low sampling rate signals and has a practical voice generation function, which can meet the needs of scenarios such as silent communication, and overall improves the effect and practicality of recovering voice from facial actions.

[0025] In this embodiment, the data acquisition module is used to write a mobile phone software program, call the mobile phone acceleration sensor and set the sampling rate, collect the user's facial acceleration data (signal) and voice data (signal) in real time and synchronously, and transmit the data to the computer through the socket communication principle.

[0026] Specifically, develop a mobile phone application, use the sensor interface provided by the mobile phone operating system, write code to call the built-in accelerometer of the mobile phone, and set an appropriate sampling rate (such as 400hz). During the collection process, start the voice data collection function at the same time to ensure the real-time synchronous collection of facial acceleration data and voice data. By establishing a stable socket communication connection, the collected data is transmitted to the computer in real time. For example, in the Android system, use the SensorManager class to obtain the accelerometer instance, and use the AudioRecord class to collect voice data, and then use the Socket class to establish a TCP connection to send the data to the computer.

[0027] In this embodiment, the data processing module is located on the computer side, which is used to write a matlab program as a receiving server, receive data and draw and display it in real time, align the acceleration data and voice data in time through timestamps, and then save the aligned data pairs locally, and remove noise in the acceleration data by designing a suitable filter algorithm to enhance the data signal-to-noise ratio.

[0028] Specifically, on the computer side, use matlab to write the receiving server program. After receiving the data from the mobile phone, first parse and store the data, and then accurately align the acceleration data and voice data in time according to the timestamp information in the data. Design and implement a suitable filter algorithm to solve the noise problem in the acceleration data (signal). For example, the Kalman filter algorithm is used to estimate the true value of the signal in real time based on the dynamic characteristics of the signal and the measurement noise characteristics, effectively remove the noise such as gravity acceleration, slight shaking of the human body, and vibration of the device itself, significantly enhance the data signal-to-noise ratio, and provide high-quality data for subsequent model training.

[0029] In this embodiment, the model training module is used to construct a facial action recovery speech model. The model first recovers the input acceleration data into the Mel spectrogram of the corresponding speech data through the pre-established convolutional neural network attention (CNN_Attention) model, and then puts the Mel spectrogram into the vocoder (HIFIGAN) to recover the final speech data. Figure 2 shown.

[0030] Furthermore, the overall architecture of the convolutional neural network attention (CNN_Attention) model established by the present invention is as follows: Figure 3 As shown in Figure 2. The main purpose of this model is to convert the input acceleration data into a Mel-spectrogram. The model consists of multiple modules that process the input data in sequence and gradually extract and convert features.

[0031] The structure of the CNN_Attention model is as follows: Input layer: The acceleration data is used as the input layer, which is the raw data processed by the entire model.

[0032] The first layer is composed of 5 residual shrinkage block units (RSBU). These residual shrinkage block units, as specific residual structural units, can perform preliminary feature extraction and noise processing on the input acceleration data.

[0033] Second layer: After being processed by 5 residual shrinkage block units, the data enters the positional encoding layer. The role of the positional encoding layer is to add position information to the data, which is crucial for the subsequent self-attention mechanism to process the data, because the self-attention mechanism needs to know the order and relative position of the data to better capture the relationship between the data.

[0034] The third layer: The position-encoded data enters the multi-head self-attention layer. The multi-head self-attention mechanism allows the model to focus on different parts of the input data in parallel in different representation subspaces. By calculating the correlation between different positions of the data, it can effectively capture long-distance dependencies and perform more comprehensive and in-depth feature extraction on the input data.

[0035] The fourth layer: The data processed by the multi-head self-attention mechanism enters the feed-forward network layer. The feed-forward network is a simple fully connected neural network that performs nonlinear transformation on the data processed by the self-attention mechanism, further extracts and processes features, and enhances the expressiveness of the model.

[0036] The fifth layer: The data processed by the feedforward network enters the layer normalization layer. Layer normalization normalizes the input of each layer, which helps to speed up the model training process and prevent the gradient disappearance or explosion problems that may occur during the deep neural network training process, making the model training more stable and efficient.

[0037] The sixth layer: consists of 3 channel attention convolution (SE Conv) modules. The channel attention convolution module can adaptively weight the features of different channels. In this way, the model can highlight the features of important channels and suppress the features of unimportant channels, thereby further improving the model's feature extraction capabilities and enabling the model to focus more on key features for processing.

[0038] Output layer: The final output of the model is a Mel spectogram. The entire model extracts and processes features from the original acceleration data through a layer-by-layer structure, which can be applied to speech-related fields, such as restoring speech through facial acceleration data.

[0039] Furthermore, if Figure 4 As shown, the structure of the residual shrinkage block unit is as follows: Input layer: The dimension of input data is ,in Indicates the number of channels, Indicates width.

[0040] First layer: Batch normalization (BN), rectified linear unit (ReLU) and convolution (Conv) operations, convolution kernel size , the dimensions of both input and output are .

[0041] Second layer: Batch normalization (BN), rectified linear unit (ReLU) and convolution (Conv) operations are also performed, and the convolution kernel size is , the dimensions of both input and output are .

[0042] Intermediate processing: Input The data passes through a The convolution operation obtains a The intermediate variable .

[0043] Feature processing branch: output of the second layer The data is processed by absolute value (Absolute) and global average pooling (GAP) to obtain a The eigenvector of Then, the fully connected layer (FC) is operated in sequence ( When you get ,in represents the number of neurons in the fully connected layer), batch normalization (BN), rectified linear unit (ReLU), fully connected layer (FC) operations ( When you get ), and then get it through the Sigmoid function , and finally through get (Dimensions are ),in represents the adaptive weight adjustment parameter, is an intermediate variable.

[0044] Output layer: Output the second layer Data and intermediate variables Multiply, then multiply , and finally perform residual connection with the input data to obtain the final output, whose dimension is still .

[0045] The key operations in the residual shrinkage block unit structure include batch normalization, rectified linear units, convolution operations, absolute value processing in the feature processing branch, global average pooling, and fully connected layer operations. Through these operations, the input data can be effectively extracted and processed. At the same time, the residual connection helps in the training and optimization of the model.

[0046] In addition, the parameters of the soft threshold in the residual shrinkage block unit are learned to perform soft threshold processing on each channel while extracting features through convolution, and output intermediate features that remove redundant data and suppress the influence of noise.

[0047] Furthermore, if Figure 5As shown, the structure of the channel attention convolution module is as follows: First, the input dimension is The feature data is then subjected to the first layer of one-dimensional convolution (conv1d) operation, and its output dimension is still . Then the second layer of one-dimensional convolution (conv1d) operation is performed, and the output dimension is still ,in Indicates length, Indicates the number of channels.

[0048] Then, after two layers of convolution, Data at length scale ( Dimensions) to obtain a global pooling operation of dimension The channel length sequence.

[0049] Next, a series of operations are performed on the data obtained by global pooling: First, the first full connection (FC) operation is performed, and the input dimension is , the output dimension is ; Then perform a rectified linear unit operation on it, and the input and output dimensions are ; Then the second full connection (FC) operation is performed, and the input dimension is , the output dimension is ; Finally, the Sigmoid activation operation is performed, and the input and output dimensions are ,in Represents the feature dimension after the transformation of the intermediate fully connected layer. Get the Sigmoid activated After thresholding the data, apply it to the output of the second convolution layer The result is scaled and the output dimension is .

[0050] Finally, the scaled Data and original input The data is residually connected (added) to obtain the final output, whose dimension is The module realizes adaptive weighting of different channel features, highlights important channel features and suppresses unimportant channel features, and the residual connection helps to optimize the module performance.

[0051] Furthermore, in the model training process, the present invention uses the MSE mean square error loss function as a method to judge the difference between the model prediction output and the actual label. Specifically, during the training process, the processed acceleration data is used as input, the Mel spectrogram of the corresponding speech data is used as the target output, and the MSE mean square error loss function is used to calculate the difference between the model prediction result and the target output. Through the back propagation algorithm, all weight parameters in the model are adjusted according to the value of the loss function, and the training is continuously iterated until the model converges to obtain a trained model.

[0052] In addition, the vocoder of the present invention is HIFIGAN, which is used to convert the Mel spectrogram into final voice data. HIFIGAN is based on the architecture of Generative Adversarial Network (GAN). GAN consists of a generator and a discriminator. In HIFIGAN, the generator is responsible for converting the Mel spectrogram into voice data, and the discriminator is used to determine whether the generated voice data is close to the real voice data. The generator tries to generate voice data (signals) that are as realistic as possible by learning the mapping relationship between the Mel spectrogram and the corresponding real voice data (signals). The discriminator continuously evaluates the voice data generated by the generator and provides feedback to the generator so that it can continuously improve the generated results.

[0053] In this embodiment, in actual use, the user opens the mobile phone application, puts the phone close to the face, and starts to speak silently. The mobile phone software collects facial acceleration data in real time, and after denoising, inputs it into the trained model. The model quickly generates corresponding voice data based on the learned mapping relationship. The user can pre-set the transmission method and target device of the voice data in the mobile phone software. For example, if the Bluetooth headset is paired, choose to transmit the voice data to the Bluetooth headset via Bluetooth to achieve the output of silent voice.

[0054] Specific example: Taking an Android-based mobile phone as an example, a mobile phone application named "SilentSpeech" is developed. The trained facial movement speech recovery model is integrated into the program.

[0055] When the user needs to convey voice information silently, such as in a quiet conference room, open the "SilentSpeech" application, hold the phone close to the face, and speak normally and silently. The mobile phone application automatically starts the acceleration sensor to collect facial acceleration data at a sampling rate of 400hz, and collects voice data at the same time (the voice data is mainly used as a time alignment reference in subsequent data processing).

[0056] The collected data is transmitted to the computer in real time through socket communication. After receiving the data, the MATLAB receiving program on the computer performs data analysis, time alignment and noise removal, and stores the processed acceleration data locally.

[0057] The trained model is loaded into the mobile application. When the acceleration data of the user speaking silently is processed and input into the model, the model quickly generates the corresponding voice data. If the user is in a meeting and has paired a Bluetooth headset with the mobile phone, the user can choose to transmit the voice data to the Bluetooth headset via Bluetooth in the mobile application, and other participants can hear the voice content of the user's "silent speech".

[0058] Through the above specific implementation methods, the present invention can effectively realize the function of restoring voice data based on facial movements of the built-in accelerometer of the mobile phone, and provides an innovative and practical solution for silent voice communication, which has broad application prospects. Embodiment 2

[0059] like Figure 6 As shown, the present invention provides a method for restoring speech from facial movements on a mobile phone based on a convolution plus attention mechanism. The method adopts the facial movement restoration speech system on a mobile phone based on a convolution plus attention mechanism of the above-mentioned embodiment 1, and specifically comprises: S1: Call the mobile phone acceleration sensor and set the sampling rate to collect the user's facial acceleration data and voice data in real time and transmit the data to the computer; S2: Receive data on the computer and display it in real time. Use timestamps to align the acceleration data and voice data in time. Then save the aligned data pairs locally. Design a suitable filter algorithm to remove noise from the acceleration data and enhance the data signal-to-noise ratio. S3: Construct a facial action recovery speech model. The facial action recovery speech model first recovers the input acceleration data into the Mel spectrogram of the corresponding speech data through the pre-established convolutional neural network attention model, and then puts the Mel spectrogram into the vocoder to recover the final speech data; S4: The facial acceleration data of the user speaking silently after data processing and noise removal is input into the trained facial movement recovery speech model to generate corresponding speech data and transmit it according to the user's requirements.

[0060] The present embodiment provides a mobile phone facial movement recovery speech method based on convolution plus attention mechanism, which adopts the aforementioned mobile phone facial movement recovery speech system based on convolution plus attention mechanism. Therefore, the specific implementation method of the mobile phone facial movement recovery speech method based on convolution plus attention mechanism can be found in the embodiment section of the mobile phone facial movement recovery speech system based on convolution plus attention mechanism mentioned above. In order to avoid redundancy, it will not be repeated here. Embodiment 3

[0061] An embodiment of the present invention provides an electronic device, which includes a processor, a memory and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned mobile phone facial action voice recovery method based on convolution and attention mechanism. Embodiment 4

[0062] An embodiment of the present invention provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned mobile phone facial action voice recovery method based on convolution and attention mechanism.

[0063] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0064] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0065] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A mobile phone facial action speech recovery system based on convolution plus attention mechanism, characterized in that: include: The data acquisition module is used to call the mobile phone acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration data and voice data in real time and transmit the data to the computer; The data processing module is located on the computer side and is used to receive data and display it in real time. It uses timestamps to align the acceleration data and voice data in time, then saves the aligned data pairs locally, and removes noise from the acceleration data by designing a suitable filter algorithm to enhance the data signal-to-noise ratio. A model training module is used to construct a facial action recovery speech model, wherein the facial action recovery speech model first recovers the input acceleration data into a Mel spectrogram corresponding to the speech data through a pre-established convolutional neural network attention model, and then puts the Mel spectrogram into a vocoder to recover the final speech data; The speech generation module is used to input the facial acceleration data of the user speaking silently after data processing to remove noise into the trained facial movement recovery speech model, generate corresponding speech data and transmit it according to user requirements.

2. The mobile phone facial action voice recovery system based on convolution plus attention mechanism according to claim 1, characterized in that: The structure of the convolutional neural network attention model is: The acceleration data is used as the input layer; the first layer is composed of 5 residual shrinkage block units, which are used as specific residual structural units to perform preliminary feature extraction and noise processing on the input acceleration data; the second layer, i.e., the position encoding layer, is used to add position information to the data; The third layer is a multi-head self-attention mechanism layer, which is used to focus on different parts of the input data in different representation subspaces in parallel. It effectively captures long-distance dependencies by calculating the correlation between different positions of the data, and then deeply extracts features. The fourth layer is a feedforward network layer, which is used to perform nonlinear transformation on the data processed by the self-attention mechanism to enhance the feature extraction ability of the model; the fifth layer is a layer normalization layer, which is used to normalize the input, accelerate model training and prevent gradient disappearance or explosion problems; the sixth layer consists of 3 channel attention convolution modules, which are used to adaptively weight different channel features, highlight important channel features and suppress unimportant channel features; The output layer finally generates a mel-spectrogram.

3. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 2 is characterized in that: The parameters of the soft threshold in the residual shrinkage block unit are obtained through learning, so as to perform soft threshold processing on each channel while extracting features through convolution, and output intermediate features that remove redundant data and suppress the influence of noise.

4. The mobile phone facial action voice recovery system based on convolution plus attention mechanism according to claim 2, characterized in that: The structure of the residual shrinkage block unit is: Input layer: The dimension of input data is ,in Indicates the number of channels, Indicates width; First layer: batch normalization, rectified linear unit and convolution operations, convolution kernel size , the dimensions of both input and output are ; Second layer: batch normalization, rectified linear unit and convolution operations are also performed, and the convolution kernel size , the dimensions of both input and output are ; Intermediate Processing: Input The data passes through a The convolution operation obtains a The intermediate variable ; Feature processing branch: output of the second layer The data is processed by absolute value and global average pooling to obtain a The eigenvector of Then it goes through the fully connected layer operation, batch normalization, rectified linear unit, fully connected layer operation, and then through the Sigmoid function to get , and finally through get ,in represents the adaptive weight adjustment parameter, is an intermediate variable; Output layer: Output the second layer Data and intermediate variables Multiply, then multiply , and finally perform residual connection with the input data to obtain the final output, whose dimension is still .

5. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 2, characterized in that: The structure of the channel attention convolution module is: First, the input dimension is The feature data is then processed by the first layer of one-dimensional convolution operation, and its output dimension is still ; Then the second layer of one-dimensional convolution operation is performed, and the output dimension is still ,in Indicates length, Indicates the number of channels; Then, after two layers of convolution, The data is globally pooled on the length scale, resulting in a dimension of The channel length sequence of Next, a series of operations are performed on the data obtained by global pooling: first, the first full connection operation is performed, and the input dimension is , the output dimension is ; Then perform a rectified linear unit operation on it, and the input and output dimensions are ; Then the second full connection operation is performed, and the input dimension is , the output dimension is ; Finally, the Sigmoid activation operation is performed, and the input and output dimensions are ,in Represents the feature dimension after the transformation of the intermediate fully connected layer; Get the Sigmoid activated After thresholding the data, apply it to the output of the second convolution layer The result is scaled and the output dimension is ; Finally, the scaled Data and original input The data is residually connected to obtain the final output, whose dimension is .

6. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 1, characterized in that: The vocoder is HIFIGAN, which is used to convert the Mel spectrogram into final speech data.

7. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 1, characterized in that: The MSE mean square error loss function is used during model training to determine the difference between the model's predicted output and the actual label.

8. A method for restoring speech from facial movements on a mobile phone based on a convolutional and attention mechanism, characterized in that: The method adopts the mobile phone facial action voice recovery system based on convolution plus attention mechanism according to any one of claims 1 to 7, and specifically comprises: S1: Call the mobile phone acceleration sensor and set the sampling rate to collect the user's facial acceleration data and voice data in real time and transmit the data to the computer; S2: Receive data on the computer and display it in real time. Use timestamps to align the acceleration data and voice data in time. Then save the aligned data pairs locally. Design a suitable filter algorithm to remove noise from the acceleration data and enhance the data signal-to-noise ratio. S3: constructing a facial action recovery speech model, wherein the facial action recovery speech model first recovers the input acceleration data into a Mel spectrogram of the corresponding speech data through a pre-established convolutional neural network attention model, and then puts the Mel spectrogram into a vocoder to recover the final speech data; S4: The facial acceleration data of the user speaking silently after data processing and noise removal is input into the trained facial movement recovery speech model to generate corresponding speech data and transmit it according to the user's requirements.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the mobile phone facial action speech recovery method based on convolution and attention mechanism as described in claim 8.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, which includes a number of instructions for enabling a computer device to execute the mobile phone facial action speech recovery method based on convolution and attention mechanism as described in claim 8.

Images