Mobile phone facial action speech recovery system based on convolution and attention mechanism

Abstract

The present invention provides a mobile phone facial movement speech recovery system based on a convolution-plus-attention mechanism, relating to the field of speech processing technology. The system comprises a data acquisition module, a data processing module, a model training module, and a speech generation module. The data acquisition module uses the mobile phone's accelerometer to synchronously collect facial acceleration and speech data in real time and transmit it to a computer. The data processing module receives, aligns, stores, and performs denoising on the computer to enhance the signal-to-noise ratio. The model training module constructs a model, first converting the acceleration data into a Mel-level spectrogram using a convolutional neural network attention model, and then generating speech through a vocoder. The speech generation module inputs the processed facial acceleration data into the trained model, generates speech, and transmits it on demand. This system effectively solves existing problems, achieving facial movement-to-speech recovery, and has promising application prospects.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a mobile phone facial action speech recovery system based on a convolution plus attention mechanism. Background Art

[0002] In the tide of modern technology, smartphones have become a core device in our lives, and the development of accelerometer technology within them has undergone several stages. In the early days, accelerometers were primarily used to implement basic phone functions, such as controlling screen rotation by determining the phone's status, counting steps, and providing input for game operations. However, due to the highly integrated design requirements of mobile phones at the time, key performance indicators such as accelerometer sampling rate and response speed were limited, making it difficult to meet the needs of more complex applications.

[0003] With the continuous advancement of technology, the performance of built-in accelerometers in mobile phones has improved, with sampling rates reaching 400-500 Hz. However, in practical application scenarios, many challenges still exist. On the one hand, this sampling rate is still insufficient for applications requiring high-precision acceleration signal analysis. For example, in some scenarios that require high-definition motion capture, it cannot provide sufficiently accurate data support. On the other hand, the acceleration signal acquisition process is highly susceptible to interference from external factors. Gravity, human movement, and the device's own vibrations all generate noise, seriously affecting signal quality. This in turn restricts the development of applications based on acceleration signals and makes it difficult to run more complex related programs or software on mobile phones.

[0004] At the same time, deep learning technology has flourished, sparking a wave of technological change in numerous fields. Deep learning models, with their advanced network architectures, such as the Transformer and its variants, have opened up new avenues for data processing. For example, the Vision Transformer (ViT), with its innovative image segmentation processing approach, has achieved comparable or even superior results to traditional convolutional neural networks (CNNs) in image recognition. Residual networks (ResNet) and their improved versions have driven rapid development in fields such as image recognition by addressing the challenges of training deep networks. These deep learning models, trained on vast amounts of data, are playing a key role in areas such as autonomous driving, facial recognition, and security detection, achieving a leap forward in intelligent capabilities across these fields.

[0005] When deep learning models are applied to speech signal processing, traditional convolutional neural networks (CNNs) incorporate attention mechanisms to enhance focus on important information. The channel-wise attention mechanism automatically assigns channel weights, highlighting key channel information. However, traditional CNNs still have significant shortcomings when processing speech signals, particularly when converting low-sampling-rate signals (such as the 400Hz signal from a mobile phone accelerometer) to high-sampling-rate signals (such as the 22050Hz signal required for high-quality audio). Their computing power and learning capabilities are insufficient. While vocoders can convert digitized speech signals into audible sound, they rely on accurate sound feature input. This makes accurate conversion from low-sampling-rate acceleration signals to high-sampling-rate speech signals a challenge for current technology. There is a lack of deep learning models specifically designed to achieve this mapping from low-sampling-rate one-dimensional time series signals to high-sampling-rate time series signals.

[0006] In summary, current technological development faces numerous challenges in optimizing smartphone accelerometer performance, applying deep learning models to speech signal processing, and converting low- to high-sampling-rate signals. Innovative solutions are urgently needed to overcome these bottlenecks and enable efficient and accurate speech signal recovery and processing based on the phone's built-in accelerometer, providing stronger technical support for future applications such as intelligent voice interaction. Summary of the Invention

[0007] To this end, an embodiment of the present invention provides a mobile phone facial movement recovery speech system based on a convolution plus attention mechanism, which is used to solve the problems in the prior art such as the low sampling rate and high acquisition noise of the built-in accelerometer of the mobile phone, the lack of a deep learning model specifically for generating high-sampling-rate speech signals based on low-sampling-rate acceleration signals, and the difficulty in capturing one-dimensional time series signal features and mapping relationships.

[0008] To solve the above problems, an embodiment of the present invention provides a mobile phone facial action voice recovery system based on a convolution plus attention mechanism, which includes:

[0009] The data acquisition module is used to call the mobile phone's acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration data and voice data in real time and transmit the data to the computer;

[0010] The data processing module, located on the computer side, is used to receive data and display it in real time. It uses timestamps to align the acceleration data with the voice data, then saves the aligned data pairs locally. It also removes noise from the acceleration data by designing a suitable filter algorithm to enhance the data signal-to-noise ratio.

[0011] A model training module is used to build a facial action speech recovery model. The facial action speech recovery model first recovers the input acceleration data into a Mel-level spectrogram of the corresponding speech data through a pre-established convolutional neural network attention model, and then puts the Mel-level spectrogram into a vocoder to recover the final speech data;

[0012] The speech generation module is used to input the facial acceleration data of the user speaking silently after data processing and noise removal into the trained facial movement recovery speech model, generate corresponding speech data and transmit it according to user requirements.

[0013] Preferably, the structure of the convolutional neural network attention model is:

[0014] The acceleration data is used as the input layer; the first layer is composed of 5 residual shrinkage block units, which are specific residual structure units and are used to perform preliminary feature extraction and noise processing on the input acceleration data; the second layer, namely the position encoding layer, is used to add position information to the data; the third layer is a multi-head self-attention mechanism layer, which is used to focus on different parts of the input data in parallel in different representation subspaces, and effectively capture long-distance dependencies by calculating the correlation between different positions of the data, and then deeply extract features; the fourth layer is a feedforward network layer, which is used to perform nonlinear transformation on the data processed by the self-attention mechanism to enhance the feature extraction ability of the model; the fifth layer is a layer normalization layer, which is used to normalize the input, accelerate model training and prevent gradient disappearance or explosion problems; the sixth layer is composed of 3 channel attention convolution modules, which are used to adaptively weight different channel features, highlight important channel features and suppress unimportant channel features; the output layer finally generates a Mel spectrum map.

[0015] Preferably, the parameters of the soft threshold in the residual shrinkage block unit are obtained through learning, so as to perform soft threshold processing on each channel while extracting features through convolution, and output intermediate features that remove redundant data and suppress the influence of noise.

[0016] Preferably, the structure of the residual shrinkage block unit is:

[0017] Input layer: The dimension of input data is ,in Indicates the number of channels, Indicates width;

[0018] First layer: batch normalization, rectified linear unit and convolution operations, convolution kernel size , the dimensions of both input and output are ;

[0019] Second layer: batch normalization, rectified linear unit and convolution operations are also performed, and the convolution kernel size , the dimensions of both input and output are ;

[0020] Intermediate processing: input The data passes through a The convolution operation obtains a Intermediate variables ;

[0021] Feature processing branch: output of the second layer The data is processed by absolute value and global average pooling to obtain a The eigenvector of

[0022] Then it goes through the full connection layer operation, batch normalization, rectified linear unit, full connection layer operation, and then through the Sigmoid function to get , and finally through get ,in represents the adaptive weight adjustment parameter, is an intermediate variable;

[0023] Output layer: Output the second layer Data and intermediate variables Multiply, then multiply , and finally perform residual connection with the input data to obtain the final output, whose dimension is still .

[0024] Preferably, the structure of the channel attention convolution module is:

[0025] First, the input dimension is The feature data of the first layer is then subjected to one-dimensional convolution operation, and its output dimension is still ; Then the second layer of one-dimensional convolution operation is performed, and the output dimension is still ,in Indicates length, Indicates the number of channels;

[0026] Then, after two layers of convolution, The data is globally pooled on the length scale, resulting in a dimension of The channel length sequence of

[0027] Next, a series of operations are performed on the data obtained by global pooling: first, the first full connection operation is performed, and the input dimension is , the output dimension is ; Then perform a rectified linear unit operation on it, and the input and output dimensions are ; Then the second full connection operation is performed, and the input dimension is , the output dimension is ; Finally, the Sigmoid activation operation is performed, and the input and output dimensions are ,in Represents the feature dimension after the transformation of the middle fully connected layer; after Sigmoid activation After thresholding the data, it is applied to the output of the second convolution layer. The scale transformation operation is performed on the result, and the output dimension is ;

[0028] Finally, the scaled Data and original input The data is residually connected to obtain the final output, whose dimension is .

[0029] Preferably, the vocoder is HIFIGAN, which is used to convert the Mel-spectrogram into final speech data.

[0030] Preferably, the mean square error (mse) loss function is used during model training to determine the difference between the model's predicted output and the actual label.

[0031] An embodiment of the present invention further provides a method for restoring speech from facial movements on a mobile phone based on a convolution and attention mechanism. The method adopts the above-mentioned system for restoring speech from facial movements on a mobile phone based on a convolution and attention mechanism, and includes the following steps:

[0032] S1: Call the mobile phone's accelerometer and set the sampling rate to collect the user's facial acceleration data and voice data in real time and transmit the data to the computer;

[0033] S2: Receive data on the computer and display it in real time. Use timestamps to align the acceleration data with the voice data. Then, save the aligned data pairs locally. Design a suitable filter algorithm to remove noise from the acceleration data and enhance the signal-to-noise ratio.

[0034] S3: Constructing a facial action speech recovery model. The facial action speech recovery model first recovers the input acceleration data into a Mel-spectrogram of the corresponding speech data through a pre-established convolutional neural network attention model, and then puts the Mel-spectrogram into a vocoder to recover the final speech data.

[0035] S4: The facial acceleration data of the user speaking silently after data processing and noise removal is input into the trained facial movement recovery speech model to generate corresponding speech data and transmit it according to the user's requirements.

[0036] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned mobile phone facial action voice recovery method based on the convolution plus attention mechanism.

[0037] An embodiment of the present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned mobile phone facial action voice recovery method based on convolution and attention mechanism.

[0038] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0039] (1) This invention achieves real-time synchronous acquisition and transmission of mobile phone acceleration and voice data, ensuring accurate data association and solving the problem of asynchronous acquisition. The computer terminal uses a filter algorithm to effectively remove noise from the acceleration data, improve the signal-to-noise ratio, and provide reliable data for model training.

[0040] (2) Construct a facial action speech recovery model that integrates a convolutional neural network attention model and a vocoder to fill the gap in the low-sampling rate to high-sampling rate speech signal model and achieve effective mapping. With the help of the convolutional neural network attention model, the one-dimensional time series signal characteristics and mapping relationship are accurately captured, overcoming the existing technical difficulties and improving the accuracy of speech recovery.

[0041] (3) It has the function of converting processed facial acceleration data into voice and transmitting it on demand, meeting the needs of scenarios such as silent communication and having practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. Those skilled in the art can derive other drawings based on these drawings without inventive effort. Among them:

[0043] Figure 1 A block diagram of a mobile phone facial action speech recovery system based on a convolutional plus attention mechanism provided in an embodiment;

[0044] Figure 2 This is a schematic diagram of the architecture of the facial action speech recovery model in the embodiment;

[0045] Figure 3Schematic diagram of the structure of the convolutional neural network attention (CNN_Attention) model in the embodiment;

[0046] Figure 4 2 is a schematic diagram of the structure of a residual shrinkage block unit (RSBU) in an embodiment;

[0047] Figure 5 Schematic diagram of the structure of the channel attention convolution (SE Conv) module in the embodiment;

[0048] Figure 6 This is a flowchart of a method for restoring speech using facial movements on a mobile phone based on a convolutional and attention mechanism provided in an embodiment. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention. Example 1

[0050] In order to solve the problems in the existing technology of low sampling rate and high noise of built-in accelerometers in mobile phones, lack of deep learning models specifically for generating high sampling rate voice signals based on low sampling rate acceleration signals, and difficulty in capturing one-dimensional time series signal features and mapping relationships. Figure 1 As shown, the embodiment of the present invention proposes a mobile phone facial action voice recovery system based on a convolution plus attention mechanism, the system comprising:

[0051] The data acquisition module is used to call the mobile phone's acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration data and voice data in real time and transmit the data to the computer;

[0052] The data processing module, located on the computer side, is used to receive data and display it in real time. It uses timestamps to align the acceleration data with the voice data, then saves the aligned data pairs locally. It also removes noise from the acceleration data by designing a suitable filter algorithm to enhance the data signal-to-noise ratio.

[0053] The model training module is used to build a facial action speech recovery model. The facial action speech recovery model first recovers the input acceleration data into the Mel-level spectrogram of the corresponding speech data through a pre-established convolutional neural network attention model. The Mel-level spectrogram is then put into the vocoder to recover the final speech data.

[0054] The speech generation module is used to input the facial acceleration data of the user speaking silently after data processing and noise removal into the trained facial movement recovery speech model, generate corresponding speech data and transmit it according to user requirements.

[0055] As can be seen from the above technical solution, the present invention proposes a mobile phone facial movement voice recovery system based on the convolution plus attention mechanism. The data acquisition module synchronously collects facial acceleration and voice data in real time and transmits them to the computer. The data processing module performs data alignment, denoising and storage on the computer to enhance the signal-to-noise ratio. The facial movement voice recovery model constructed by the model training module first uses the convolutional neural network attention model to convert the acceleration data into a Mel-speech spectrogram, and then generates voice through the vocoder, solving the problems of lack of a specialized model and difficulty in capturing features and mapping relationships. The voice generation module can generate voice from the processed facial acceleration data and transmit it on demand. The system achieves efficient and accurate data acquisition and processing. The innovative model architecture can effectively convert low-sampling rate signals and has a practical voice generation function that can meet the needs of scenarios such as silent communication, thereby improving the overall effect and practicality of voice recovery from facial movements.

[0056] In this embodiment, the data acquisition module is used to write a mobile phone software program, call the mobile phone acceleration sensor and set the sampling rate, collect the user's facial acceleration data (signal) and voice data (signal) in real time, and transmit the data to the computer through the socket communication principle.

[0057] Specifically, develop a mobile app, utilize the sensor interface provided by the phone's operating system, write code to access the phone's built-in accelerometer, and set an appropriate sampling rate (e.g., 400 Hz). During the acquisition process, simultaneously enable voice data collection to ensure real-time, synchronized collection of facial acceleration and voice data. Establish a stable socket communication connection to transmit the collected data to the computer in real time. For example, on Android, use the SensorManager class to obtain an accelerometer instance, use the AudioRecord class to collect voice data, and then use the Socket class to establish a TCP connection to send the data to the computer.

[0058] In this embodiment, the data processing module is located on the computer side and is used to write a MATLAB program as a receiving server to receive data and plot and display it in real time. The acceleration data and voice data are temporally aligned using timestamps, and the aligned data pairs are then saved locally. The noise in the acceleration data is removed by designing a suitable filter algorithm to enhance the data signal-to-noise ratio.

[0059] Specifically, on the computer side, a receiving server program was written using MATLAB. Upon receiving data from the mobile phone, the data was first parsed and stored. Then, based on the timestamp information in the data, the acceleration data and voice data were accurately aligned in time. To address the noise issues in the acceleration data (signal), appropriate filter algorithms were designed and implemented. For example, a Kalman filter algorithm was used to estimate the true value of the signal in real time based on the dynamic characteristics of the signal and the measurement noise characteristics. This effectively removed noise such as gravity acceleration, slight body movement, and device vibration, significantly improving the signal-to-noise ratio and providing high-quality data for subsequent model training.

[0060] In this embodiment, the model training module is used to build a facial action recovery speech model. The model first recovers the input acceleration data into the Mel spectrogram of the corresponding speech data through the pre-established convolutional neural network attention (CNN_Attention) model, and then puts the Mel spectrogram into the vocoder (HIFIGAN) to recover the final speech data. Figure 2 shown.

[0061] Furthermore, the overall architecture of the convolutional neural network attention (CNN_Attention) model established by the present invention is as follows Figure 3 The main purpose of this model is to convert the input acceleration data into a mel-spectrogram. The model consists of multiple modules that process the input data in sequence, gradually extracting and converting features.

[0062] The structure of the CNN_Attention model is as follows:

[0063] Input layer: The acceleration data is used as the input layer, which is the raw data processed by the entire model.

[0064] The first layer consists of five residual shrinkage block units (RSBUs). These RSBUs serve as specific residual structural units and can perform preliminary feature extraction and noise processing on the input acceleration data.

[0065] Second layer: After being processed by five residual shrinkage blocks, the data enters the positional encoding layer. The positional encoding layer adds position information to the data, which is crucial for the subsequent self-attention mechanism to process the data, as the self-attention mechanism needs to know the order and relative position of the data to better capture the relationship between the data.

[0066] The third layer: The position-encoded data enters the multi-head self-attention layer. This mechanism allows the model to concurrently focus on different parts of the input data in different representation subspaces. By calculating the correlation between different data positions, it effectively captures long-range dependencies and performs more comprehensive and in-depth feature extraction on the input data.

[0067] Layer 4: Data processed by the multi-head self-attention mechanism enters the feedforward network layer. The feedforward network is a simple, fully connected neural network that performs nonlinear transformations on the data processed by the self-attention mechanism to further extract and process features, enhancing the model's expressive power.

[0068] Layer 5: Data processed by the feedforward network enters the layer normalization layer. Layer normalization normalizes the input of each layer, helping to accelerate model training and prevent gradient vanishing or exploding problems that can occur during deep neural network training, making model training more stable and efficient.

[0069] The sixth layer consists of three channel-attention convolution (SE Conv) modules. These modules adaptively weight features from different channels. This allows the model to highlight features from important channels and suppress features from less important channels, further improving its feature extraction capabilities and allowing it to focus on key features.

[0070] Output layer: The final output of the model is a Mel spectogram. The entire model extracts and processes features from the raw acceleration data through a layered structure. It can be applied to speech-related fields, such as recovering speech from facial acceleration data.

[0071] Furthermore, if Figure 4 As shown in , the structure of the residual shrinkage block unit is as follows:

[0072] Input layer: The dimension of input data is ,in Indicates the number of channels, Indicates width.

[0073] First layer: Batch normalization (BN), rectified linear unit (ReLU) and convolution (Conv) operations, convolution kernel size , the dimensions of both input and output are .

[0074] Second layer: Batch normalization (BN), rectified linear unit (ReLU) and convolution (Conv) operations are also performed, and the convolution kernel size is , the dimensions of both input and output are .

[0075] Intermediate processing: input The data passes through a The convolution operation obtains a Intermediate variables .

[0076] Feature processing branch: output of the second layer The data is processed by absolute value (Absolute) and global average pooling (GAP), and a The eigenvector of

[0077] Then it goes through the fully connected layer (FC) operation ( Get ,in represents the number of neurons in the fully connected layer), batch normalization (BN), rectified linear unit (ReLU), fully connected layer (FC) operations ( Get ), and then get it through the Sigmoid function , and finally through get (Dimensions are ),in represents the adaptive weight adjustment parameter, is an intermediate variable.

[0078] Output layer: Output the second layer Data and intermediate variables Multiply, then multiply , and finally perform residual connection with the input data to obtain the final output, whose dimension is still .

[0079] The key operations in the residual shrinkage block unit structure include batch normalization, rectified linear units, convolution operations, absolute value processing in the feature processing branch, global average pooling, and fully connected layer operations. Through these operations, the input data can be effectively extracted and processed. At the same time, the residual connection helps in model training and optimization.

[0080] In addition, the parameters of the soft threshold in the residual shrinkage block unit are learned to perform soft threshold processing on each channel while convolution is extracting features, and output intermediate features that remove redundant data and suppress the influence of noise.

[0081] Furthermore, if Figure 5As shown in Figure 2, the structure of the channel attention convolution module is as follows:

[0082] First, the input dimension is The feature data of , followed by the first layer of one-dimensional convolution (conv1d) operation, the output dimension is still . Then the second layer of one-dimensional convolution (conv1d) operation is performed, and the output dimension is still ,in Indicates length, Indicates the number of channels.

[0083] Then, after two layers of convolution, Data at length scale ( Dimensions) to obtain a global pooling operation of dimension The channel length sequence.

[0084] Next, a series of operations are performed on the data obtained by global pooling: first, the first full connection (FC) operation is performed, and the input dimension is , the output dimension is ; Then perform a rectified linear unit operation on it, and the input and output dimensions are ; Then the second full connection (FC) operation is performed, and the input dimension is , the output dimension is ; Finally, the Sigmoid activation operation is performed, and the input and output dimensions are ,in Represents the feature dimension after the transformation of the middle fully connected layer. Get the Sigmoid activated After thresholding the data, it is applied to the output of the second convolution layer. The scale transformation operation is performed on the result, and the output dimension is .

[0085] Finally, the scaled Data and original input The data is residually connected (added) to obtain the final output, whose dimension is This module implements adaptive weighting of different channel features, highlighting important channel features and suppressing unimportant channel features, and the residual connection helps optimize module performance.

[0086] Furthermore, the present invention uses the mean squared error (MSE) loss function during model training to determine the difference between the model's predicted output and the actual label. Specifically, during training, the processed acceleration data is used as input, and the corresponding mel-spectrogram of the speech data is used as the target output. The mean squared error (MSE) loss function is used to calculate the difference between the model's predicted output and the target output. Using the backpropagation algorithm, all weight parameters in the model are adjusted based on the loss function's value. Training is then iterated until the model converges, resulting in a trained model.

[0087] Furthermore, the vocoder of this invention is a HIFIGAN, which is used to convert mel-level spectrograms into final speech data. HIFIGAN is based on the architecture of a generative adversarial network (GAN). A GAN consists of a generator and a discriminator. In HIFIGAN, the generator is responsible for converting mel-level spectrograms into speech data, while the discriminator is responsible for determining whether the generated speech data is close to real speech data. The generator attempts to generate speech data (signals) that are as realistic as possible by learning the mapping relationship between mel-level spectrograms and corresponding real speech data (signals). The discriminator continuously evaluates the speech data produced by the generator and provides feedback to the generator, enabling it to continuously improve its results.

[0088] In actual use, the user opens the mobile app, holds the phone close to their face, and begins speaking silently. The mobile software collects facial acceleration data in real time, de-noises it, and then inputs it into the trained model. Based on the learned mapping, the model quickly generates corresponding speech data. The user can pre-set the voice data transmission method and target device in the mobile software. For example, if a Bluetooth headset is paired, the user can select Bluetooth transmission to transmit the voice data to the headset, achieving silent speech output.

[0089] Specific example: Taking an Android phone as an example, develop a mobile app called "SilentSpeech" and integrate a trained facial movement speech recovery model into the app.

[0090] In scenarios where a user needs to convey a voice message silently, such as in a quiet conference room, they can open the "SilentSpeech" app, hold their phone close to their face, and speak normally and silently. The app automatically activates the accelerometer to collect facial acceleration data at a sampling rate of 400 Hz, along with voice data (this data is primarily used as a time alignment reference for subsequent data processing).

[0091] The collected data is transmitted to the computer in real time via socket communication. After receiving the data, the MATLAB receiving program on the computer performs data analysis, time alignment, and noise removal, and then stores the processed acceleration data locally.

[0092] The trained model is loaded into the mobile app. When the user's silent speech acceleration data is processed and fed into the model, the model quickly generates the corresponding speech data. If the user is in a meeting and has paired a Bluetooth headset with their phone, they can choose to transmit the voice data to the headset via Bluetooth in the mobile app, allowing other participants to hear the user's "silent" speech.

[0093] Through the above specific implementation methods, the present invention can effectively realize the function of restoring voice data based on facial movements of the built-in accelerometer of the mobile phone, providing an innovative and practical solution for silent voice communication, and has broad application prospects. Example 2

[0094] like Figure 6 As shown, the present invention provides a mobile phone facial action voice restoration method based on a convolution plus attention mechanism. The method adopts the mobile phone facial action voice restoration system based on a convolution plus attention mechanism of the above embodiment 1, and specifically includes:

[0095] S1: Call the mobile phone's accelerometer and set the sampling rate to collect the user's facial acceleration data and voice data in real time and transmit the data to the computer;

[0096] S2: Receive data on the computer and display it in real time. Use timestamps to align the acceleration data with the voice data. Then, save the aligned data pairs locally. Design a suitable filter algorithm to remove noise from the acceleration data and enhance the signal-to-noise ratio.

[0097] S3: Construct a facial action speech recovery model. The facial action speech recovery model first recovers the input acceleration data into the Mel-level spectrogram of the corresponding speech data through the pre-established convolutional neural network attention model, and then puts the Mel-level spectrogram into the vocoder to recover the final speech data.

[0098] S4: The facial acceleration data of the user speaking silently after data processing and noise removal is input into the trained facial movement recovery speech model to generate corresponding speech data and transmit it according to the user's requirements.

[0099] The present embodiment provides a mobile phone facial movement recovery voice method based on the convolution plus attention mechanism, which adopts the aforementioned mobile phone facial movement recovery voice system based on the convolution plus attention mechanism. Therefore, the specific implementation method of the mobile phone facial movement recovery voice method based on the convolution plus attention mechanism can be found in the embodiment section of the mobile phone facial movement recovery voice system based on the convolution plus attention mechanism above. In order to avoid redundancy, it will not be repeated here. Example 3

[0100] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a bus system. The processor and the memory are connected through the bus system. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the above-mentioned mobile phone facial action speech recovery method based on the convolution plus attention mechanism. Example 4

[0101] An embodiment of the present invention provides a computer storage medium storing a computer software product. The computer software product includes several instructions for enabling a computer device to execute the above-mentioned mobile phone facial action voice recovery method based on the convolution plus attention mechanism.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications derived therefrom remain within the scope of protection of the present invention.

Claims

1. A mobile phone facial action speech recovery system based on convolution and attention mechanism, characterized by: include: The data acquisition module is used to call the mobile phone's acceleration sensor and set the sampling rate to synchronously collect the user's facial acceleration data and voice data in real time and transmit the data to the computer; The data processing module, located on the computer side, is used to receive data and display it in real time. It uses timestamps to align the acceleration data with the voice data, then saves the aligned data pairs locally. It also removes noise from the acceleration data by designing a suitable filter algorithm to enhance the data signal-to-noise ratio. A model training module is used to build a facial action speech recovery model. The facial action speech recovery model first recovers the input acceleration data into a Mel-level spectrogram of the corresponding speech data through a pre-established convolutional neural network attention model, and then puts the Mel-level spectrogram into a vocoder to recover the final speech data; The speech generation module is used in actual use. When a user opens a mobile application, holds the phone close to his face, and starts speaking silently, the facial acceleration data of the user while speaking silently, which has been processed to remove noise, is input into the trained facial movement recovery speech model to generate corresponding speech data and transmit it according to the user's requirements.

2. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 1 is characterized in that: The structure of the convolutional neural network attention model is: The acceleration data is used as the input layer; the first layer consists of five residual shrinkage block units, which are specific residual structural units used to perform preliminary feature extraction and noise processing on the input acceleration data; the second layer, the position encoding layer, is used to add position information to the data; The third layer is a multi-head self-attention mechanism layer, which is used to focus on different parts of the input data in parallel in different representation subspaces. It effectively captures long-range dependencies by calculating the correlation between different positions of the data, and then deeply extracts features. The fourth layer is a feedforward network layer, which is used to perform nonlinear transformations on the data processed by the self-attention mechanism to enhance the model's feature extraction capabilities. The fifth layer is a layer normalization layer, which is used to normalize the input, accelerate model training, and prevent gradient vanishing or exploding problems. The sixth layer consists of three channel attention convolution modules, which are used to adaptively weight different channel features, highlighting important channel features and suppressing unimportant channel features. The output layer finally generates a mel-spectrogram.

3. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 2 is characterized in that: The parameters of the soft threshold in the residual shrinkage block unit are obtained through learning, so as to perform soft threshold processing on each channel while extracting features through convolution, and output intermediate features that remove redundant data and suppress the influence of noise.

4. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 2 is characterized in that: The structure of the residual shrinkage block unit is: Input layer: The dimension of the input data is C×W×1, where C represents the number of channels and W represents the width; First layer: Batch normalization, rectified linear units, and convolution operations are performed. The convolution kernel size is K = C, and the dimensions of the input and output are both C × W × 1. The second layer also performs batch normalization, rectified linear units, and convolution operations. The convolution kernel size is K = C, and the input and output dimensions are both C × W × 1. Intermediate processing: The input C×W×1 data undergoes a 1×1×1 convolution operation to obtain a 1×1×1 intermediate variable x; Feature processing branch: Perform absolute value processing and global average pooling on the C×W×1 data output from the second layer to obtain a C×1×1 feature vector; Then it goes through the fully connected layer operation, batch normalization, rectified linear unit, and fully connected layer operation in sequence, and then z is obtained through the Sigmoid function. Finally, α is obtained through α=1-z, where α represents the adaptive weight adjustment parameter and z is the intermediate variable. Output layer: Multiply the C×W×1 output data of the second layer by the intermediate variable x, then multiply it by α, and finally perform a residual connection with the input data to obtain the final output, whose dimension is still C×W×1.

5. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 2 is characterized in that: The structure of the channel attention convolution module is: First, the input dimension is L×C feature data, followed by the first layer of one-dimensional convolution operation, whose output dimension is still L×C; then the second layer of one-dimensional convolution operation is performed, and the output dimension is also L×C, where L represents the length and C represents the number of channels; Then, a global pooling operation is performed on the L×C data after two layers of convolution on the length scale to obtain a channel length sequence with a dimension of 1×C; Next, a series of operations are performed on the data obtained by global pooling: first, a full connection operation is performed with an input dimension of 1×C and an output dimension of 1×H; then a rectified linear unit operation is performed on it, with both input and output dimensions of 1×H; then a second full connection operation is performed with an input dimension of 1×H and an output dimension of 1×C; finally, a Sigmoid activation operation is performed with an input and output dimension of 1×C, where H represents the feature dimension after the transformation of the intermediate fully connected layer; after obtaining the 1×C threshold data after Sigmoid activation, it is applied to the L×C result output of the second convolution layer for a scale transformation operation, with an output dimension of L×C; Finally, the scale-transformed L×C data is residually connected with the original input L×C data to obtain the final output, whose dimension is L×C.

6. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 1 is characterized in that: The vocoder is HIFIGAN, which is used to convert the Mel spectrogram into final speech data.

7. The mobile phone facial action voice recovery system based on convolution and attention mechanism according to claim 1 is characterized in that: During model training, the mean square error (mse) loss function is used to determine the difference between the model's predicted output and the actual label.

8. A mobile phone facial action voice recovery method based on convolution and attention mechanism, characterized by: The method adopts the mobile phone facial action voice recovery system based on the convolution plus attention mechanism according to any one of claims 1 to 7, and specifically includes: S1: Call the mobile phone's accelerometer and set the sampling rate to collect the user's facial acceleration data and voice data in real time and transmit the data to the computer; S2: Receive data on the computer and display it in real time. Use timestamps to align the acceleration data with the voice data. Then, save the aligned data pairs locally. Design a suitable filter algorithm to remove noise from the acceleration data and enhance the signal-to-noise ratio. S3: Constructing a facial action speech recovery model. The facial action speech recovery model first recovers the input acceleration data into a Mel-spectrogram of the corresponding speech data through a pre-established convolutional neural network attention model, and then puts the Mel-spectrogram into a vocoder to recover the final speech data. S4: The facial acceleration data of the user speaking silently after data processing and noise removal is input into the trained facial movement recovery speech model to generate corresponding speech data and transmit it according to the user's requirements.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the mobile phone facial action speech recovery method based on the convolution plus attention mechanism as described in claim 8.

10. A computer storage medium, characterized in that The computer storage medium stores a computer software product, which includes several instructions for enabling a computer device to execute the mobile phone facial action voice recovery method based on convolution and attention mechanism as described in claim 8.

Images