A method for reconstructing silent speech based on vocalized surface electromyography signals

Abstract

The present invention discloses a silent speech reconstruction method based on surface electromyography (EMG) signals. Electrodes are used to collect speech-related surface electromyography (EMG) signals and speech signals during speech production. Furthermore, speech-related surface electromyography (EMG) signals are collected in silent conditions. A deep learning network is used to link the silent surface electromyography signals with speech signals of the same text. This method reconstructs speech signals in silent conditions, transfers information content in silent conditions, and enables direct transmission of information in silent conditions. In situations where speech is impossible (e.g., military or police covert operations), the vocal cords remain silent, and information transfer is accomplished through silent reading.

Description

Technical Field

[0001] The present invention relates to the fields of human-machine interface and artificial intelligence, and in particular to a silent speech reconstruction method based on vocal surface electromyography signals, which is used for completing speech reconstruction during silent speaking. Background Art

[0002] Human-computer interfaces (HMIs) are a fundamental approach that uses hardware and software to explore human neural activity. They enable humans to interact with their surroundings and communicate their inner thoughts without speech. Silent speech decoding is one of the most popular areas of HMI. Silent speech decoding aims to use biosignals to detect biological activity associated with speech (rather than acoustic data) and decode human thoughts. This technology has a wide range of real-world applications, such as speech rehabilitation, silent manipulation, and as a supplement to acoustic speech.

[0003] Existing research on silent speech decoding can be divided into two main categories: biosignal-to-text and biosignal-to-speech. Biosignal-to-text approaches can overlook speaker identity, and classification errors can propagate to downstream processes. Some work has attempted to achieve silent speech decoding by reconstructing speech from biosignals detected in silent speech. However, problems persist, such as low output speech accuracy, unclear speech quality, and low speech quality. Summary of the Invention

[0004] The present invention mainly aims to solve the technical problems of low accuracy in decoding and reconstructing EEG signals and unclear voice quality in the prior art, and provides a high-quality silent speech reconstruction method based on vocal surface electromyography signals.

[0005] The present invention solves the above technical problems mainly through the following technical solutions: a silent speech reconstruction method based on vocal surface electromyography signals, based on a speech reconstruction model, including a model training process and a reconstruction process, the model training process is as follows:

[0006] S1: The subject reads the text aloud normally. The electrode array attached to the body surface collects the acoustic surface electromyography signal during the reading, and the microphone simultaneously collects the speech signal of the reading. The acoustic surface electromyography signal and the speech signal are time-synchronized.

[0007] S2: The subject silently reads the same text as in step S1, and the electrode array collects silent surface electromyographic signals during the silent reading;

[0008] S3. Preprocess and frame the voiced surface electromyography signal, the silent surface electromyography signal and the speech signal. The voiced surface electromyography feature obtained after the voiced surface electromyography signal is framed is recorded as x 1：M , M is the total number of frames of the voiced surface electromyography feature; the speech feature obtained after the speech signal is framed is recorded as Y1：M , the number of frames of speech features is the same as that of voiced surface electromyography features; the silent surface electromyography features obtained after the silent surface electromyography signal is framed are recorded as X 1：N , N is the total number of frames of silent surface EMG features;

[0009] S4, according to the speech signal and its corresponding text, obtain the real phoneme ph corresponding to the speech signal by forced alignment 1：M ;

[0010] S5. Calculate the duration d of the silent surface electromyography signal of the i-th frame by using the sound surface electromyography signal and the silent surface electromyography signal. i , the silent surface electromyography signal X of the i-th frame 1:N [i] corresponds to [Y 1：M [j],…,Y 1：M [j+p-1]],d i =p, 1≤i≤N, d 1:N It is d i This step is because the silent surface electromyography signal and the voiced surface electromyography signal are inconsistent in the speaking speed during the subject's repetition, and need to be realigned;

[0011] S6, the silent surface electromyography feature X 1：N Input to the encoder to get the encoded latent variable h 1:N ;

[0012] S7, encode the latent variable h 1:N Input into the persistence length predictor to get the predicted persistence length Calculate the predicted duration The actual frame duration d 1:N Mean Square Error (MSE)

[0013] S8, encode the latent variable h 1:N The actual frame duration d 1:N Input to the length adjuster to adjust the number of frames to obtain the decoding latent variable h 1:M ;

[0014] S9, decode the latent variable h 1:M Input to the phoneme classifier and output the predicted phoneme Calculate predicted phonemes With the real phoneme ph 1：M Cross Entropy Error (CEE)

[0015] S10, decode the latent variable h 1:M Input into the acoustic surface electromyography feature reconstructor, and output the predicted acoustic surface electromyography feature Computational prediction of acoustic surface electromyography characteristics Surface electromyography with sound 1：M Mean square error loss;

[0016] S11, decode the latent variable h 1:M Input to the decoder to get the predicted speech features Then predict the speech features After the post-processing network (post-net), the residual M frame speech features are obtained The predicted M-frame speech features and residual M-frame speech features and the actual speech features Y 1：M Calculate the mean absolute error loss (MAE, Mean Absolute Error);

[0017] S12. Backpropagation is performed according to the loss function to update the training weights of the speech reconstruction model until the speech reconstruction model converges. The loss function formula is:

[0018]

[0019] Where λ ph is the cross entropy loss weight, λ recons is the mean square error loss weight of the voiced signal. Both weights are between 0 and 1 and are obtained through training.

[0020] The reconstruction process is to input the collected silent surface electromyography signal into the speech reconstruction model to obtain the reconstructed speech signal.

[0021] Preferably, the reconstruction process is specifically as follows:

[0022] A1. Collect silent surface electromyography signals through an electrode array attached to the human body surface, pre-process the collected silent surface electromyography signals and divide them into frames to obtain silent surface electromyography features X1′ :n , n is the total number of frames of silent surface electromyography features, and the silent surface electromyography features X1′ :n Input to the encoder to get the encoded latent variable

[0023] A2. Encoding latent variables Input to the persistence length predictor to obtain the predicted persistence length

[0024] A3. Encoding latent variables and the predicted duration Input to the length adjuster to get the decoded latent variable m is the total number of frames of the adjusted decoding latent variable,

[0025] A4. Decode latent variables After inputting into the decoder, the predicted speech features are obtained Predicting speech features Input into post-net to get residual speech features

[0026] A5. Residual speech features The speech signal is input into a vocoder, which can be a neural vocoder such as Griffin-lim, Parallel Wavegan or Wavenet.

[0027] Preferably, the electrodes of the electrode array are silver / silver chloride surface wet electrodes, including a bias electrode, a reference electrode and a collection electrode;

[0028] At least one bias electrode and one reference electrode are placed behind the ear at the mastoid process;

[0029] The acquisition electrodes include any one or more types located in the buccal muscle area, the external tongue muscle area, the zygomatic muscle area, and the external laryngeal muscle area.

[0030] The electrodes placed on the buccal and extrinsic tongue muscles are single electrodes, while those placed on the zygomatic and extrinsic laryngeal muscles are differential electrodes. This means there are two electrodes, and the difference between the two collected signals yields a single signal. The signal acquisition device receives and amplifies the electrical signals transmitted by the electrode array. The signal reconstruction module performs a series of processing on the silent signal, reconstructing it into speech through a trained network. The speech device then plays the reconstructed speech signal, which is received by the human ear.

[0031] Preferably, the preprocessing includes filtering and feature extraction, and the feature extraction is specifically extracting time domain and time-frequency domain features by frame through a moving window.

[0032] Filtering includes but is not limited to power frequency filtering and bandpass filtering. Time domain features mainly include but are not limited to zero crossing, average value, absolute average value and variance. Time-frequency domain features are the amplitude values ​​after performing short-time Fourier transform on the signal within each frame.

[0033] Preferably, step S4 specifically includes: inputting the speech features and text into the forced alignment software MFA (Montreal Forced Aligner) to obtain the time period corresponding to each phoneme, thereby calculating the phoneme corresponding to each frame of speech features.

[0034] Preferably, the encoder includes a fully connected layer, a position embedding layer, and a feedforward converter built based on the Fastspeech framework.

[0035] Preferably, the persistence length predictor includes a convolutional layer, a ReLU activation layer and a fully connected layer, and the number of convolutional layers is between 1 and 3.

[0036] Preferably, the decoder includes a feedforward converter and a fully connected layer built on the Fastspeech framework; the post-net includes a convolutional layer with the number of layers ranging from 1 to 5.

[0037] As a preference, the specific process of the length adjuster is:

[0038] The encoding latent variable h i Copyd i times, and the obtained variable is the decoding latent variable.

[0039] The purpose of the length adjuster is to or ) adjusts the encoding latent variables.

[0040] Preferably, the phoneme classifier and the voiced surface electromyography feature reconstructor are both fully connected layers.

[0041] Preferably, step S5 is specifically as follows:

[0042] S51. Obtain pairs of each frame between the voiced surface electromyography features and the silent surface electromyography features through dynamic time warping (DTW). The loss function of dynamic time warping is as follows:

[0043] ‖X 1:N [i]-x 1:M [j]‖

[0044] S52. Calculate the input-output frame-level correspondence sequence A that minimizes the loss function of dynamic time warping 1:M , where A 1:M [j] represents the number of the silent surface electromyography feature frame corresponding to the j-th frame of the sound surface electromyography feature. The calculation formula for the actual frame duration is:

[0045]

[0046] Among them, d i Also called the real frame duration, where

[0047] As a preference, the loss function of dynamic time warping is updated during training, and the formula is as follows:

[0048]

[0049] in, is the encoding latent variable h 1:NDirectly input into the decoder to obtain N frames of output speech features, λ align is the alignment weight, which is between 5 and 10 and is obtained through training;

[0050] The update of the loss function of dynamic time warping is to train all training data K times and then update it, where the value of K is 5 to 20.

[0051] Preferably, the vocoder needs to be pre-trained with the subject's speech to obtain a pre-trained model of a neural vocoder that converts the subject's speech features into speech.

[0052] Preferably, when the model is trained and used on multiple experimental data, the number of experiments is input into the Embedding network as a label, merged with the input surface electromyography features, and input into the encoder.

[0053] Preferably, when the model is trained and used on multiple experimental data, the persistence lengths obtained in a single experiment can be combined and directly used as the true persistence lengths during the training of multiple experimental data and input into the persistence length predictor without the need for calculation.

[0054] The basic concept of this invention is to use electrodes to collect surface electromyographic signals (SEM) and speech signals during speech production, and to collect SEM signals during silence. Using a deep learning network, the SEM signals are linked to the speech signals of the same text, reconstructing the speech signals in silence and transmitting information in silence. This allows for direct transmission of information in silence. For situations where speech is impossible (e.g., military or police covert operations), the vocal cords remain silent, and information transmission is accomplished through silent reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a block diagram of a speech reconstruction system based on surface electromyography of the present invention;

[0056] Figure 2 It is a flow chart of a model training structure of the present invention;

[0057] Figure 3 It is a model usage structure flow chart of the present invention;

[0058] Figure 4 It is a structural flow chart of the use of a phoneme classifier and a voiced surface electromyography feature reconstructor in a model of the present invention;

[0059] Figure 5 This is a graph showing the results of five single-person experiments conducted by the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention will be further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0061] Example: This embodiment is a silent speech reconstruction system based on surface electromyography signals, such as Figure 1 Shown, including:

[0062] Electrode array 101, which includes at least two electrodes and is placed on the face to collect surface electromyographic signals related to vocalization;

[0063] The microphone 102 is used to collect the voice signal during the audio reading;

[0064] The signal acquisition module 103 amplifies the collected potential signal through an operational amplifier and sends it to the host computer via wired or wireless communication. At the same time, during the audio reading, the voice signal collected by the microphone is sent to the host computer via wired or wireless communication.

[0065] Signal reconstruction module 104, during the training phase, collects surface electromyography signals during spoken word reading, aligns them with the signals collected by the microphone, performs preprocessing and feature extraction, collects surface electromyography signals during silent reading, performs preprocessing and feature extraction, and uses the surface electromyography signal features as input and the speech signal as output to train the network; during the testing phase, only silent surface electromyography signals are collected, preprocessed, feature extracted, and input into the network for reconstruction;

[0066] The voice playing device 105 plays the reconstructed voice through the voice playing device.

[0067] The basic idea of ​​the present invention is to use electrodes to collect surface electromyography from the surface of the vocal organ, and after signal processing, voice reconstruction and other steps, voice can be output even in silent conditions.

[0068] The electrode array is a set of sound-generating surface electromyographic sensors located on and around the face to capture electrical signals from the human body. A microphone is used to collect speech signals during aloud reading. The signal acquisition device receives and amplifies the electrical signals transmitted by the electrode array. The signal reconstruction module performs a series of processing on the silent signal and reconstructs it into speech through a trained network. The speech device then plays the reconstructed speech signal, which is received by the human ear.

[0069] A silent speech reconstruction training method based on surface electromyography signals of the present invention is described in detail. Figure 2 As shown, the following steps are included:

[0070] Step 1: First, the subject reads the text displayed on the screen normally. The electrode array collects the surface electromyography signals of the spoken reading, and the microphone collects the voice signal. The surface electromyography signals of the spoken reading are time-synchronized with the voice signal. Then, the subject recites the same text silently, and the silent surface electromyography signals of the silent recitation are collected.

[0071] Step 2: Signal preprocessing: preprocess and frame the surface electromyography signal and speech signal collected in step 1, where the N-frame silent surface electromyography signal feature is X 1：N , the characteristics of M-frame surface electromyographic signals with sound are x 1：M , the speech signal feature obtained synchronously with the voiced surface electromyography signal is Y 1：M ;

[0072] Step 3: According to the voiced speech signal and its corresponding text pinyin and phoneme, the voiced speech signal Y of each frame is obtained by forced alignment tool 1：M The corresponding phoneme ph 1：M ;

[0073] Step 4: Since the speech speed of the silent surface electromyography signal and the voiced surface electromyography signal is inconsistent during the subject's repetition, they need to be realigned. 1：M With silent surface electromyography feature X 1：N Calculate the duration d of the i-th frame of the silent surface electromyography feature i , 1≤i≤N, silent feature i-th frame X 1：N [i] The corresponding vocal feature Y M middle[Y j ,…,Y j+p-1 ], d i =p then X 1：N [i] The corresponding duration length.

[0074] Step 4.1: Get the pairs of frames between the two time series through DTW (Dynamic Time Warping):

[0075] The loss function of DTW is as follows:

[0076] ||X 1：N [i]-x 1：M [j]||

[0077] Step 4.2: According to the above loss function, calculate the input-output frame-level correspondence sequence that minimizes the loss function, that is, A 1：M , where A 1：M [j] represents the number of the silent feature frame corresponding to the j-th frame with voice feature. The calculation formula for the actual frame duration is as follows:

[0078]

[0079] Among them, d i Also called the real frame duration, where

[0080] Step 5: Reconstruction network training for converting surface electromyography features into speech features:

[0081] Step 5.1: Set the surface electromyography feature X in step 2 1：N Input to the encoder to get the encoded latent variable h 1：N ;

[0082] Step 5.2: Encode the latent variable h described in step 5.1 1：N Input to the persistence length predictor, which outputs the predicted persistence length The actual frame duration d obtained in step 4 1：N Calculate MSE (Mean Square Error, mean square error loss);

[0083] Step 5.3: Encode the latent variable h described in step 5.1 1：N The actual frame duration d obtained in step 4 1：N Input to the length adjuster to adjust the number of frames to obtain the decoding latent variable h 1：M ;

[0084] Step 5.4: Decode the latent variable h described in step 5.3 1：M Input to the phoneme classifier and output the predicted phoneme and the real phoneme frame ph obtained in step 3 1：M Calculate CEE (Cross Entropy Error, cross entropy loss), such as Figure 4 As shown;

[0085] Step 5.5: Decode the latent variable h described in step 5.3 1：M Input into the voiced surface electromyography feature reconstructor and output the predicted phoneme The actual corresponding acoustic surface electromyography feature x obtained in step 3 1：M Calculate MSE as Figure 4 As shown;

[0086] Step 5.6: Decode the latent variable h described in step 5.3 1：M Input to the decoder to get the predicted speech features Then After post-net, the residual M frame speech features are obtained Will The actual speech features Calculate MAE (Mean Absolute Error);

[0087] Step 5.7: Update the DTW loss function during training. The formula is as follows:

[0088]

[0089] in, For input feature X during training 1：N Enter the encoder to get the hidden variable h 1：N , and then h 1：N Then it is directly input into the decoder without being adjusted by the length regulator to obtain N frames of output speech features, λ align is the alignment weight. The persistence length is calculated as described in step 4.2;

[0090] Step 5.8: The loss function formula of the multi-task training model is as follows:

[0091]

[0092] Step 6: Input the silent surface electromyography features into the network to directly obtain the speech features;

[0093] In step 1, the basic process of collecting surface electromyographic signals during aloud reading and silent recitation is as follows:

[0094] Step 1.1: The subject cleans his / her face, applies electrodes, and wears the microphone.

[0095] Step 1.2: Debug the sEMG acquisition system hardware and software.

[0096] Step 1.3: The subject reads the sentence normally according to the screen display, and the mouse controls the start and end;

[0097] Step 1.4: The subject reads the sentence displayed on the screen normally, with the mouse controlling the start and end, avoiding lip movement and only moving the tongue, reading silently, and recording the surface electromyography signal, with the mouse controlling the start and end. This step is repeated 5 times.

[0098] Step 1.5: Change the sentence content and repeat steps 1.3 and 1.4.

[0099] The electrode array described in step 1 is a set of sensors that capture electrical signals from the human body. The signal acquisition device receives the electrical signals sent by the electrode array and amplifies them. The signal reconstruction module performs a series of processing on the silent signal and reconstructs it into speech through the trained network. The speech device plays the reconstructed speech signal, which is received by the human ear.

[0100] The electrode array electrodes described in step 1 are silver / silver chloride surface wet electrodes, including a bias electrode, a reference electrode, and a collection electrode;

[0101] At least one bias electrode and one reference electrode are placed behind the ear at the mastoid process;

[0102] The collecting electrodes include at least one of the following types: located in the buccal muscle area, the external tongue muscle area, the zygomatic muscle area, and the external laryngeal muscle area;

[0103] The electrodes placed on the buccal and extrinsic tongue muscles are single electrodes, while the electrodes placed on the zygomatic and extrinsic laryngeal muscles are differential electrodes, i.e., there are two electrodes, and the difference between the two collected signals is used to obtain one signal.

[0104] The step 2 comprises the following steps:

[0105] Step 2.1: Filtering the surface electromyography signal, including but not limited to power frequency filtering and bandpass filtering;

[0106] Step 2.2: Extract features from the surface EMG signal. Specifically, extract time-frequency and time-frequency domain features frame by frame using a moving window. Time-domain features include, but are not limited to, zero crossings, mean, absolute mean, and variance. Time-frequency domain features are the amplitude values ​​obtained by performing a short-time Fourier transform on the signal within each frame.

[0107] In step 3, the subject's speech and text are input into the forced alignment software Montreal ForcedAligner (MFA) to obtain the time period corresponding to each phoneme, thereby calculating the phoneme corresponding to each frame of speech feature;

[0108] The encoder in step 5.1 includes a fully connected layer, a position embedding layer, and an FFT (feed-forward transformer) built on the Fastspeech framework;

[0109] The decoder of step 5.6 includes an FFT (feed-forward transformer) and a fully connected layer built on the Fastspeech framework;

[0110] The post-net of step 5.6 includes a convolutional layer;

[0111] The DTW loss function in step 5.7 is updated by training all training data K times before updating, where the value of K is between 5 and 20;

[0112] Described step 6 comprises the following steps, as Figure 3 As shown:

[0113] Step 6.1: Silent Surface EMG Characterization X 1：N Input to the encoder to get the encoded latent variable

[0114] Step 6.2: Encode the latent variables described in step 6.1 Used as input to the pre-trained duration predictor to obtain the predicted duration

[0115] Step 6.3: Encode the latent variables described in step 6.2 and the persistence length predicted by the persistence length predictor Input to the length adjuster to get the decoded latent variable

[0116] Step 6.4: Decode the latent variables described in step 6.3 After inputting into the decoder, the predicted speech features are obtained Will Input into post-net to obtain residual M frame speech features

[0117] Step 6.5: The residual M-frame speech features described in step 6.4 The speech signal is obtained by inputting it into a vocoder, wherein the vocoder used can be a traditional Griffin-lim vocoder or a neural vocoder such as Parallel Wavegan or Wavenet;

[0118] To make the neural vocoder in step 6, it is necessary to pre-train the subject's speech to obtain a pre-trained model of the neural vocoder that converts the subject's speech features into speech;

[0119] When the model is trained and used on multiple experimental data, the number of experiments is input into the Embedding network as a label, merged with the input surface electromyography features, and input into the encoder;

[0120] When the model is trained and used on multiple experimental data, the persistence lengths obtained in a single experiment can be directly combined as the true persistence lengths during training on multiple experimental data and input into the persistence length predictor without the need for calculation.

[0121] The speech reconstruction effect of this example is as follows: Figure 5 As shown in the figure, the reconstructed speech of the 8 subjects' dictation were 5 speech sounds in the test set of each experiment. In the five mixed experiments, the reconstructed speech of the 8 subjects' dictation were 25 speech sounds in the test set. The subjects had no prior knowledge of the test speech sounds, and the average spelling error rate in a single experiment was 3.39%.

[0122] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

[0123] Although the terms "preprocessing" and "vocoder" are frequently used herein, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.

Claims

1. A silent speech reconstruction method based on vocal surface electromyography signals, based on a speech reconstruction model, characterized in that: Including model training process and reconstruction process, the model training process is as follows: S1: The subject reads the text content normally. The electrode array attached to the body surface collects the acoustic surface electromyographic signals during the reading, and the microphone simultaneously collects the voice signals of the reading; S2: The subject silently reads the same text as in step 1, and the electrode array collects silent surface electromyographic signals during silent reading; S3. Preprocess and frame the voiced surface electromyography signal, the silent surface electromyography signal and the speech signal. The voiced surface electromyography feature obtained after the voiced surface electromyography signal is framed is recorded as x 1：M , M is the total number of frames of the voiced surface electromyography feature; the speech feature obtained after the speech signal is framed is recorded as Y 1：M , the number of frames of speech features is the same as that of voiced surface electromyography features; the silent surface electromyography features obtained after the silent surface electromyography signal is framed are recorded as X 1：N , N is the total number of frames of silent surface EMG features; S4, according to the speech signal and its corresponding text, obtain the real phoneme ph corresponding to the speech signal by forced alignment 1：M ; S5. Calculate the duration d of the silent surface electromyography signal of the i-th frame by using the sound surface electromyography signal and the silent surface electromyography signal. i , the silent surface electromyography signal X of the i-th frame 1:N [i] corresponds to [Y 1：M [j],…,Y 1：M [j+p-1]],d i =p,d 1:N It is d i A collection of S6, the silent surface electromyography feature X 1：N Input to the encoder to get the encoded latent variable h 1:N ; S7, encode the latent variable h 1:N Input into the persistence length predictor to get the predicted persistence length Calculate the predicted duration The actual frame duration d 1:N Mean square error loss; S8, encode the latent variable h 1:N The actual frame duration d 1:N Input to the length adjuster to adjust the number of frames to obtain the decoding latent variable h 1:M ; S9, decode the latent variable h 1:M Input to the phoneme classifier and output the predicted phoneme Calculate predicted phonemes With the real phoneme ph 1：M Cross entropy loss; S10, decode the latent variable h 1:M Input into the acoustic surface electromyography feature reconstructor, and output the predicted acoustic surface electromyography feature Computational prediction of acoustic surface electromyography characteristics Surface electromyography with sound 1：M Mean square error loss; S11, decode the latent variable h 1:M Input to the decoder to get the predicted speech features Then predict the speech features After the post-processing network, the residual M-frame speech features are obtained The predicted M-frame speech features and residual M-frame speech features and the actual speech features Y 1：M Calculate mean absolute error loss; S12. Backpropagation is performed according to the loss function to update the training weights of the speech reconstruction model until the speech reconstruction model converges. The loss function formula is: Where λ ph is the phoneme classification loss weight, λ recons Reconstruct loss weights for voiced signals; The reconstruction process is to input the collected silent surface electromyography signal into the speech reconstruction model to obtain the reconstructed speech signal.

2. The method for reconstructing silent speech based on surface electromyographic signals according to claim 1, characterized in that: The reconstruction process is specifically as follows: A1. Collect silent surface electromyography signals through an electrode array attached to the human body surface, pre-process the collected silent surface electromyography signals and divide them into frames to obtain silent surface electromyography features X1′ :n , n is the total number of frames of silent surface electromyography features, and the silent surface electromyography features X1′ :n Input to the encoder to get the encoded latent variable A2. Encoding latent variables Input to the persistence length predictor to obtain the predicted persistence length A3. Encoding latent variables and the predicted duration Input to the length adjuster to get the decoded latent variable m is the total number of frames of the adjusted decoding latent variable, A4. Decode latent variables After inputting into the decoder, the predicted speech features are obtained Predicting speech features Input into the post-processing network to obtain residual speech features A5. Residual speech features Input into the vocoder to obtain the speech signal.

3. A method for reconstructing silent speech based on vocalized surface electromyography signals according to claim 1 or 2, characterized in that: The electrodes of the electrode array are silver / silver chloride surface wet electrodes, including a bias electrode, a reference electrode and a collection electrode; At least one bias electrode and one reference electrode are placed behind the ear at the mastoid process; The acquisition electrodes include any one or more types located in the buccal muscle area, the external tongue muscle area, the zygomatic muscle area, and the external laryngeal muscle area.

4. A method for reconstructing silent speech based on vocalized surface electromyographic signals according to claim 1 or 2, characterized in that: The preprocessing includes filtering and feature extraction. Feature extraction specifically involves extracting time domain and time-frequency domain features by frame through a moving window.

5. A method for reconstructing silent speech based on vocalized surface electromyography signals according to claim 1 or 2, characterized in that: The encoder includes a fully connected layer, a position embedding layer, and a feedforward converter built on the Fastspeech framework.

6. The method for reconstructing silent speech based on vocalized surface electromyographic signals according to claim 5, characterized in that: The persistence length predictor includes a convolutional layer, a ReLU activation layer, and a fully connected layer, and the number of convolutional layers is between 1 and 3.

7. The method for reconstructing silent speech based on vocalized surface electromyography signals according to claim 6, characterized in that: The decoder includes a feedforward converter and a fully connected layer built on the Fastspeech framework; the post-processing network includes convolutional layers, with the number of layers ranging from 1 to 5.

8. The method for reconstructing silent speech based on vocalized surface electromyography signals according to claim 1, characterized in that: The specific process of the length adjuster is: The encoding latent variable h i Copyd i times, and the obtained variable is the decoding latent variable.

9. The method for reconstructing silent speech based on vocalized surface electromyography signals according to claim 1, characterized in that: Step S5 is specifically as follows: S51. Pairs of frames between the voiced surface electromyography features and the silent surface electromyography features are obtained through dynamic time warping. The loss function of dynamic time warping is as follows: ‖X 1:N [i]-x 1:M [j]‖ S52. Calculate the input-output frame-level correspondence sequence A that minimizes the loss function of dynamic time warping 1:M , where A 1:M [j] represents the number of the silent surface electromyography feature frame corresponding to the j-th frame of the sound surface electromyography feature. The calculation formula for the actual frame duration is: Among them, d i Also called the real frame duration, where 10. The method for reconstructing silent speech based on vocalized surface electromyographic signals according to claim 9, characterized in that: During the training process, the loss function of dynamic time warping is updated as follows: in, is the encoding latent variable h 1:N Directly input into the decoder to obtain N frames of output speech features, λ align is the alignment weight; The update of the loss function of dynamic time warping is to train all training data K times and then update it, where the value of K is 5 to 20.

Images