A tongue ultrasound image synthesis speech method of a self-supervised model

Abstract

The tongue ultrasonic image synthesis speech method of the self-supervised model belongs to the electronic information technology field. The tongue ultrasonic image synthesis speech method of the self-supervised model aims to find out the characteristics of tongue movement when a person speaks directly through an ultrasonic imaging method through deep learning technology, establish a mapping relationship between ultrasonic tongue images and speech, and thus realize end-to-end speech synthesis. The method is not easily affected by the environment and has strong confidentiality. In view of the problem that a large amount of supervised data is required during training and the model is difficult to converge, a self-supervised learning framework is adopted. The self-supervised learning framework can mask part of the video or audio input during the training stage and predict the automatically discovered and iteratively refined multi-modal hidden unit, enhance the generalization ability of the model, appropriately reduce the training data used, and make the model more easily converge.

Description

Technical Field

[0001] This case involves electronic information technology, and in particular, a self-supervised model for synthesizing speech from tongue ultrasound images. Background Technology

[0002] The basic principle of silent speech interface technology is as follows: Sensors acquire motion signals from human vocal organs such as the tongue and lips, process and encode these signals, then select an appropriate model and algorithm for decoding, and finally synthesize and reconstruct speech. This achieves the conversion of non-acoustic signals into acoustic signals, enabling silent communication.

[0003] In recent years, numerous studies have emerged addressing issues related to silent speech interface technology. One approach involves speech synthesis based on optical images of the lips or ultrasound images of the tongue from video streams. This method acquires data non-invasively, is clinically safe, and can obtain a large amount of natural data for training. However, directly synthesizing speech from lip-reading videos is susceptible to environmental influences when interacting with smart devices, presenting limitations; for example, it may not perform well in poor lighting conditions or when the user is wearing a mask. Furthermore, for individuals with lip-reading experience, directly using lip-reading videos carries the risk of personal information leakage, resulting in weak confidentiality. Such technologies typically employ encoder-decoder frameworks with sequence models as a solution. However, training these neural networks often requires extensive supervised training with synchronized video and audio data, and the models often struggle to converge. Summary of the Invention

[0004] To address the aforementioned issues, this project aims to utilize deep learning technology to directly identify the characteristics of tongue movement during speech through ultrasound imaging, establishing a mapping relationship between ultrasound tongue images and speech, thereby achieving end-to-end speech synthesis. This technical solution is less susceptible to environmental influences and offers strong confidentiality. To address the challenges of requiring extensive supervised data during training and the difficulty of model convergence, a self-supervised learning framework is introduced, along with a multimodal hidden unit designed for automatic prediction discovery and iterative refinement. This unit is used to extract some effective features from image or acoustic features after masking operations. Through training, it can accurately reconstruct complete acoustic features based on these effective features, thereby enhancing the model's generalization ability, appropriately reducing the amount of training data used, and making the model converge more easily.

[0005] Firstly, this case proposes a self-supervised model for synthesizing speech from tongue ultrasound images, the method comprising the following steps:

[0006] Acquire a sequence of ultrasound images of the tongue, use a trained student network to predict the acoustic features corresponding to the sequence of ultrasound images of the tongue, and synthesize speech.

[0007] During training, the student network is trained in conjunction with the teacher network, forming a self-supervised learning framework. It employs a masking strategy to automatically discover and predict useful information in the unlabeled dataset.

[0008] In the above technical solution, deep learning technology is used to establish a mapping relationship between ultrasound tongue images and speech using a trained student network. This allows for the direct acquisition of tongue movement features during speech from ultrasound imaging, thus achieving end-to-end speech synthesis. The method acquires data non-invasively, is less susceptible to environmental influences, and offers strong confidentiality. Furthermore, a self-supervised learning framework addresses the need for supervision during training, reducing the workload of manual annotation.

[0009] In one implementation, the specific structure of the student network and the teacher network is as follows: both include an encoder, a decoder, and an audio feature extraction unit; the audio feature extraction unit is used to separate the audio portion in the tongue ultrasound video and convert the audio waveform into a Mel spectrum; the encoder uses a 3D-CNN network to extract visual features of the tongue image; the decoder uses a Transformer module to process the hidden state feature vector output by the encoder, and the hidden feature vector is used to infer the Mel spectrum and synthesize speech.

[0010] As a further improvement to the above technical solution, by masking some visual features of the video or Mel-spectral features of the audio during the training phase, multimodal hidden units that are automatically discovered through prediction are designed to enhance the model's generalization ability. Through iterative refinement, the amount of training data used is appropriately reduced, making the model converge more easily. A specific implementation of this improvement is as follows:

[0011] Set up the first audiovisual fusion module in the student network, before the Transformer module in the student network;

[0012] A second audiovisual fusion module is set up in the teacher network, before the Transformer module in the teacher network;

[0013] The first audiovisual fusion module and the second audiovisual fusion module are used to receive two modal data: visual features and visual feature-corresponding Mel spectrum data.

[0014] During training, two modalities of data are input to the first audiovisual fusion module, processed by random masking, and then selectively discarded by the first audiovisual fusion module. In one embodiment, the first audiovisual fusion module selectively uses one of the modalities of data using the following probability calculation formula:

[0015]

[0016] in, This indicates the final feature sequence used. For audio feature sequences, For video feature sequences, `concat()` indicates concatenation by channel, p m To determine the probabilities of using both modes, p a This represents the probability of using the audio modality when only one modality is used.

[0017] As a further improvement to the above technical solution, random noise is added to the audio training input data of the student network to enhance the robustness of the model. That is, the training input data of the student network consists of audio with added random noise and tongue image sequences that correspond one-to-one with the audio before adding random noise, while the training input data of the teacher network consists of completely clean audio. That is, the input data of the teacher network consists of pure audio and tongue image sequences that correspond one-to-one with the pure audio.

[0018] In the above technical solution, one implementation method for training the student network and the teacher network is as follows:

[0019] The teacher network generates learning objectives, and the student network generates predicted objectives. The error between the learning objectives and the predicted objectives is calculated using the following loss function:

[0020]

[0021] Where: x t y is the predicted target value for the student network. t M represents the learning objective value of the teacher network. a and M v These are the mask data for audio and video, respectively;

[0022] The student network updates its gradient using the backpropagation algorithm, while the gradient of the teacher network remains fixed.

[0023] The parameter updates for the teacher network only occur in the Transformer module. During the update process, the parameters are updated based on the Transformer module in the student network.

[0024] In one embodiment of the above technical solution, the Transformer module in the teacher network is transmitted by the Transformer module in the student network using exponentially moving average (EMA), and the update formula is:

[0025]

[0026] Where, θ i and Let i represent the Transformer module parameters in the teacher network and the Transformer module parameters in the student network, respectively. Let i be the update count identifier and λ be the hyperparameter controlling the update count.

[0027] In one embodiment of the above technical solution, the output features of the Transformer module in the teacher network are processed by regularization and averaged.

[0028] In the above technical solution, the acoustic features output by the decoder, and the audio waveform of the synthesized speech, satisfy the conditional probability P(W|U):

[0029] P(W|U)=Decoder(h u )

[0030] in:

[0031] P(w|U)=Π k (w k |w k＜t U)

[0032] In the formula, w k For the audio waveform with time step k, w k＜t Let t be the audio waveform up to time step k, t be the number of time steps up to time step k, U be the input image sequence of the encoder, W be the audio waveform sequence synchronized with the image sequence U in time, and h be the audio waveform sequence synchronized with the image sequence U in time. u The hidden state features are extracted by the encoder based on the image sequence U.

[0033] Secondly, this case proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the methods described above. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A schematic diagram of the process of synthesizing speech from tongue ultrasound images in the implementation method;

[0036] Figure 2 A schematic diagram of a self-supervised learning framework consisting of a "teacher network" and a "student network" in the implementation method. Detailed Implementation

[0037] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0038] Image and video stream-based data acquisition methods are non-invasive, clinically safe, and naturally generate large datasets with numerous training samples. However, methods that directly synthesize speech from lip-reading videos are susceptible to environmental influences when interacting with smart devices, presenting limitations; for example, they may not perform well in poor lighting conditions or when the user is wearing a mask. Furthermore, for individuals with lip-reading experience, directly using lip-reading videos carries the risk of personal information leakage, resulting in weak confidentiality. After careful consideration, this invention uses only ultrasound tongue images as the dataset source. Such techniques typically employ sequence models with an encoder-decoder framework as a solution. However, training these neural networks often requires extensive supervised training with synchronized video and audio data, and the models often struggle to converge.

[0039] In one implementation, a self-supervised model-based method for synthesizing speech from tongue ultrasound images is used to address the aforementioned technical problems. The method is as follows:

[0040] Acquire a sequence of ultrasound images of the tongue, use a trained student network to predict the acoustic features corresponding to the sequence of ultrasound images of the tongue, and synthesize speech.

[0041] During training, the student network is trained in conjunction with the teacher network, forming a self-supervised learning framework. It employs a masking strategy to automatically discover and predict useful information in the unlabeled dataset.

[0042] The principle of acoustic feature prediction based on tongue ultrasound image sequences is as follows: given an audio waveform sequence W = [w1, w2, ..., w...] with a time length of t... t ] and the sequence of ultrasound images of the tongue from the same speaker U = [u1, u2, ..., u t These two sequences are consistent and synchronized in time. The audio waveform output at the current time step k is w. k It can be represented as the previous time step w k＜t The conditional probability distribution of the ultrasound image sequence U, i.e.:

[0043] P(w|U)=Π k (w k |w k＜t U)

[0044] Therefore, after acquiring ultrasound images of the tongue, visual features are extracted from the tongue ultrasound image sequence using deep learning neural network technology, and the corresponding acoustic features are inferred using the learned conditional probability P(W|U). Based on these acoustic features, audible sounds can be synthesized.

[0045] During the procedure, tongue ultrasound images can be acquired from tongue ultrasound videos. An appropriate frame rate is selected to convert the video into an image sequence and label it with a number. For example... Figure 1 As shown, visual features are extracted using a visual encoder. The encoder converts the input image sequence into a hidden state vector h, and the total output of the encoder is a vector H = [h1, h2, ..., h2]. t This will be fed into the attention module of the acoustic decoder, which outputs acoustic features. Both the encoder and decoder are deep learning neural network models. The acoustic features output by the decoder, and the audio waveform of the synthesized speech, satisfy the conditional probability distribution P(W|U):

[0046] P(W|U)=Decoder(h u )

[0047] And h u The hidden state features extracted by the encoder based on the image sequence U:

[0048] h u =Encoder(U)

[0049] Synchronized audio waveform and image sequences can be obtained by acquiring ultrasound video of the tongue with sound using a B-mode ultrasound machine. As mentioned above, the image sequence can be obtained by converting the video at an appropriate frame rate, while the audio waveform sequence can be obtained by first separating the audio portion of the video and then converting the audio waveform into a Mel spectrum with richer acoustic features. Each video recording corresponds one-to-one with an image sequence and a Mel spectrum, which is used as the dataset for model training. For the Mel spectrum, Mel Frequency Cepstral Coefficients (MFCCs) are used to extract acoustic features; the formula for these coefficients is:

[0050] Mel(f) = 2595(1 + f / 700)

[0051] In the formula, f is the audio frequency.

[0052] To address the challenges of requiring extensive supervised data during training and the difficulty of model convergence, a self-supervised learning framework is adopted. This framework uses a masking strategy to automatically discover and predict useful information in unlabeled datasets, replacing tedious manual labeling.

[0053] During implementation, the following methods are adopted: Figure 2The self-supervised learning framework shown includes a student network and a teacher network, both with almost identical structures. Both are cross-modal deep learning models based on an encoder-decoder architecture. Each includes an encoder, a decoder, and an audio feature extraction unit. Specifically: the audio feature extraction unit separates the audio portion from the tongue ultrasound video, converting the audio waveform into a Mel spectrum; the encoder uses a 3D-CNN network to extract visual features from the tongue image; and the decoder uses a Transformer module to process the hidden state feature vector output by the encoder. This hidden feature vector is used to infer the Mel spectrum and synthesize speech.

[0054] The difference lies in the approach: the student network includes a first audiovisual fusion module, located before the Transformer module; the teacher network includes a second audiovisual fusion module, located before the Transformer module. Both modules receive two modalities: visual features and their corresponding Mel-spectrum data. During training, the two modalities input to the first audiovisual fusion module are randomly masked and selectively discarded. In other words, during training, the student network input includes two modalities: audio with added random noise to increase model robustness, and a sequence of tongue ultrasound images. These two modalities are concatenated through the first audiovisual fusion module. Before concatenation, some data features of both modalities are randomly masked—randomly selecting positions in the input sequence and replacing the data at those positions with a special marker indicating that the data at that position is masked, thus partially masking some data features. After concatenation, one or both modalities are selected for use with a certain probability.

[0055]

[0056] in, This indicates the final feature sequence used. For audio feature sequences, For video feature sequences, `concat()` indicates concatenation by channel, p m To determine the probabilities of using both modes, p a This represents the probability of using the audio modality when only one modality is used.

[0057] The teacher network is trained on clean audio data and corresponding tongue ultrasound image sequences. Furthermore, the encoder outputs two modal data streams without masking, and the concatenated modal data are not discarded. Additionally, the Transformer module's output features are regularized and averaged to accelerate model convergence and prevent overfitting.

[0058] During training, the teacher network generates a complete learning objective, and the student network generates a predicted objective. The acoustic feature structures of the learning objective and the predicted objective are compared, and the error between the two is calculated using the following loss function:

[0059]

[0060] Where: x t y is the predicted target value for the student network. t M represents the learning objective value of the teacher network. a and M v These are the mask data for audio and video, respectively. Specifically, the student network updates its gradients via backpropagation, while the teacher network's gradients remain constant. The teacher network's parameter updates occur only within the Transformer module, propagated by the student module through exponentially moving average (EMA).

[0061] The updated formula is:

[0062]

[0063] Where, θ i and Let i represent the Transformer module parameters in the teacher network and the Transformer module parameters in the student network, respectively. Let i be the update count identifier and λ be the hyperparameter controlling the update count.

[0064] Let λ b As the initial value, λ e The final value is N, and the total number of updates is N.

[0065]

[0066] Where, λ i This is the current value.

[0067] exist Figure 2 In this model, the entire part after masking (weights), 3D convolutional modules, and before the transformer module can be considered a multimodal hidden unit (MLU). The MLU is used to discover some effective features within the masked image or acoustic features. Through iterative learning during training, it automatically predicts the relationship between some effective features and the complete acoustic features, gradually improving accuracy and enhancing the model's generalization ability. This allows for a more appropriate reduction in the amount of training data used and makes the model converge more easily.

[0068] After model training, ultrasound video of the tongue without speech is used for prediction to generate a Mel-spectral prediction map for speech reconstruction. The Griffin-Lim algorithm is then used to reconstruct the time-domain signal from the predicted Mel-spectrum to obtain the audible sound waveform.

[0069] Based on the implementation process of the above method, it can be further implemented as a self-supervised model tongue ultrasound image synthesis speech system. Furthermore, through the description of the above embodiments, those skilled in the art can clearly understand that the method or system of this disclosure can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for this disclosure, software program implementation is more often a preferred implementation method.

[0070] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A self-supervised model for synthesizing speech from tongue ultrasound images, characterized in that, The method is as follows: By acquiring a sequence of ultrasound images of the tongue, a trained student network is used to predict a segment of acoustic features corresponding to the sequence of ultrasound images of the tongue, and then speech is synthesized. During training, the student network is trained in conjunction with the teacher network, forming a self-supervised learning framework. It uses a masking strategy to automatically discover and predict useful information in the unlabeled dataset. Both the student network and the teacher network include encoders, decoders, and audio feature extraction units. An audio feature extraction unit is used to separate the audio portion of the tongue ultrasound video and convert the audio waveform into a Mel spectrum. The encoder uses a 3D-CNN network to extract visual features from the tongue image; The decoder uses the Transformer module to process the hidden state feature vector output by the encoder, which is used to infer the Mel spectrum and synthesize speech; The student network also includes a first audiovisual fusion module, which is located before the Transformer module in the student network; The teacher network also includes a second audiovisual fusion module, which is located before the Transformer module in the teacher network; The first audiovisual fusion module and the second audiovisual fusion module are used to receive two modal data: visual features and visual feature-corresponding Mel spectrum data. During training, the two modal data input to the first audiovisual fusion module are processed by random masking and then selectively discarded by the first audiovisual fusion module.

2. The method according to claim 1, characterized in that: The training input data for the student network are audio with added random noise and tongue image sequences that correspond one-to-one with the audio before adding random noise. The training input data for the teacher network are pure audio and tongue image sequences that correspond one-to-one with the pure audio.

3. The method according to claim 1, characterized in that, The training implementation process for the student network and teacher network is as follows: The teacher network generates learning objectives, and the student network generates predicted objectives. The error between the learning objectives and the predicted objectives is calculated using the following loss function: Where: x t y is the predicted target value for the student network. t M represents the learning objective value of the teacher network. a and M v These are the mask data for audio and video, respectively; The student network updates its gradient using the backpropagation algorithm, while the gradient of the teacher network remains fixed. The parameter updates for the teacher network only occur in the Transformer module. During the update process, the parameters are updated based on the Transformer module in the student network.

4. The method according to claim 1, characterized in that, The Transformer module in the teacher network is propagated by the Transformer module in the student network using an exponentially moving average (EMA), with the update formula as follows: Where, θ i and Let i represent the Transformer module parameters in the teacher network and the Transformer module parameters in the student network, respectively. Let i be the update count identifier and λ be the hyperparameter controlling the update count.

5. The method according to claim 1, characterized in that, The output features of the Transformer module in the teacher network are processed by averaging after regularization.

6. The method according to claim 1, characterized in that, The acoustic features output by the decoder, and the audio waveform of the synthesized speech, satisfy the conditional probability P(W|U): P(W|U)=Decoder(h u ) in: PW|U)=Π k (w k |w k<t ,U) In the formula, w k For the audio waveform with time step k, w k<t Let t be the audio waveform up to time step k, t be the number of time steps up to time step k, U be the input image sequence of the encoder, W be the audio waveform sequence synchronized with the image sequence U in time, and h be the audio waveform sequence synchronized with the image sequence U in time. u The hidden state features are extracted by the encoder based on the image sequence U.

7. The method according to claim 1, characterized in that, The first audiovisual fusion module selectively uses one or more modal data using the following probability calculation formula: in, This indicates the final feature sequence used. For audio feature sequences, For video feature sequences, `concat()` indicates concatenation by channel, p m To determine the probabilities of using both modes, p a This represents the probability of using the audio modality when only one modality is used.

8. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed according to any one of claims 1 to 7.

Images