An ultrasonic tongue silent speech recognition method based on modal transfer learning

Abstract

The application discloses an ultrasonic tongue silent speech recognition method based on modal transfer learning, relates to the technical field of speech information processing, and proposes a multi-stage and multi-modal modeling pipeline for two main challenges of SSR, namely, a limited data set and a confusing silent signal. Four-step processes are introduced to avoid overfitting and two additional modules are used to extract multi-modal information. Based on a joint training strategy, modal preference learning (MPL) is proposed to promote the optimization of the target task by utilizing cross-modal prior knowledge.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech information processing, and particularly relates to an ultrasonic tongue silent speech recognition method based on modal transfer learning. BACKGROUND

[0002] Speech and language disorders have a profound impact on the lives of those who suffer from them, causing them to adapt to daily communication routines. Silent speech recognition systems (SSR) rely on non-acoustic signals during the speech production process for recognition. For example, observing the movement of multiple organs during the speech production process, including the tongue, lips, chin, vocal cords, and lungs. It currently has a wide range of applications, such as conversing with people who cannot produce sound (e.g., phonemes), visualizing rehabilitation training to restore oral communication for patients with dysarthria, and exploring auxiliary technologies for multi-modal speech perception. In noisy environments or environments where it is inconvenient to communicate by speech, the speech signal will be severely weakened, thereby reducing the recognition rate. However, the silent speech signal required by SSR will not be affected. Many researchers use ultrasound tongue imaging (UTI) technology to develop silent speech communication systems, which are portable, non-invasive, clinically safe, and relatively inexpensive compared to other sensing technologies. Common SSR systems based on lip movements cannot capture anything other than oral movements, and by using UTI technology to monitor internal oral movements, complementary data is provided. Recently, SSR from ultrasound images has been used for various applications such as restoring speech communication for users with speech pairs, allowing silent communication in cases where hearing speech is not ideal, speech therapy, language learning, etc. (Ball, Martin J., ed. Manual of clinical phonetics. Routledge, 2021.).

[0003] In the past few years, as deep learning techniques have made extraordinary achievements in various fields, the focus of SSR research has also changed. Instead of manually designing some feature extraction algorithms to extract features, researchers use the powerful representation learning ability of deep networks to automatically learn good features according to the task target. Asael et al. proposed LipNet based on spatiotemporal convolutional networks and recurrent neural networks in 2017, and used CTC as the network loss function in LipNet; the WLAS network proposed by Chung et al. greatly improved the baseline performance; in 2017, a breakthrough was made on the sentence-level SSR recognition task with 10,000 samples, which was composed of CNN and recurrent neural network (RNN); in 2020, Nookiet al. first tried to apply end-to-end speech recognition to synchronous lip movement videos and other silent speech signals; Wang et al. trained the network to predict future frames in ultrasonic and optical lip images, and created features for a continuous speech recognition system based on HMM. However, due to two reasons, SSR is still a challenging task. The first is the lack of large-scale silent speech corpus. Most of the research in this field today uses small corpus recorded by internal biological signal recording devices, so the performance of the SSR model is not good enough. The second is that the information from a single non-acoustic modality is not enough to fully characterize speech.

[0004] Inspired by the fact that speech perception is a multi-modal process, one potential solution to the above challenges is to utilize information from multiple modalities, such as electroencephalography (EEG) signals, lip movement signals, and surface electromyography (sEMG) signals. Multi-modal SSR aims to match the dynamic relationship between the text and time-aligned records of multi-modal signals by utilizing the complementary information of each modality. Nevertheless, this system requires all inputs in the selected modalities to work properly, so it cannot work if any of them is missing. To overcome this limitation, another potential solution attempts to provide cross-modal information when only one input is available by finding a shared latent space of multi-modal knowledge. Thanks to the development of current multi-modal modeling techniques, there are many solutions to handle this kind of problem, for example, knowledge distillation learns cross-modal knowledge by encoding three modalities (audio, video, and their combination) as a "master" network, which is then used to help "student" network training (Ren, Sucheng, et al. "Learning from the master: Distilling cross-modal advanced knowledge for lipreading." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021.); based on memory networks, it is able to utilize cross-modal information augmented single input and handle the scenario of missing modalities at test time, etc. (Kim, Minsu, et al. "Cromm-vsr: Cross-modal memory augmented visual speech recognition." IEEE Transactions on Multimedia 24 (2021): 4342-4355.). However, the above methods often require large-scale synchronized audio-visual signals, and it is difficult to model the complex many-to-many relationship between multi-modalities. SUMMARY

[0005] The present application aims to provide an ultrasound tongue silent speech recognition method based on modal transfer learning to solve the problems raised in the above background art.

[0006] To alleviate the above two challenges in SSR, we propose a multi-stage multi-modal modeling pipeline that optimizes a unified learning objective for both silent speech streams and other auxiliary streams. During training, the fusion module appropriately augments the limited information of silent speech samples by learning multi-modal prior knowledge. Specifically, we first implement a UTI-based SSR baseline by introducing a four-step procedure to avoid overfitting due to the lack of large-scale data. To handle the many-to-many association between text and UTI signals, the model is further fine-tuned based on a joint training strategy from three modalities: UTI modality, additional modality, and their combination. Meanwhile, considering the uncoordinated convergence phenomenon in multi-modal training, we also introduce modal preference learning (MPL) to converge the fusion module to a relatively optimal one, where UTI-based SSR can be further optimized as the main task. Finally, the proposed method is validated on publicly available databases. Moreover, we elaborate the benefits of decoding UTI representations using additional modalities (even when such additional information is absent at test time) in both speaker-dependent and speaker-independent scenarios.

[0007] An ultrasonic tongue silent speech recognition method based on modal transfer learning, characterized in that it comprises the following steps: first, a UTI-based SSR baseline is implemented by introducing a four-step procedure to avoid overfitting due to the lack of large-scale data; to handle the many-to-many association between text and UTI signals, the model is further fine-tuned based on a joint training strategy from three modalities: UTI modality, additional modality, and their combination; meanwhile, considering the uncoordinated convergence phenomenon in multi-modal training, modal preference learning (MPL) is introduced to converge the fusion module to a relatively optimal one, where UTI-based SSR is further optimized as the main task; the four-step procedure is as follows: first, an initial model is trained using a Bi-GRU backend; second, a Bi-LSTM is attached while the front-end weights are frozen; third, the entire module is trained end-to-end until convergence; fourth, the trained front-end is used to generate UTI features for all data, which are used as input to the Transformer classifier.

[0008] Further, the method of the present application further comprises: using additional modules to compensate for necessary information and provide prior knowledge to constrain the target task, the first candidate additional module is ASR, wherein the audio signal contains rich information, and the second candidate additional module is lip reading, each additional module is composed of a feature extraction front-end and a feature mapping back-end, for sentence-level lip reading training of video streams, a general end-to-end training scheme is followed, wherein 1) the visual front-end is preprocessed on the LRW dataset, 2) the entire network is pre-trained on the LRS2 dataset using a curriculum learning setting, and 3) fine-tuning and inference are performed on the target dataset.

[0009] Further, the method further comprises: further optimizing the UTI-based SSR, implementing a fusion module based on a joint training strategy (JTS) to integrate comprehensive knowledge from additional modules, since early fusion and late fusion can suppress intra-modal or inter-modal interactions, the fusion module focuses on an intermediate approach, fusion occurs at the decoder layer of the converter-based model, and is trained using CTC loss, in addition, the JTS-based fusion module f(·) represents x u and auxiliary flow representation x a as input, while providing three types of knowledge: f(x u ; θ) is generated by the UTI flow, f(x a ; θ) is generated by the visual flow or the audio flow, and f(x u , x a ; θ) is generated by their combination.

[0010] Further, the method further comprises: in the jointly trained multi-modal model, a preference factor λ is put into the joint loss to emphasize the optimization of the target task, in addition, different weight settings are checked in each fusion experiment, and the setting with the best SSR performance is maintained, and the cost function of the fusion module is defined as follows:

[0011]

[0012] Compared with the prior art, the method has the beneficial effects that:

[0013] The method proposes a multi-stage, multi-modal modeling pipeline for the two main challenges of SSR, i.e., limited data set and easily confused silent signal. Four-step processes are introduced to avoid overfitting and two additional modules are introduced to extract multi-modal information. Based on the joint training strategy, modal preference learning (MPL) is proposed to promote the optimization of the target task by utilizing cross-modal prior knowledge. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of an ultrasound tongue silent speech recognition method based on modal transfer learning.

[0015] Figure 2 A model effect comparison diagram of the method. DETAILED DESCRIPTION

[0016] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0017] Figure 1 The overall framework of the proposed method is shown. It consists of a UTI module (yellow), an additional module (green), and a fusion module (blue). The additional module is optional at test time.

[0018] Limited by the first challenge, direct end-to-end training on sentence-level SSRs can tend to overfit, so we adopt a four-step procedure to gradually scale up the network. First, we train an initial model using a Bi-GRU backend; second, we remove the Bi-GRU and attach a Bi-LSTM while freezing the front-end weights; third, the entire module is trained end-to-end until convergence; fourth, the trained front-end is used to generate UTI features for all data and used as input to the Transformer classifier. Considering the effectiveness of the training process and learning ambiguity at the beginning of training, we incorporate a curriculum learning strategy. UTI flows start with short utterances and gradually increase sequence length to facilitate better convergence. Due to the limited training data, directly training a Transformer-based model with a large number of parameters can still be suboptimal. We adopt the following steps for fine-tuning.

[0019] While challenge two describes that information from a single non-acoustic modality is insufficient to fully characterize speech, thus additional modules are needed to compensate for the necessary information and provide prior knowledge to constrain the target task. The first candidate additional module is ASR, where audio signals contain rich information. In addition, the second is lip-reading, as visual lip movement information can be easily collected. Each additional module consists of a feature extraction front-end and a feature mapping back-end. For sentence-level lip-reading training of video streams, we follow the general end-to-end training scheme, where 1) the visual front-end is pre-trained on the LRW dataset, 2) the entire network is pre-trained on the LRS2 dataset using a curriculum learning setup, and 3) fine-tuning and inference are performed on the target dataset. The pipeline of the audio module is almost the same as that of the video module, except that the front-end is a one-dimensional convolutional filter without pre-processing.

[0020] To further optimize UTI-based SSR, a fusion module based on Joint Training Strategy (JTS) is implemented to integrate comprehensive knowledge from additional modules. Since early fusion and late fusion can suppress intra- or inter-modality interactions, our fusion module focuses on an intermediate approach, where fusion occurs at the decoder layer of the transducer-based model and is trained using CTC loss. Moreover, the JTS-based fusion module f(·) takes as input the UTI representation x u and the auxiliary stream representation x a , while providing three types of knowledge: f(x u ; θ) is generated from the UTI stream, f(x a ; θ) is generated from the visual or audio stream, and f(x u , x a ; θ) is generated from their combination. The multi-input design based on joint loss is essential for the fusion module to learn complementary information across modalities from scratch.

[0021] Furthermore, in the jointly trained multi-modal model, it can be outperformed by single-modal models in some cases. In particular, due to the imbalance of information carried by the two modalities, the network can constantly converge to local minima where only one modality is effectively learned. To overcome this problem, we propose a Modality Preference Learning (MPL) strategy to ensure that the UTI stream can benefit from joint training. The first reason is that the easily learned modality (e.g., audio) helps to achieve a lower joint discriminative loss, which then dominates the optimization progress by propagating limited gradients on other modalities. Therefore, we put a preference factor λ into the joint loss to emphasize the optimization of the target task. Moreover, we examine different weight settings in each fusion experiment and keep the setting with the best SSR performance. The cost function of the fusion module is defined as follows:

[0022]

[0023] The second reason is that different modalities overfit and generalize at different rates, so jointly training them with a single optimization strategy is suboptimal. Therefore, we employ several standard regularization methods to specifically enhance the generalization capability of the main task. Specifically, we stop the fusion training early and adjust the dropout only according to the performance of the main task. Moreover, we use the early-stopped UTI module to initialize the UTI encoder in the fusion module. To speed up the convergence of the fusion module, we freeze the UTI encoder weights and then train the entire fusion module after convergence. In summary, the most notable difference between JTS and MPL is that the former focuses on the optimization of the fused stream regardless of the single-modal streams, while the latter aims to facilitate the optimization of the target modality by using multi-modal streams.

[0024] Implementation details are as follows:

[0025] The UTI input stream is a 64x138 pixel, frame rate reduced to 30fps, adjusted from the raw ultrasound image. Through manual calibration, the additional input stream is synchronized with the UTI stream. The lip input is 112x112 pixels, with the region of interest (ROI) covering the area around the lips, with a sampling rate of 30fps. The audio input is a spectrogram obtained by applying a short-time Fourier transform (STFT). UTI and audio have different frame rates, with approximately one UTI frame corresponding to four audio frames. We convolve the speech signal from four frames into one group to align all modalities.

[0026] The UTI front-end applies a 3D convolution with kernel size 3x5x5, followed by a 2D ResNet, gradually reducing the spatial dimension with depth 2

[14] . The output is a 512-dimensional feature vector for each input frame. For the deployed back-end, we have three variants: a 2-layer Bi-GRU, a 2-layer Bi-LSTM, and a Transformer classifier with 4 encoder layers and 6 decoder layers, model size 2048, and 8 attention heads. In particular, the setup of the fusion decoder is consistent with the UTI decoder. Each module is trained using the CTC loss, emitting character probabilities for each input frame.

[0027] For training and inference, we use the Tongue and Lips (TaL) corpus. The TaL corpus is a sentence-level silent speech database containing approximately 24 hours of synchronized and parallel ultrasound, video, and audio. Word error rate (WER) and character error rate (CER) are reported in both cases. For the speaker-independent scenario, we train the model using data from speaker 19, -81 in the TaL80 and TaL1 datasets, and test and validate it using data from speakers 1-18, while the multi-speaker-dependent system contains data from all 82 speakers. The training set contains approximately 11,700 utterances, with an average of 136 utterances per speaker, and the test set contains approximately 13,00 utterances, approximately one-tenth of the corresponding training set. For transfer learning in additional modules and extra language models, we also use open-source large datasets such as the LRW and LSR2 datasets.

[0028] We investigated the effectiveness of the proposed modeling pipeline, including the 4-step process, multi-modal joint training, and modal preference learning. We used the UTI module as a baseline, as shown in Table 1. For the multi-speaker dependent scenario (SD), the Transformer with curriculum learning (CL) achieved the best WER of 45.9%, which improved by 10% and 17% compared to the WERs achieved by Bi-LSTM and Bi-GRU, respectively. For the speaker independent scenario (SI), the UO model achieved the best WER of 70.5%, which improved by 5% and 10% compared to the WERs achieved by the other two backends. Similarly, CER showed the same trend. Compared to the baseline WER of 83.81% in the SI scenario in

[19] , the proposed method achieved a significant improvement.

[0029] Table 1: Different SSR system performance comparison based on UTI input

[0030]

[0031] Tables 2 and 3 show the performance of the proposed models. Here, UO and UV / AU represent the UTI module and the fusion module, respectively. MPL and JTS represent the training strategies used in the fusion module. At first glance, both the additional visual and audio information improved compared to the UO baseline. Table 2 shows that the UTI stream in the UV-MPL model reduced by 8.2% and 3.2% in the SD scenario, respectively. Moreover, the UTI stream achieved 13.3% and 6.6% improvement in the SI scenario, respectively. Similarly, Table 3 shows that the UTI stream in the AU-MPL model significantly outperformed the baseline in both cases. These results indicate that the proposed method improved the performance of the UO baseline by incorporating multi-modal prior knowledge. Notably, the gain against the UO baseline was comparable to the two fusion modules. This phenomenon indicates that the gain mainly came from fine-tuning the UTI module during joint training. Furthermore, our method's benefits were more pronounced in the SI scenario. This phenomenon can mean that the additional modalities also played a role in data augmentation to improve the system's robustness. Moreover, all experiments benefited from an external language model (+LM).

[0032] Table 2: Using lip motion input as an auxiliary stream

[0033]

[0034] Table 3: Using audio input as an auxiliary stream

[0035]

[0036] Next, we evaluate the effectiveness of the MPL approach by comparing it with the conventional joint training strategy (JTS). Table 2-3 shows that the UTI flows in the JTS-based models are lower than the corresponding UO baselines. This indicates that there is an unbalanced convergence problem in the fusion modules based on JTS. As expected, the MPL-based models outperform the JTS-based models in both SSR scenarios, which indicates that the MPL approach alleviates the aforementioned problem and improves the performance of the UTI flows. For each step in MPL, we record the best results in the figure below to confirm their effectiveness. Based on the JTS-based models with the standard regularization strategy, we adjust the preference factor λ through extensive experiments and then introduce one adaptive regularization method at a time. The results show that most of them are effective, among which early stopping and dropout provide significant improvements. At the same time, the MPL methods that include them all achieve the best performance, even if λ can not be optimal.

[0037] We report the performance of pure audio, pure vision, and multi-model input in Tables 4-5. To make comparisons, we list the baseline results of the TaL training set and the LRS2 database, which improve by more than 4% compared to the 33.5% reported previously. The results in Table 4 show that even in the case of clean audio signals, the lip movements provide important cues in speech recognition. In addition, we observe that implementing the same pre-training strategy on two different datasets achieves the expected results, verifying the effectiveness of curriculum learning in cross-dataset training. We also observe that the performance of noisy audio on both datasets is significantly lower compared to the clean audio performance, but the latter's performance is higher. One possible reason is that the TaL training set is small in size and contains repeated training sentences.

[0038] Table 4: Performance of lip-reading, ASR, and audio-visual input systems

[0039]

[0040] Table 5: Performance of multi-modal input systems

[0041]

[0042] We present the multi-modal recognition results on the TaL database in Table 5, where the AU and AV results demonstrate that the tongue's movements provide more cues than the lips when the audio signal is noisy, even when the audio signal is clean. UV in the table indicates many examples where the model could not predict the correct sentence from the lips or tongue alone, but successfully deciphered the words when both streams were present. We achieved the best performance in AUV, regardless of whether the audio input was noisy.

[0043] In summary, we propose a multi-stage, multi-modal modeling pipeline to address the two main challenges of SSR, i.e., limited dataset and confusible silent signals. We introduce a four-step procedure to avoid overfitting and two additional modules to extract multi-modal information. Based on a joint training strategy, we propose modal preference learning (MPL) to facilitate the optimization of the target task by exploiting cross-modal prior knowledge.

[0044] The framework framework directly extracts features from lip pixels, tongue ultrasound data, and audio waveforms using joint learning and uses a Transformer for fusion recognition. We complete lip-reading pre-training on a large-scale dataset and obtain comparable baseline results after fine-tuning on the TaL corpus. After combining ultrasound features obtained from an independently trained encoder, we further improve the baseline, verifying the effectiveness of our proposed method. Specifically, tongue ultrasound data contribute more to the speech recognition system in noisy environments than lip data, indicating that tongue movements contain more practical information.

[0045] Moreover, even when only silent inputs are provided during inference, the proposed method achieves consistent improvements over the unimodal baseline in both cases

[0046] It is apparent to those skilled in the art that the present application is not limited to the details of the foregoing exemplary embodiments, and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in a descriptive sense only and not for purposes of limitation. The scope of the present application should be defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. No reference signs in the claims shall be considered to limit the scope of the claims in any way.

[0047] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A method for ultrasonic tongue-silent speech recognition based on modality transfer learning, characterized in that, The process includes the following steps: First, a UTI-based SSR baseline is established by introducing a four-step process to avoid overfitting due to a lack of large-scale data; to handle many-to-many associations between text and UTI signals, the model is fine-tuned again based on a joint training strategy from three modalities: the UTI modality, the additional modality, and their combination; simultaneously, considering the inconsistent convergence phenomenon in multimodal training, modality preference learning is introduced to converge the fusion module to a relative optimum, with further optimization of the UTI-based SSR as the main task; the four-step process is as follows: First, an initial model is trained using a Bi-GRU backend; second, while freezing the frontend weights, the Bi-GRU is removed and a Bi-LSTM is added; Third, the entire module is trained end-to-end until convergence; fourth, the trained front end is used to generate UTI features for all data and used as input to the Transformer classifier. The UTI input stream is a 64×138 pixel scaled-down version of the original ultrasound image, with the frame rate reduced to 30fps. An additional input stream is synchronized with the UTI stream via manual calibration. The lip input is 112×112 pixels, with the region of interest covering the area around the lips, and a sampling rate of 30fps. The audio input is a spectrogram obtained by applying a short-time Fourier transform. The UTI and audio have different frame rates, with one UTI frame corresponding to four audio frames. The speech signals from the four frames are convolved into a group to align all modalities. The UTI front-end uses a 3D convolution kernel with a kernel size of 3×5×5, followed by 2DResNet. The spatial dimension gradually decreases with depth. The output is a 512-dimensional feature vector for each input frame. Regarding the back-end deployment, there are three variants: a 2-layer Bi-GRU, a 2-layer Bi-LSTM, and a Transformer classifier with 4 encoder layers and 6 decoder layers, a model size of 2048, and 8 attention heads. The fusion decoder settings are consistent with the UTI decoder. Each module is trained using CTC loss to emit character probabilities for each input frame.

2. The ultrasonic tongue-silent speech recognition method based on modality transfer learning according to claim 1, characterized in that, Also includes: Additional modules are employed to compensate for necessary information and provide prior knowledge to constrain the target task. The first candidate additional module is ASR, where the audio signal contains rich information, and the second candidate additional module is lip reading. Each additional module consists of a feature extraction front-end and a feature mapping back-end. For sentence-level lip reading training of video streams, a general end-to-end training scheme is followed, in which 1) the visual front-end is preprocessed on the LRW dataset, 2) the entire network is pre-trained on the LRS2 dataset using a course learning setting, and 3) fine-tuning and inference are performed on the target dataset.

3. The ultrasonic tongue silence speech recognition method based on modality transfer learning according to claim 2, characterized in that, Also includes: Further optimization of UTI-based SSR is achieved by implementing a fusion module based on a joint training strategy to integrate comprehensive knowledge from additional modules. Since early and late fusion can suppress intra- or inter-modal interactions, the fusion module focuses on an intermediate approach. Fusion occurs at the decoder layer of the transformer-based model and is trained using CTC loss. Furthermore, a JTS-based fusion module f(·) integrates the UTI representation x... u And auxiliary flow representation x a As input, three types of knowledge are provided simultaneously: f(x) u ;θ) is generated by the UTI stream, f(x) a ;θ) is generated from the visual stream or the audio stream, f(x) u ,x a ;θ) is generated by their combination.

4. The ultrasonic tongue silence speech recognition method based on modality transfer learning according to claim 3, characterized in that, Also includes: In the jointly trained multimodal model, the preference factor λ is incorporated into the joint loss to emphasize the optimization of the target task. Furthermore, different weight settings are examined in each fusion experiment, maintaining the settings that yield the best SSR performance. The cost function of the fusion module is defined as follows: 。

Images