Multi-mode dysarthria speech reconstruction system and method

Abstract

The invention provides a multi-mode dysarthria speech reconstruction system and a multi-mode dysarthria speech reconstruction method. The system comprises an encoder and a decoder. The encoder is configured to: extract a multi-modal feature from a multi-modal input of dysarthria speech and generate phoneme embedding based on the multi-modal feature; predicting a sequence of phoneme durations based on the phoneme embedding and generating an extended phoneme embedding and a predicted tone sequence; and extracting a speaker identity characterization of the dysarthria speech speaker. The decoder is configured to reconstruct a normal speech waveform corresponding to dysarthria speech with the extended phoneme embedding, the predicted tone sequence, and the speaker identity representation as inputs.

Description

Technical Field

[0001] The present application relates to the technical field of speech processing, and more particularly, to a multimodal dysarthric speech reconstruction system and method thereof. Background Art

[0002] Dysarthria is a common form of speech disorder associated with neurological movement disorders such as Parkinson's disease and cerebral palsy, as well as brain injuries caused by stroke or head trauma. Dysarthria can lead to a significant reduction in speech quality, a high degree of variation in speech characteristics, and a large deviation from normal speech, thus significantly affecting the communication between dysarthric speakers and their family members or caregivers.

[0003] In response to the above problems, dysarthric speech reconstruction (DSR) technology has been proposed, which aims to convert dysarthric speech into normal speech by improving the intelligibility and naturalness of dysarthric speech. Therefore, DSR is one of the effective solutions to assist dysarthric individuals in communication.

[0004] However, DSR is a challenging task, especially for patients with severe dysarthria and dysarthric speakers speaking in complex acoustic environments with noise. Currently, many related research works are underway to improve the speech quality, intelligibility, and naturalness of the reconstructed speech. For example, the method based on a speech bank uses speech recordings collected before the deterioration of the pronunciation ability of dysarthric speakers to build a personalized text-to-speech (TTS) system. However, this method is limited by the speech recording situation before the occurrence of dysarthria in dysarthric speakers and is not applicable to all patients. There are also some studies that attempt to solve the reconstruction problem through voice conversion (VC) technology. VC technology keeps the content of the dysarthric speech signal unchanged while adjusting it to a clearer and more natural speech signal. Rule-based VC modifies the temporal or frequency characteristics of speech according to specially designed rules. Statistical VC methods are based on Gaussian mixture models (GMMs), non-negative matrix factorization (NMF), and partial least squares (PLS), etc., to create a mapping function between the auditory features of dysarthric speech and those of normal speech. Subsequently, a DSR system based on end-to-end VC (E2E-VC) was also proposed, which extracts a speech encoder from a pre-trained speech recognition model to replace the text encoder of a sequence-to-sequence (seq2seq) TTS system for cross-modal knowledge distillation (KD). Compared with a simple concatenation system that provides the automatic speech recognition (ASR) result to the TTS model, E2E-VC does not limit the intermediate representation to text characters and can generate speech with lower errors and higher fidelity. In addition, an additional prosody corrector and a deep speaker encoder are added to improve prosody and speaker similarity. Summary of the Invention

[0005] [Technical Problem]

[0006] Previous research work has mainly focused on speakers with less severe dysarthria. In addition, complex noisy environments are also crucial for the extraction of semantic representations. Therefore, the intelligibility and naturalness of existing reconstructed speech are still not satisfactory, especially for speakers with more severe dysarthria. Therefore, a dysarthria speech reconstruction (DSR) system or method is needed to improve the speech reconstruction performance of the DSR system in noisy environments and for disordered speech.

[0007] [Technical Solution]

[0008] To at least partially solve the above problems existing in the related art, the present application provides a system, a method, an electronic device, and a computer-readable medium for multi-modal dysarthria speech reconstruction.

[0009] On the one hand, the present application provides a multi-modal dysarthria speech reconstruction system, the system includes: an encoder, configured to: extract multi-modal features from the multi-modal input of dysarthria speech, and generate a phoneme embedding based on the multi-modal features; based on the phoneme embedding, generate a predicted phoneme duration sequence, an extended phoneme embedding, and a predicted pitch sequence; and extract the speaker identity representation of the dysarthria speech; and a decoder, configured to: use the extended phoneme embedding, the predicted pitch sequence, and the speaker identity representation as inputs to reconstruct a normal speech waveform corresponding to the dysarthria speech.

[0010] On the other hand, the present application provides a dysarthria speech reconstruction method, the method includes: extracting multi-modal features from the multi-modal input of dysarthria speech, and generating a phoneme embedding based on the multi-modal features; based on the phoneme embedding, generating a predicted phoneme duration sequence, an extended phoneme embedding, and a predicted pitch sequence; extracting the speaker identity representation of the dysarthria speech; and based on the extended phoneme embedding, the predicted pitch sequence, and the speaker identity representation, reconstructing a normal speech waveform corresponding to the dysarthria speech.

[0011] On the other hand, the present application provides an electronic device for multi-modal dysarthria speech reconstruction, the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the multi-modal dysarthria speech reconstruction method.

[0012] On the other hand, the present application provides a computer-readable storage medium storing computer instructions, wherein when the computer instructions are executed, the computer is caused to execute the multi-modal dysarthric speech reconstruction method.

[0013] [Technical effects]

[0014] The multi-modal dysarthric speech reconstruction system and method provided by the present application have been significantly improved in terms of speech intelligibility and naturalness, especially for speakers with more severe dysarthria. Description of the drawings

[0015] Other features, objects, and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings. Among them:

[0016] Figure 1 is a schematic structural diagram of a multi-modal dysarthric speech reconstruction system according to an embodiment of the present application.

[0017] Figure 2 is a schematic structural diagram of an existing single-modal content encoder.

[0018] Figure 3A is a schematic structural diagram of a multi-modal encoder according to another embodiment of the present application.

[0019] Figure 3B is a schematic structural diagram of a multi-modal encoder according to an embodiment of the present application.

[0020] Figure 4A is a schematic structural diagram of a phoneme duration predictor in a prosody encoder according to an embodiment of the present application.

[0021] Figure 4B is a schematic structural diagram of a pitch predictor in a prosody encoder according to an embodiment of the present application.

[0022] Figure 5 is a schematic structural diagram of a speaker encoder according to an embodiment of the present application and a schematic diagram of the training process of the speaker encoder.

[0023] Figure 6 is a flowchart of a dysarthric speech reconstruction method according to an embodiment of the present application.

[0024] Figure 7 shows a schematic block diagram of an electronic device capable of implementing the dysarthric speech reconstruction method according to various embodiments of the present application.

[0025] Figure 8 shows exemplary video snapshots of four selected dysarthric speakers while speaking.

[0026] Figure 9 Figure showing the comparison of MOS test results between the original dysarthric speech and the reconstructed speech. Detailed implementation manners

[0027] To better understand the present application, various aspects of the present application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application and do not limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0028] It should be noted that in this specification, the expressions such as first, second, third, etc. are only used to separate one feature from another feature region and do not represent any limitation on the features, especially do not represent any order of precedence.

[0029] In the accompanying drawings, for the sake of illustration, the thickness, dimensions and shapes of the components have been slightly adjusted. The drawings are only examples and are not drawn to an exact scale. As used herein, the terms "substantially", "about" and similar terms are used as terms indicating approximation and not as terms indicating degree, and are intended to account for the inherent deviations in measured or calculated values that would be recognized by a person of ordinary skill in the art.

[0030] It should also be understood that expressions such as "including", "comprising", "having", "containing" and / or "comprising of" are open-ended rather than closed-ended expressions in this specification, which means the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when an expression such as "at least one of..." appears after a list of listed features, it modifies the entire list of features rather than just individual elements in the list. In addition, when describing the embodiments of the present application, the use of "may" means "one or more embodiments of the present application". And the term "exemplary" is intended to refer to an example or illustration.

[0031] Unless otherwise defined, all terms used herein (including engineering terms and scientific and technical terms) have the same meaning as commonly understood by a person of ordinary skill in the art to which this application belongs. It should also be understood that unless there is a clear statement in the present application, words defined in a common dictionary should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal sense.

[0032] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. In addition, unless explicitly defined or inconsistent with the context, the specific steps included in the methods described in the present application do not have to be limited to the recorded order, but may be executed in any order or executed in parallel. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0033] In addition, in the present application, when using "connect" or "couple", it may indicate direct contact or indirect contact between the corresponding components, unless there are clear other limitations or can be deduced from the context.

[0034] Damage or degradation of one or more parts of the speech generation control system can lead to dysarthria, which exhibits acoustic characteristics significantly different from those of the speech uttered by healthy people. Dysarthria may be caused by different complex factors and present different pronunciation patterns, which brings difficulties to accurately and completely describe the causes and characteristics of dysarthria.

[0035] Generally, rule-based and statistical voice conversion (VC) methods can be applied to DSR. Specifically, rule-based VC tends to use manually designed and speaker-dependent rules to correct the content errors of the speaker, or to improve the intelligibility of the speaker's speech by modifying the temporal and frequency characteristics of the speaker's speech. Statistical VC automatically maps the features of, for example, dysarthric speech to the features of normal speech, and typical methods include Gaussian Mixture Model (GMM), Nonnegative Matrix Factorization (NMF), Partial Least Squares (PLS), and deep learning methods including sequence-to-sequence (seq2seq) models and gated convolutional networks.

[0036] Figure 1 It is a schematic structural diagram of a dysarthric speech reconstruction system 1000 according to an embodiment of the present application.

[0037] As Figure 1 shown, the DSR system 1000 can improve the intelligibility of the speech of dysarthric patients, making it understandable to the general population. Specifically, for any given sentence X spoken by a dysarthric patient, it can be decomposed into three parts: display content C, abnormal prosody P, and speaker identity S.

[0038] The displayed content C may include errors such as repetition, deletion, and replacement of content, which are likely to mislead the listener about the true meaning of the patient with dysarthria; the abnormal prosody P may include, for example, inaccurate phoneme durations and pitches, which may prevent the patient with dysarthria from expressing their emotional state well and also affect the intelligibility of their speech; and the speaker identity S can be understood as the timbre of the patient with dysarthria, and the speaker identity S of different patients with dysarthria is different. Through the DSR system 1000 provided by the present application, speech can be generated that clearly and accurately conveys the intention of the patient with dysarthria in a natural manner.

[0039] The DSR system 1000 is a scalable SED framework structure. Optionally, as Figure 1 shown, the DSR system 1000 uses a set of encoders (not shown), which may include, for example, a multimodal encoder 1100, a prosody encoder 1200, and a speaker encoder 1300. These encoders can model different components of the speech signal, where the different components of the speech signal may include, for example, content, prosody, and speaker identity. As an option, these different components may respectively correspond to the core sub-problems of DSR, such as how to restore the content of the speech with speech disorders, how to correct the prosody of the speech with speech disorders, and how to obtain the speaker identity of the patient with dysarthria. In addition, the DSR system 1000 further includes a decoder 1400, and the decoder 1400 can combine the restored content representation, the corrected prosody representation, and the saved speaker identity representation to generate normally pronounced speech.

[0040] As Figure 1 shown, a dysarthric speech reconstruction (DSR) system 1000 includes: an encoder (not shown) and a decoder 1400. The encoder is configured to perform at least one of the following: extract the content of the dysarthric speech and generate a phoneme embedding; predict a phoneme duration sequence based on the phoneme embedding and generate an extended phoneme embedding and a predicted fundamental frequency sequence; and extract a speaker identity representation of the dysarthric speech. The decoder 1400 takes the extended phoneme embedding, the predicted fundamental frequency sequence, and the speaker identity representation as inputs and reconstructs a speech waveform corresponding to the speech with dysarthria.

[0041] Optionally, the DSR system 1000 may include an encoder and a decoder 1400, where the encoder may include at least one of a content encoder 1100, a prosody encoder 1200, and a speaker encoder 1300.

[0042] The content encoder 1100 can be configured to extract the content of dysarthric speech and generate phoneme embeddings. The prosody encoder 1200 can be configured to predict a phoneme duration sequence based on the phoneme embeddings, and generate extended phoneme embeddings and a predicted fundamental frequency sequence. The speaker encoder 1300 can be configured to extract the speaker identity representation of dysarthric speech.

[0043] In other words, the DSR system 1000 proposed in this application is a scalable encoder-decoder (SED) framework structure. It uses one or more encoders to meet the requirements of DSR, so it has scalability.

[0044] Specifically, the SED framework structure proposed in this application can gradually add encoders to improve the DSR system 1000. For example, a DSR system 1000 can be constructed using only one content encoder 1100 and one decoder 1400 to generate a normal speech waveform (e.g., Mel spectrogram), where the prosody of dysarthric speech can be implicitly modeled and the preservation of the speaker identity of dysarthric patients can be ignored.

[0045] Furthermore, on this basis, by adding a prosody encoder 1200 for prosody correction, the prosody encoder 1200 can predict normal values such as phoneme duration and fundamental frequency (F0) through explicit modeling to replace abnormal items corresponding to the above normal values. In addition, a speaker encoder 1300 can be added to extract the speaker identity representation for modeling speaker characteristics, so as to obtain a more complete DSR system 1000, which can handle the three core sub-problems of DSR, namely how to restore the content of dysarthric speech, how to correct the prosody of dysarthric speech, and how to preserve the identity of dysarthric patients.

[0046] In addition, the DSR system 1000 provided in this application provides flexible control over different components of dysarthric speech. Specifically, the above different encoders are used to separately model different components of dysarthric speech (e.g., content, prosody, and speaker identity). In addition, these components can be flexibly modified as follows: (1) To modify the content, parallel normal content representations (if available) can be used to replace their original corresponding content to generate speech with more accurate content reconstruction; (2) To modify the prosody, the phoneme duration or F0 can be automatically or manually changed so that the generated speech has the desired prosody pattern; (3) To modify the speaker characteristics, the speaker identity representation can be switched so that the reconstructed speech has a different timbre.

[0047] Refer back to Figure 1, in one embodiment of the present application, the proposed DSR system 1000 may include four modules: a multi-modal encoder 1100, a prosody encoder 1200, a speaker encoder 1300, and a decoder 1400.

[0048] The multi-modal encoder 1100 may be configured to extract multi-modal features regarding dysarthric speech from various modal inputs such as audio input and video input of a dysarthric speaker, and generate more robust phoneme embeddings based on the multi-modal features. The prosody encoder 1200 may be configured to predict a phoneme duration sequence based on the phoneme embeddings, and generate extended phoneme embeddings and a predicted pitch sequence based thereon. The speaker encoder 1300 may be configured to extract a speaker identity representation from the multi-modal inputs of the dysarthric speech. The decoder 1400 will reconstruct a normal speech waveform corresponding to the dysarthric speech based on the extended phoneme embeddings, the predicted pitch sequence, and the speaker identity representation as inputs.

[0049] Next, in conjunction with Figures 2 to 5 each component of the multi-modal dysarthric speech reconstruction system 1000 proposed in the present application will be described in detail. Figure 2 is a schematic structural diagram of an existing single-audio content encoder. Figure 3A is a schematic structural diagram of a multi-modal encoder 1100-1 according to one embodiment of the present application. Figure 3B is a schematic structural diagram of a multi-modal encoder 1100-2 according to another embodiment of the present application. Figure 4A is a schematic structural diagram of a phoneme duration predictor 1201 in a prosody encoder 1200 according to one embodiment of the present application. Figure 4B is a schematic structural diagram of a pitch predictor 1202 in a prosody encoder 1200 according to one embodiment of the present application. Figure 5 is a schematic structural diagram of a speaker encoder 1300 according to one embodiment of the present application and a schematic diagram of the training process of the speaker encoder 1300.

[0050] To more clearly illustrate the difference between the multi-modal encoder adopted in the present application and the existing single-audio encoder, first refer to Figure 2 describe the structure of the existing single-audio encoder.

[0051] As Figure 2 shown, as an example, the single-audio encoder 200 may include an auditory feature extractor 210 and an auto-regressive sequence-to-sequence (AR seq2seq) ASR model Φ220, where the auditory feature extractor 210 is used to extract the auditory feature x of the dysarthric speech a , and the ASR model Φ220 is used to generate text based on the auditory feature x aOutput phoneme embedding p, which can be expressed as p = Φ(x a ).

[0052] In addition, dysarthric audio usually has strong background noise, which will reduce the performance of the speech encoder. Optionally, before performing auditory feature extraction, the log-MMSE speech enhancement method can be used to preprocess the dysarthric speech first, and then extract 80-dimensional filter bank (FBK)+Δ features as the auditory feature x a .

[0053] Optionally, the ASR model Φ may include a VGG-based encoder 221 (a convolutional network proposed by the Visual Geometry Group) and an AR decoder 222.

[0054] For example, the VGG-based encoder 221 may include a VGG extractor with 6 layers (a convolutional network proposed by the Visual Geometry Group), and a 5-layer bidirectional long short-term memory network (BLSTM) with 512 units in each direction. For another example, the AR decoder 222 includes a 512-dimensional position-aware attention module and 2 long short-term memory networks (LSTM) with 1024 units in each layer.

[0055] For another example, the AR decoder 222 may also have connectionist temporal classification (CTC), and can optimize the prediction of the content sequence from the input speech by minimizing only the sum of the cross entropy (CE) between the true content sequence and the predicted content sequence and the loss of the neural network-based connectionist temporal classification (CTC), where the predicted content sequence (hereinafter referred to as phoneme embedding) may include, for example, 73 contents, 1 start token, and 1 end token.

[0056] In fact, human perception of speech is multimodal, such as involving audition and vision, etc. Moreover, the performance of ASR systems in noisy environments and disordered speech needs to be improved, so relying solely on information from the audio modality is not enough. Previous research work has shown that combining the visual modality can improve the performance of speech reconstruction. Therefore, this application proposes a new multimodal speech reconstruction framework that uses at least one other modality of information in addition to auditory information to reconstruct highly abnormal speech, such as visual information (e.g., the movement of the speaker's mouth when speaking).

[0057] In one embodiment of the present application, to generate more robust phoneme embeddings, multimodal features can be extracted from the multimodal input of dysarthric speech, and phoneme embeddings can be generated based on the multimodal features.

[0058] As an example, a VGG-based audiovisual encoder 1100-1 as shown in Figure 3A or an AV-Hubert-based audiovisual encoder 1100-2 as shown in Figure 3B can be adopted to generate phoneme embeddings based on audiovisual features, where visual features are further introduced.

[0059] Specifically, the audiovisual encoders 1100-1 and 1100-2 respectively extract visual feature x v and auditory feature x a from the visual input and auditory input of the dysarthric person; fuse the extracted visual feature x v and auditory feature x a to obtain fused audiovisual features; and generate the phoneme embeddings based on the fused audiovisual features.

[0060] As an option, as shown in Figure 3A the multimodal encoder can be a VGG-based audiovisual encoder 1100-1.

[0061] As shown in Figure 3A the audiovisual encoder 1100-1 may include: an auditory feature extractor 311 for extracting auditory feature x a from the audio input of the dysarthric person; a video feature extractor 312 for extracting visual feature x v from the video input of the dysarthric person; a fuser for fusing the extracted visual feature x v and auditory feature x a to obtain fused audiovisual features; and an autoregressive sequence-to-sequence ASR model Φ320 for generating phoneme embeddings based on the fused audiovisual features.

[0062] As an example, the audiovisual encoder 1100-1 can force-align each video input with the corresponding audio input by oversampling. Then, an existing face alignment network is used to detect lip shape landmarks on the upsampled video input, and an affine transformation is performed to keep the detected lip shape area horizontal. Finally, discrete cosine transform (DCT) and linear discriminant analysis (LDA) are used to reduce the visual features and extract the reduced visual feature x v . Since the extracted visual feature x v is temporally aligned with the auditory feature x a , the audio and video can be concatenated in the feature dimension and fused through a fully connected (FC) layer to obtain fused audiovisual features.

[0063] As an example, the audiovisual encoder 1100-1 may adopt an autoregressive sequence-to-sequence ASR model Φ320 including a VGG-based encoder 321 and an AR decoder 322, and generate a phoneme embedding P based on the fused audiovisual features. The phoneme embedding can be expressed as where represents concatenation in the feature dimension, and W1 and z1 are FC layer parameters. The structures and operations of the VGG-based encoder 321 and the AR decoder 322 are respectively similar to Figure 2 the VGG-based encoder 221 and the AR decoder 222 shown in

[0064] As another option, as shown in Figure 3B the multimodal encoder 1100-2 may be an AV-Hubert-based audiovisual encoder, in which a well-pre-trained AV-Hubert encoder 331 is adopted to replace Figure 3A the VGG-based encoder 321 shown in Figure 3B As shown in a the audiovisual encoder 1100-1 may include: an auditory feature extractor 311 for extracting an auditory feature x from the audio input of a person with dysarthria v ; a video feature extractor 312 for extracting a visual feature x from the video input of a person with dysarthria v ; a fuser for fusing the extracted visual feature x a and auditory feature x to obtain a fused audiovisual feature; and an autoregressive sequence-to-sequence ASR model for generating a phoneme embedding based on the fused audiovisual feature.

[0065] As an example, the audiovisual encoder 1100-2 may adopt an autoregressive sequence-to-sequence ASR model including an AV-Hubert encoder 331 and an AR decoder 332 to generate a phoneme embedding P based on the fused audiovisual features. The phoneme embedding can be expressed as where represents concatenation in the feature dimension, and W1 and z1 are FC layer parameters.

[0066] The AV-Hubert's audiovisual encoder obtains the mapping relationship between the fused audiovisual features and the phoneme embeddings by training a deep temporal neural network mapping model. Specifically, through masking multiple streaming video inputs and the predicted multimodal hidden units, it automatically discovers and iteratively refines, performing powerful self-supervised learning on the audiovisual speech representation, which is beneficial for both lip reading and ASR. Similar to the traditional AV-Hubert design, the audiovisual feature fusion is also carried out through concatenation in the feature dimension. The fused audiovisual features are fed into the FC layer to map the extracted features to the corresponding dimensions in AV-Hubert. Similarly, the phoneme embeddings can be represented as where represents the improved ASR model based on AV-Hubert, and W2 and z2 are the FC layer parameters.

[0067] In addition, to improve the content prediction accuracy, the multimodal encoder 1100 can be trained in two stages: (1) pre-training on a large amount of normal speech data to obtain an initialization model with strong generalization ability; (2) fine-tuning (hereinafter simply referred to as fine-tuning, or training) on the dysarthric speech of specific dysarthric patients s d to improve the content prediction performance. The output Φ p of the pre-trained multimodal encoder or the output of the pre-fine-tuned multimodal encoder p and can be used as the content representation (phoneme embeddings) representing the posterior content probability distribution. Hereinafter, Φ p and are respectively used to extract the content representation from normal speech and dysarthric speech.

[0068] Referring again to Figure 1 , in an embodiment of the present application, the prosody encoder 1200 may include a phoneme duration predictor 1201 and a pitch (F0) predictor 1202, where the predictor 1201 and the F0 predictor 1202 can be respectively used to infer the normal phoneme duration and the normal pitch. Optionally, the prosody encoder 1200 can be based on the phoneme embeddings extracted from the dysarthric speech to infer the normal values of the phoneme duration and the normal value of F0, and these normal values can be used as standard values to replace their corresponding abnormal values, thereby generating speech with a normal prosody pattern. Therefore, the present application does not need to provide the original prosody features to the prosody encoder 1200. In other words, the prosody encoder 1200 omits the steps of extracting abnormal phoneme durations and abnormal pitches from the dysarthric speech.

[0069] Optionally, as Figure 4AAs shown in the figure, the phoneme duration predictor 1201 may include: a prediction unit 1211, an output unit 1221, and an optimization unit 1231. The prediction unit 1211 may be, for example, a stacked bidirectional gated recurrent unit (Bidirectional Gated Recurrent Units, Bi-GRU). As an option, the output unit 1221 may include multiple groups of parallel modules, such as three groups of parallel modules. The parallel module may include: a Bi-GRU and an FC layer; an FC layer; and a filter bank. In addition, as an option, the optimization unit 1231 may use a rectified linear unit (Rectified Linear Units, ReLU) activation function to optimize the output value of the output unit 1221. Optionally, the stacked Bi-GRU may be, for example, a three-layer structure with 256 units in each direction; the Bi-GRU may be, for example, a one-layer structure with 256 dimensions; the filter bank may include three convolutional layers, and their kernel sizes may be 5, 9, and 19 respectively; and the FC layer may be, for example, a one-dimensional structure for predicting the duration value and the F0 value.

[0070] Specifically, in Figure 4A the phoneme duration predictor 1201 shown in the figure, the stacked Bi-GRU of the prediction unit 100 first introduces the phoneme embedding and makes full use of the context in the two directions before and after the phoneme embedding to obtain the output D of the stacked Bi-GRU s . After that, the output D s of the stacked Bi-GRU enters the output unit 1221 and is consumed in the following three ways: (1) through the Bi-GRU set in the output unit 1221 and the FC layer behind the Bi-GRU, the output D of the Bi-GRU in the phoneme duration predictor is obtained g ; (2) through the FC layer set in the output unit 1221, the weight ω of the output D g of the middle Bi-GRU is obtained; and (3) through the filter bank set in the output unit 1221, the output D of the filter bank in the phoneme duration predictor is obtained f . Then, using formula (1), the linear weighted sum of the Bi-GRU output D g and the filter bank output D f can be determined, and through the ReLU activation function of the optimization unit 1231, the content-level, predicted phoneme duration is obtained

[0071]

[0072] where, is the predicted duration sequence; D g is the output of the Bi-GRU in the phoneme duration predictor; D fis the output of the filter bank in the phoneme duration predictor; and ω is the weight of the output of the Bi-GRU in the phoneme duration predictor.

[0073] In addition, in an embodiment of the present application, in order to train the phoneme duration predictor 1201 that can give normal duration predictions, the present application can be trained using normal speech from healthy speakers, and tools such as the Montreal Forced Aligner (MFA) toolkit can be used to perform the alignment between the content and the speech segments, so as to obtain the true duration of each content. As shown in Equation (2), the training loss of the phoneme duration predictor 1201 is the predicted duration sequence and the L1 norm between the true duration sequence d:

[0074]

[0075] where is the predicted duration sequence; d is the true duration sequence, and is the training loss of the phoneme duration predictor.

[0076] After obtaining the predicted phoneme duration sequence, the prosody encoder 1200 can expand the phoneme embedding into an extended phoneme embedding based on the predicted phoneme duration sequence, and generate a predicted pitch sequence based on the extended phoneme embedding.

[0077] As an option, Figure 4B the F0 predictor 1202 shown in Figure 4A can have a similar architecture to the phoneme duration predictor 1201 shown in The difference is that the input of the F0 predictor 1202 is the frame-level, expanded phoneme embedding, and because the normalized logarithmic scale F0 (e.g., applying z-normalization) is used as the training target, the training process of the F0 predictor 1202 is stabilized, so the F0 predictor 1202 does not use an optimization module including an activation function such as ReLU to optimize the final output. Therefore, as shown in Equation (3), the predicted F0 sequence is:

[0078]

[0079] where is the predicted F0 sequence; H g is the output of the Bi-GRU in the F0 predictor; H f is the output of the filter bank in the F0 predictor; and ω is the weight of the output of the Bi-GRU in the F0 predictor.

[0080] In addition, as shown in formula (4), by training the F0 predictor 1202, the L1 norm between the true F0 sequence v and the predicted F0 sequence can be minimized as the training loss of the F0 predictor 1202 wherein is the predicted F0 sequence; v is the true F0 sequence, and

[0081]

[0082] is the training loss of the F0 predictor is the predicted F0 sequence; v is the true F0 sequence, and is the training loss of the F0 predictor

[0083] Refer to Figure 1 and Figure 5 According to an embodiment of the present application, the speaker encoder 1300 may be an extensible neural network that can receive mel spectrograms of any length and generate speaker embedding vectors with a fixed dimension. Specifically, in an embodiment of the present application, the speaker encoder 1300 may include an LSTM and a projection layer located after the LSTM. Optionally, the LSTM may be a three-layer structure with 256 hidden nodes, and the projection layer may have 256 units. In addition, the projection layer may be an FC layer. The output of the speaker encoder 1300 is the L2-normalized hidden state of the last layer in the LSTM, that is, a vector of 256 elements. The last layer in the LSTM can be understood as the structure closest to the projection layer in the LSTM

[0084] As Figure 5 shown, in order to train the speaker encoder 1300 to extract accurate speaker identity representations independent of speech content and background noise, the present application uses a Generalized End-to-End (GE2E) loss to represent the training loss of the speaker encoder 1300

[0085] In an embodiment of the present application, obtaining the training loss of the speaker encoder 1300 may include: obtaining all the embedding vectors of the speech dataset for training, where each embedding vector is the L2-normalized norm of the output of the speaker encoder; obtaining the centroid of all the utterances of each speaker in the speech dataset based on all the embedding vectors; obtaining a similarity matrix set for each embedding vector based on the centroid of all the utterances of each speaker; applying a classification function to the similarity matrix set, and determining the training loss of the speaker encoder based on the similarity matrix set after applying the classification function to obtain the speaker identity representation

[0086] Specifically, the speaker encoder 1300 may include, for example, a 3-layer LSTM, with an additional FC layer. The speech data set used for training the speaker encoder 1300 may be a batch of speech data containing I speakers, with each speaker having J utterances, where the I speakers may include spk1, spk2, spk3, etc., m ij may represent the mel spectrogram of the j-th utterance from the i-th speaker, where both i and I are positive integers, and i ∈ (1, 2... I); and both j and J are positive integers, and j ∈ (1, 2... J). The last hidden state of the last LSTM layer of the speaker encoder 1300 can be projected onto a single output vector f s (m ij ; θ s ), where θ s represents the parameters of the speaker encoder 1300. Then, the embedding vector e ij (i.e., the speaker identity representation) of the j-th utterance of the i-th speaker can be defined as the L2-normalized norm of the speaker encoder output:

[0087]

[0088] where e ij is the embedding vector of the j-th utterance of the i-th speaker; and f s (m ij ; θ s ) is the single vector in the speaker encoder output representing the j-th utterance of the i-th speaker.

[0089] On this basis, in order to obtain the above GE2E loss and obtain an accurate speaker identity representation independent of speech content and background noise, the centroid of the embedding vectors of the remaining utterances of the i-th speaker except the j-th utterance can be obtained through formula (6)

[0090]

[0091] where, is the centroid of the embedding vectors of the remaining utterances of the i-th speaker except the j-th utterance; n is a positive integer, n ∈ (1, 2... J) and n ≠ j; and e in is the embedding vector of the n-th utterance of the i-th speaker.

[0092] After obtaining the centroid of the embedding vectors of the remaining utterances of the i-th speaker except the j-th utterance , the centroid c k1 of {e k2 , e kJ} can be obtained through formula (7) k, the centroid can represent the voiceprint composed of J utterances of the k-th speaker.

[0093]

[0094] where c k is the centroid of J utterances of the k-th speaker; e kj is the embedding vector of the j-th utterance of the k-th speaker; and k is a positive integer, and k ∈ (1, 2... I).

[0095] Through formula (6) and formula (7), the centroid of all utterances of each speaker in the batch of I speakers with J utterances per speaker can be obtained, and then the similarity matrix M can be obtained through formula (8). The similarity matrix M can be defined as the cosine similarity ratio between each embedding vector and the centroid of different speakers:

[0096]

[0097] where c k is the centroid of J utterances of the k-th speaker; e ij is the embedding vector of the j-th utterance of the i-th speaker; and is the centroid of the embedding vectors of the remaining utterances of the i-th speaker except the j-th utterance.

[0098] On this basis, the similarity matrix set {M ij of the embedding vector e ij，1 of the j-th utterance of i speakers and the centroids of I different speakers, M ij，2 ,..., M ij，I} (hereinafter referred to as similarity scores) can be obtained. As shown in formula (9), classification functions such as the normalized exponential function (softmax) can be applied to the similarity scores {M ij of the embedding vector e ij，1 of the j-th utterance of the i-th speaker and the centroids of I different speakers, M ij，2 ,..., M ij，I}. After this operation, if k = i, the value of M ij，k will become very large; if k ≠ i, the value of M ij，k will become very small. Referring to Figure 5 , through this operation, the pattern area of the similarity matrix M can be forced to be as large as possible, and the gray area can be forced to be as small as possible. Therefore, the embedding vector of each utterance should be similar to the centroid of all embedding vectors of that speaker and far from the centroids of other speakers. Thus, the speaker embedding vectors extracted by the speaker encoder 1300 can represent distinguishable speaker identity features among different speakers.

[0099]

[0100] Among them, is the embedding vector e of the j-th utterance of the i-th speaker ij and the similarity scores {M ij，1 , M ij，2 ,..., M ij，I} applied to the centroids of I different speakers; e ij is the embedding vector of the j-th utterance of the i-th speaker; and M ij，i is the similarity score between the embedding vector e ij of the j-th utterance of the i-th speaker and the centroids of I different speakers.

[0101] Finally, the GE2E loss can be calculated by formula (10) to obtain an accurate speaker identity representation independent of speech content and background noise.

[0102]

[0103] Among them, is the loss of the speaker encoder; e ij is the embedding vector of the j-th utterance of the i-th speaker; and is the normalized exponential function applied to the similarity scores {M ij between the embedding vector e ij，1 of the j-th utterance of the i-th speaker and the centroids of I different speakers, M ij，2 ,..., M ij，I}.

[0104] Most current DSR technologies ignore the problem of preserving the speaker identity of dysarthric patients, that is, the timbre of the reconstructed speech is different from that of the original dysarthric speech, which makes it impossible for dysarthric patients to display their social roles or personalities through auditory features in their own timbre. In the DSR system proposed in at least one embodiment of the present application, a speaker encoder is used, and the speaker encoder is trained for the speaker verification (i.e., speaker identification) task (SpeakerVerification, SV) using a large amount of data of thousands of speakers, so that the speaker encoder can capture the speaking characteristics of the target speaker. Therefore, the speaker encoder can be used to receive dysarthric speech and extract effective speaker identity representations, so that the decoder can generate normal speech while retaining the dysarthric patients, and display their social roles or personalities through auditory features in their own timbre.

[0105] Refer again to Figure 1, the decoder 1400 is used to generate mel spectrograms. In one embodiment of the present application, the decoder 1400 may include two 512-dimensional FC layers, a 4-layer BLSTM with 320 units in each direction, and an 80-dimensional FC layer for predicting mel spectrograms.

[0106] Optionally, the concatenation of the speaker identity representation extracted from the speaker encoder 1300, the extended phoneme embedding, and the F0 sequence through repetition can be used as the input to the decoder 1400.

[0107] As shown in Equation (11), assuming represents the parameters of the speaker encoder 1300 after being trained on the speaker verification task, then the decoder 1400 with parameters θ g can predict the mel spectrogram as follows:

[0108] z = f g (p, v, e; θ g ) (11)

[0109] where z is the mel spectrogram predicted by the decoder; θ g are the parameters of the decoder; p is the predicted extended phoneme embedding; v is the predicted F0 sequence; and e is the saved speaker identity.

[0110] In addition, as an option, the decoder 1400 can also be trained by using normal speech data from a group of healthy speakers . Each speaker has a training data set T si = {(m j , p j , v j )}, where each training data corresponds to an utterance and includes a mel spectrogram m j , an extended phoneme embedding p j and a pitch feature v j . Then, the decoder 1400 can be optimized by minimizing the generated training loss . Optionally, as shown in Equations (12) and (13), the training loss can be the L2 norm between the predicted mel spectrogram and the mel spectrogram m in the training data j . In other words, the training loss of the decoder is the L2 norm between the predicted speech waveform and the mel spectrogram of the real speech:

[0111]

[0112] where, is the training loss of the decoder; is the predicted Mel spectrogram;

[0113] and m j is the Mel spectrogram in the training data.

[0114]

[0115] Wherein, is the predicted Mel spectrogram; m j is the Mel spectrogram of the j-th speaker in the training data; θ g are the parameters of the decoder; p j is the extended phoneme embedding of the j-th speaker in the training data; v j is the pitch feature of the j-th speaker in the training data; is the embedding vector of the utterance of the j-th speaker; and is a single vector representing the utterance of the j-th speaker in the speaker encoder output.

[0116] In the reconstruction stage, the DSR system receives the dysarthric speech of speaker s d and generates a reconstructed Mel spectrogram. The reconstructed Mel spectrogram can be expressed as Where is the phoneme embedding extracted by the fine-tuned content encoder Φ sd and extended with the predicted normal duration; is the predicted F0 sequence; θ g are the parameters of the decoder; and e sv is the speaker identity representation. Optionally, using, for example, PWG (ParallelWaveGAN) as a vocoder, can be converted into a speech waveform corresponding to the dysarthric speech.

[0117] Therefore, each encoder of the DSR system 1000 generates meaningful intermediate outputs that can be used by the decoder, so the DSR system 1000 provided in this application is interpretable. For example, the content encoder can derive a symbol-level representation of the speech content; the prosody encoder can infer the normal values of phoneme durations and F0, which can be used as standard values to replace the corresponding abnormal values; the speaker encoder generates a speaker identity representation, which can provide the necessary acoustic characteristics of the speaker. In addition, the decoder can aggregate the above multiple representations to generate a normal Mel spectrogram.

[0118] To verify the effectiveness of introducing visual information in improving DSR performance, performance comparisons were made from the following three aspects.

[0119] (1) As Figure 2 , Figure 3A andFigure 3B Performance Comparison of the Three Types of Encoders Shown

[0120] Specifically, in some embodiments of the present application, the speech datasets UASpeech, LJSpeech, and VCTK are selected for experiments. The UASpeech corpus is a benchmark unordered speech corpus with some background noise, which is recorded by an 8-channel microphone array and a camera, and 7 channels are segmented into single-word audio. PWG (Parallel WaveGAN) is used as a vocoder to synthesize the converted mel spectrogram. The VCTK corpus of 105 native speakers is used to train the speaker encoder, mel decoder, and PWG. LJSpeech, which is normal speech from female speakers, is used to train the duration predictor and F0 predictor in the prosody adapter.

[0121] Since the severity of dysarthria varies among different patients, which increases the modeling difficulty of building a general DSR system for all patients, DSR systems related to four speakers are respectively constructed for the four speakers with the lowest speech clarity, including two male speakers (M12 and M16) and two female speakers (F02 and F04). Figure 8 Exemplary video snapshots of the four dysarthric speakers speaking are shown. Table 1 shows the speech intelligibility of the four selected speakers, that is, the correct percentage during word transcription.

[0122] First, the above three types of content encoders are trained using the entire dysarthric speech dataset of all speakers, and then the content identified from the target speaker is fine-tuned to improve the phoneme prediction accuracy. Optionally, the pre-training and fine-tuning training of the content encoder 1100 can both be performed by, for example, the Adadelta optimizer. During the optimization process, the step sizes can be 1 million steps and 2 thousand steps respectively, the learning rate can be 1, and the batch size can be 8.

[0123] In addition, both the phoneme duration predictor 1201 and the F0 predictor 1202 can be, for example, the Adam optimizer, where is 0.001, the batch size is 16, and it lasts for 30k steps. Additionally, the decoder can be optimized in a similar manner, where the number of training steps can be set to, for example, 50 thousand steps.

[0124] The experimental results show that, as Figure 3B shown, the model using the AV-Hubert transform encoder obtains the optimal result, with its average phoneme error rate PER reduced by 8.5%. In addition, it should also be noted that as the speech clarity decreases, the difficulty of reconstruction gradually increases, and at this time, the performance improvement brought by visual features becomes more obvious, from a PER reduction of 7.0% to a reduction of 10.0%.

[0125] (2) Objective comparison of speech intelligibility

[0126] To show that there is indeed an improvement in intelligibility when comparing the final reconstructed speech with the original dysarthric speech, an open - source ASR model is used and word error rate (WER) is obtained through greedy decoding. Among them, for the original dysarthric speech "Original", the PWG vocoder uses the original mel - spectrogram to generate the synthetic speech waveform Original(Mel + PWG). As shown in Table 1 below, for all four speakers, the synthetic speech waveform Original(Mel + PWG) is inferior to the original dysarthric speech "Original", which indicates that the PWG vocoder reduces the speech quality.

[0127] Table 1 Performance comparison between multi - modal encoder and single - audio encoder

[0128]

[0129] The experimental results show that compared with A - DSR, the pre - trained AV - Hubert model further improves the WER performance. For the results of all four speakers selected in the experiment, the average WER is reduced by 8.2%.

[0130] Compared with the original dysarthric speech, the average WER of the speech reconstructed by the AV - Hubert model is reduced by 32.3%. Among them, for the speaker M12 with the most severe dysarthria, its WER is reduced by 42.1%. Similarly, and importantly, as the intelligibility of the original speech decreases, the WER decreases, and the improvement in performance by DSR based on AV - Hubert becomes more and more obvious.

[0131] Both PER and WER results indicate that for speakers with extremely severe dysarthria, relying solely on auditory features is insufficient, and in these cases, visual features can play an important role.

[0132] 2) Subjective comparison of speech naturalness

[0133] Through the Mean Opinion Score (MOS) test, the effectiveness of the method in improving the clarity and naturalness of the reconstructed speech is verified. Ten subjects are invited to evaluate the speech clarity and naturalness of the test utterances using a 5 - level MOS test (1 - extremely poor, 2 - poor, 3 - qualified, 4 - good, 5 - excellent). Ten test utterances randomly selected from speakers M12, F02, M16, and F04 are evaluated.

[0134] Figure 9 The figure shows the result comparison of MOS tests for the original dysarthric speech and the speech reconstructed by the A - DSR, AV - DSR, and AV - Hubert - DSR systems.

[0135] On the one hand, compared with the original dysarthric speech, all three DSR systems (A-DSR, AV-DSR, AV-Hubert-DSR) have achieved significant improvements in subjective speech naturalness. In particular, AV-Hubert-DSR achieved the best results, with a MOS score of approximately 4 levels.

[0136] On the other hand, compared with the A-DSR system, AV-DSR and AV-Hubert obtained better MOS scores for all speakers. In particular, the improvement brought by introducing visual features was most obvious for the speaker M12 with the most severe dysarthria, increasing from 3.33 of A-DSR to 3.98 of AV-Hubert-DSR.

[0137] It can be seen that the MOS test results are consistent with the objective experimental results, further indicating that visual features are extremely helpful for the DSR task, especially for speakers with more severe dysarthria symptoms.

[0138] The above-mentioned various experimental results all show that compared with the traditional A-DSR (audio dysarthric speech reconstruction), the AV-DSR (audio-visual - dysarthric speech reconstruction) proposed in this paper, which combines visual features, has obvious improvements in speech intelligibility, clarity, and naturalness, especially for speakers with more severe dysarthria symptoms.

[0139] Figure 6 It is a flowchart of a dysarthric speech reconstruction method 2000 according to an embodiment of the present application.

[0140] As Figure 6 shown, another aspect of the present application also provides a dysarthric speech reconstruction method 2000, including:

[0141] S1: Extract multimodal features from the multimodal input of dysarthric speech and generate phoneme embeddings based on the multimodal features;

[0142] S2: Predict the phoneme duration sequence based on the phoneme embeddings and generate extended phoneme embeddings and predicted pitch sequences;

[0143] S3: Extract the speaker identity representation of the dysarthric speech; and

[0144] S4: Reconstruct the normal speech waveform corresponding to the dysarthric speech based on the extended phoneme embeddings, predicted pitch sequences, and speaker identity representations.

[0145] According to an embodiment of the present application, in step S1, extracting multimodal features from the multimodal input of dysarthric speech may include: extracting visual features and auditory features from the visual input and auditory input of the dysarthric person respectively; fusing the extracted visual features and auditory features to obtain fused audiovisual features; and generating phoneme embeddings based on the fused audiovisual features.

[0146] According to an exemplary embodiment of the present application, in step S1, fusing the extracted visual features and auditory features includes: aligning the extracted visual features and auditory features, concatenating the aligned visual features and auditory features in the feature dimension, and performing audiovisual feature fusion using a fully connected layer.

[0147] According to an exemplary embodiment of the present application, in step S1, generating phoneme embeddings based on the fused audiovisual features includes: generating phoneme embeddings using an autoregressive sequence-to-sequence model based on a VGG convolutional network based on the fused audiovisual features.

[0148] According to an exemplary embodiment of the present application, in step S1, based on the fused audiovisual features, a pre-trained AV-Hubert audiovisual encoder is used to generate phoneme embeddings, where the AV-Hubert audiovisual encoder obtains the mapping relationship between the fused audiovisual features and phoneme embeddings by training a deep temporal neural network mapping model.

[0149] According to an embodiment of the present application, in step S2, generating a predicted pitch sequence may include: extending the phoneme embeddings to extended phoneme embeddings based on the predicted phoneme duration sequence; and generating a predicted pitch sequence based on the extended phoneme embeddings.

[0150] According to an exemplary embodiment of the present application, in step S2, a prosody encoder including a phoneme duration predictor and a pitch predictor may be used to generate a predicted phoneme duration sequence, extended phoneme embeddings, and a predicted pitch sequence, where the dysarthric speech reconstruction method 2000 further includes training the phoneme duration predictor and the pitch predictor respectively with a plurality of normal speech data. During the training, the L1 norm between the predicted phoneme duration sequence and the true duration sequence of the normal speech data is used as the training loss of the phoneme duration predictor, and the L1 norm between the predicted pitch sequence and the pitch ground truth of the normal speech data is used as the training loss of the pitch predictor.

[0151] Additionally, according to an embodiment of the present application, in step S3, extracting the speaker identity representation of the dysarthric speech may include: using a speaker encoder pre-trained with a speech dataset including a plurality of normal speech data to extract the speaker identity representation, wherein the dysarthric speech reconstruction method 2000 further includes: obtaining all the embedding vectors of the speech dataset, where each embedding vector is the L2 normalization norm of the output of the speaker encoder; based on all the embedding vectors, obtaining the centroid of all the utterances of each speaker in the speech dataset; based on the centroid of all the utterances of each speaker, obtaining a set of similarity matrices for each embedding vector; applying a classification function to the set of similarity matrices, and based on the set of similarity matrices after applying the classification function, determining the training loss of the speaker encoder to obtain the speaker identity representation.

[0152] Additionally, according to an embodiment of the present application, in step S4, reconstructing the normal speech waveform corresponding to the dysarthric speech includes: using a decoder to reconstruct the normal speech waveform corresponding to the dysarthric speech, wherein the dysarthric speech reconstruction method 2000 further includes: training the decoder with a plurality of normal speech data and generating a predicted speech waveform corresponding to the true speech waveform of the normal speech data, wherein, during training, the L2 norm between the predicted speech and the Mel spectrogram of the true speech is used as the training loss of the decoder.

[0153] Since the dysarthric speech reconstruction method provided by the present application uses the multi-modal dysarthric speech reconstruction system provided by the present application, it has the same beneficial effects as the multi-modal dysarthric speech reconstruction system, which will not be elaborated here.

[0154] Figure 7 FIG. shows a schematic block diagram of an electronic device 700 that can implement the dysarthric speech reconstruction method according to various embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0155] As Figure 7As shown, the electronic device 700 includes a computing unit 701. The computing unit 701 can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0156] The computing unit 701 can be various general-purpose and / or dedicated processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as the dysarthric speech reconstruction method.

[0157] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disc, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0158] It should be noted that although the electronic device 700 is shown as having various components with data processing capabilities, it is not intended to represent any specific architecture or manner of interconnecting the components; because such details have no close relationship with the embodiments of the present disclosure. It should also be recognized that a network computer, a handheld computer, a mobile phone, a server, and / or other data processing systems with fewer components or possibly more components can also be used in conjunction with the embodiments of the present disclosure.

[0159] In some embodiments, the dysarthria speech reconstruction method 2000 described above may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the dysarthria speech reconstruction method described above may be performed. Optionally, in other embodiments, the computing unit 701 may be configured to perform the dysarthria speech reconstruction method by any other suitable means (e.g., by means of firmware).

[0160] Various embodiments of the systems and techniques described above in this document may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] The program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes may be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0162] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may include or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing.

[0163] More specific examples of machine-readable storage media would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0164] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT cathode ray tube or an LCD liquid crystal display monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0165] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0166] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0167] The above description is only for the implementation mode of this application and the explanation of the applied technical principles. Those skilled in the art should understand that the protection scope involved in this application is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the technical concept. For example, the technical solution formed by mutually replacing the above features with the technical features (but not limited to) disclosed in this application that have similar functions.

Claims

1. A multi-modal dysarthria speech reconstruction system, characterized in that, The system includes: An encoder, configured to: Extract multimodal features from the multimodal input of dysarthric speech, and generate phoneme embeddings based on the multimodal features; Based on the phoneme embeddings, generate a predicted phoneme duration sequence, an Extended phoneme embedding, and a predicted pitch sequence; and Extract the speaker identity representation of the dysarthric speech; and A decoder, configured to: take the extended phoneme embedding, the predicted pitch sequence, and the speaker identity representation as inputs, and reconstruct a normal speech waveform corresponding to the dysarthric speech.

2. The multi-modal dysarthria speech reconstruction system according to claim 1, wherein, The encoder includes a multimodal encoder, and the multimodal encoder is configured to: Extract visual features and auditory features from the visual input and auditory input of the dysarthric person respectively; Fuse the extracted visual features and auditory features to obtain fused audiovisual features; And Generate the phoneme embeddings based on the fused audiovisual features.

3. The multi-modal dysarthria speech reconstruction system according to claim 2, wherein, The multimodal encoder is further configured to: Align the extracted visual features and auditory features, concatenate the aligned visual features and auditory features in the feature dimension, and perform audiovisual feature fusion using a fully connected layer.

4. The multi-modal dysarthria speech reconstruction system according to claim 2, wherein, The multimodal encoder is further configured to: Based on the fused audiovisual features, use an autoregressive sequence-to-sequence model based on a VGG convolutional network to generate the phoneme embeddings.

5. The multimodal dysarthric speech reconstruction system according to claim 2, wherein, The multimodal encoder is further configured to: Based on the fused audiovisual features, use a pre-trained AV-Hubert audiovisual encoder to generate the phoneme embeddings, where the AV-Hubert audiovisual encoder obtains the mapping relationship between the fused audiovisual features and the phoneme embeddings by training a deep temporal neural network mapping model.

6. The multi-modal dysarthria speech reconstruction system according to claim 1, wherein, The encoder includes: A prosody encoder, configured to: predict the phoneme duration sequence based on the phoneme embeddings; extend the phoneme embeddings to the extended phoneme embeddings based on the phoneme duration sequence; and generate the predicted pitch sequence based on the extended phoneme embeddings.

7. The multimodal dysarthric speech reconstruction system according to claim 6, wherein, The prosody encoder includes a phoneme duration predictor and a pitch predictor, Wherein, the phoneme duration predictor and the pitch predictor are configured to be trained respectively with a plurality of normal speech data, wherein the training loss of the phoneme duration predictor adopts the L1 norm between the predicted phoneme duration sequence and the true duration sequence of the normal speech data, and the training loss of the pitch predictor adopts the L1 norm between the predicted pitch sequence and the pitch true value of the normal speech data.

8. The multimodal dysarthric speech reconstruction system according to claim 1, wherein, The encoder includes: A speaker encoder, configured to be trained in advance with a plurality of normal speech data to extract the speaker identity representation, Wherein, the speaker encoder is further configured to: Obtain all the embedding vectors of the speech data set for training, where each of the embedding vectors is the L2 normalization norm of the output of the speaker encoder; Based on all the embedding vectors, obtain the centroid of all the utterances of each speaker in the speech data set; Obtain a set of similarity matrices for each of the embedding vectors based on the centroid of all the utterances of each of the speakers; Apply a classification function to the set of similarity matrices, and determine the training loss of the speaker encoder based on the set of similarity matrices after applying the classification function to extract the speaker identity representation.

9. The multimodal dysarthric speech reconstruction system according to claim 1, wherein, The decoder is further configured to be trained with a plurality of normal speech data, wherein the training loss of the decoder uses the L2 norm between the predicted speech and the Mel spectrogram of the ground truth speech.

10. A method for reconstructing dysarthric speech, characterized in that, The method includes: Extract multimodal features from the multimodal input of dysarthric speech, and generate phoneme embeddings based on the multimodal features; Based on the phoneme embeddings, generate a predicted phoneme duration sequence, extended phoneme embeddings, and a predicted pitch sequence; Extract the speaker identity representation of the dysarthric speech; and Based on the extended phoneme embeddings, the predicted pitch sequence, and the speaker identity representation, reconstruct the normal speech waveform corresponding to the dysarthric speech.

11. The method according to claim 10, wherein, Extracting the multimodal features from the multimodal input of the dysarthric speech includes: Extract visual features and auditory features from the visual input and auditory input of the dysarthric person respectively; Fuse the extracted visual features and auditory features to obtain fused audio-visual features; and Generate the phoneme embeddings based on the fused audio-visual features.

12. The method according to claim 11, wherein, Fusing the extracted visual features and auditory features includes: Align the extracted visual features and auditory features, concatenate the aligned visual features and auditory features in the feature dimension, and perform audio-visual feature fusion using a fully connected layer.

13. The method according to claim 11, wherein, Generating the phoneme embeddings based on the fused audio-visual features includes: Based on the fused audio-visual features, use an autoregressive sequence-to-sequence model based on the VGG convolutional network to generate the phoneme embeddings.

14. The method according to claim 11, wherein, Generating the phoneme embeddings based on the fused audio-visual features includes: Based on the fused audio-visual features, use a pre-trained AV-Hubert audio-visual encoder to generate the phoneme embeddings, wherein the AV-Hubert audio-visual encoder obtains the mapping relationship between the fused audio-visual features and the phoneme embeddings by training a deep temporal neural network mapping model.

15. The method according to claim 10, wherein Generating the predicted pitch sequence includes: Predict the phoneme duration sequence based on the phoneme embeddings; extend the phoneme embeddings to the extended phoneme embeddings based on the phoneme duration sequence; and generate the predicted pitch sequence based on the extended phoneme embeddings.

16. The method according to claim 10, wherein, Use a prosody encoder including a phoneme duration predictor and a pitch predictor to generate the predicted phoneme duration sequence, the extended phoneme embeddings, and the predicted pitch sequence; And Wherein, the method further includes: The phoneme duration predictor and the pitch predictor are respectively trained with multiple normal speech data. Among them, the phoneme duration predictor and the pitch predictor are respectively trained with multiple normal speech data. Among them, the training loss of the phoneme duration predictor uses the L1 norm between the predicted phoneme duration sequence and the true duration sequence of the normal speech data, and the training loss of the pitch predictor uses the L1 norm between the predicted pitch sequence and the pitch truth of the normal speech data.

17. The method according to claim 10, wherein Extracting the speaker identity representation includes: Using a speaker encoder pre-trained with a speech data set including multiple normal speech data to extract the speaker identity representation, Among them, extracting the speaker identity representation further includes: Obtaining all the embedding vectors of the speech data set, where each embedding vector is the L2 normalized norm of the output of the speaker encoder; Based on all the embedding vectors, obtaining the centroid of all the utterances of each speaker in the speech data set; Based on the centroid of all the utterances of each speaker, obtaining a set of similarity matrices for each embedding vector; and Applying a classification function to the set of similarity matrices, and based on the set of similarity matrices after applying the classification function, determining the training loss of the speaker encoder to extract the speaker identity representation.

18. The method according to claim 10, wherein Reconstructing the speech waveform includes: Using a decoder to reconstruct the speech waveform; and Among them, the method further includes: Training the decoder with multiple normal speech data and generating a predicted speech waveform corresponding to the true speech waveform of the normal speech data. Among them, the L2 norm between the predicted speech and the Mel spectrogram of the true speech is used as the training loss of the decoder.

19. An electronic device for multimodal dysarthric speech reconstruction, comprising: At least one processor; And A memory communicatively connected to the at least one processor; Among them, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the method according to any one of claims 10-18.

20. A computer-readable storage medium storing computer instructions, wherein, When the computer instructions are executed, the computer executes the method according to any one of claims 10-18.

Images