A dysarthria speech synthesis method, device, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a dysarthria speech synthesis method, device, equipment and medium, which comprises the following steps: performing phoneme coding on a text sequence to obtain a phoneme coding sequence; inputting the phoneme coding sequence into a pause predictor to perform pause prediction and obtain a pause prediction result; generating corresponding pause embedding and inserting the pause embedding into the phoneme coding sequence to obtain an enhanced phoneme coding sequence; inputting the enhanced phoneme coding sequence into a duration predictor to perform duration prediction and obtain a predicted duration; obtaining a reference mel spectrum, extracting global and frame-level local style vectors, and fusing the global and frame-level local style vectors with the enhanced phoneme coding sequence to obtain a pathological acoustic style conditional mean; and inputting the predicted duration and the pathological acoustic style conditional mean into a flow matching network to generate dysarthria style speech through conditional flow matching. The application can be applied to the fields of financial technology and medical health, can improve the pathological authenticity of synthesized speech, and can provide enhanced data for an automatic speech recognition system.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and natural language processing technology, and in particular to a method, apparatus, device and medium for synthesizing speech in articulation disorders. Background Technology

[0002] Articulation disorder is a motor speech disorder caused by damage to the nervous system. Its typical clinical manifestations include significantly slowed speech rate, unclear pronunciation, abnormal rhythmic breaks, and irregular pauses. These pathological features result in a significant difference between the temporal distribution of dysarthria speech and normal speech, posing a serious challenge to automatic speech recognition systems that rely on large-scale normal speech data for training.

[0003] In the fintech sector, voice interaction is being widely used in intelligent customer service, remote identity verification, and transaction instruction confirmation. In the healthcare sector, electronic medical record entry, rehabilitation assessment, and remote consultation systems are also increasingly reliant on voice recognition technology. When these systems encounter voice input from users with speech impairments, their recognition accuracy often drops sharply, directly impacting the efficiency of financial transactions and the quality of medical services.

[0004] In existing technologies, to alleviate the scarcity of speech data for articulation disorders, text-to-speech (TTS) synthesis techniques are commonly used to generate augmented data. However, current TTS synthesis methods, such as frameworks based on FastSpeech2 or diffusion models, typically train their duration prediction modules based on the statistical regularities of normal speech to generate smooth and coherent speech. When these methods are directly applied to simulate pathological speech, the generated synthesized speech still exhibits a rhythmic smoothness and coherence that tends towards normal speech, making it difficult to accurately reproduce the rhythmic breaks and abnormal pauses unique to articulation disorders. Furthermore, this significant deviation in rhythmic statistical distribution between the synthesized speech and real pathological speech means that the generated augmented data cannot effectively represent real pathological speech samples, resulting in limited performance improvements for articulation disorder speech recognition systems when used for fine-tuning. Summary of the Invention

[0005] The present invention provides a method, apparatus, device and medium for synthesizing speech with articulation disorders, in order to solve the technical problem of: how to provide an effective solution for generating synthesized speech that is closer to the rhythm of real articulation disorders.

[0006] Firstly, a method for synthesizing speech with articulation disorders is provided, including: In response to receiving a text sequence, the text sequence is phoneme encoded to obtain a phoneme encoded sequence; The phoneme encoding sequence is input into the pause predictor to perform pause prediction and obtain the pause prediction result; Based on a preset mapping relationship, the pause embedding corresponding to the pause prediction result is obtained, and the pause embedding is inserted into the phoneme coding sequence to obtain the enhanced phoneme coding sequence. The enhanced phoneme coding sequence is input into the duration predictor to predict the duration and obtain the predicted duration. A reference Mel spectrum is obtained, a global style vector and a frame-level local style vector are extracted from the reference Mel spectrum, and the enhanced phoneme coding sequence is fused with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean. The predicted duration and the conditional mean of the pathological acoustic style are input into a flow matching network, and dysarthmic style speech is generated through conditional flow matching.

[0007] Secondly, a speech synthesis device for articulation disorders is provided, comprising: An encoding unit is configured to encode a text sequence into phonemes in response to receiving a text sequence, thereby obtaining a phoneme-encoded sequence. The pause prediction unit is used to input the phoneme encoding sequence into the pause predictor to perform pause prediction and obtain the pause prediction result. An insertion unit is used to obtain a pause embedding corresponding to the pause prediction result based on a preset mapping relationship, and insert the pause embedding into the phoneme coding sequence to obtain an enhanced phoneme coding sequence. The duration prediction unit is used to input the enhanced phoneme coding sequence into the duration predictor to predict the duration and obtain the predicted duration. The fusion unit is used to acquire a reference Mel spectrum, extract a global style vector and a frame-level local style vector from the reference Mel spectrum, and fuse the enhanced phoneme coding sequence with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean. The generation unit is used to input the predicted duration and the conditional mean of the pathological acoustic style into the flow matching network, and generate articulation disorder style speech through conditional flow matching.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described speech synthesis method for speech disorders.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described speech synthesis method for speech disorders.

[0010] In the aforementioned method, apparatus, computer equipment, and storage medium for synthesizing speech with articulation disorders, pause embeddings are generated and inserted into sequences based on pause prediction results. A duration predictor then outputs the predicted duration, thereby directly modeling the abnormal rhythms and pause distributions in pathological speech. Furthermore, global and frame-level local style vectors are extracted from the reference Mel spectrum and fused with the enhancement sequence, thus achieving a joint representation of the pathological acoustic style. Further, the rhythm and style features are input into a flow matching network, and the target speech is generated through conditional flow matching. Since both rhythm modeling and style representation are based on real pathological speech data, the generated speech closely approximates real speech with articulation disorders in terms of rhythm distribution and acoustic details, solving the problem of significant deviation between the rhythm of synthesized speech and the distribution of real pathological speech. This provides effective enhancement data for automatic speech recognition systems. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a speech synthesis method for articulation disorders according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of a speech synthesis method for articulation disorders according to an embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S2; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation of step S4; Figure 5 yes Figure 2 A flowchart illustrating a specific implementation of step S5; Figure 6 yes Figure 2 A flowchart illustrating a specific implementation of step S6; Figure 7 This is a schematic diagram of a speech synthesis device for articulation disorders according to one embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 9 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] The speech synthesis method for articulation disorders provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can receive a text sequence from the client and obtain a reference Mel spectrum; extract a phoneme encoding sequence from the text sequence; input the phoneme encoding sequence into a pause predictor to obtain a pause prediction result; generate a corresponding pause embedding based on the pause prediction result and insert the pause embedding into the phoneme encoding sequence to obtain an enhanced phoneme encoding sequence; input the enhanced phoneme encoding sequence into a duration predictor to obtain a predicted duration; extract a global style vector and a frame-level local style vector from the reference Mel spectrum, and fuse the enhanced phoneme encoding sequence with the global style vector and the frame-level local style vector to obtain a pathological acoustic style conditional mean; input the predicted duration and the pathological acoustic style conditional mean into a stream matching network, and generate articulation disorder style speech through conditional stream matching; finally, feed the generated articulation disorder style speech back to the client. Furthermore, this invention addresses the technical challenges of scarce speech data in articulation disorders and the difficulty in accurately simulating pathological rhythms and acoustic styles. It utilizes a rhythm- and style-aware speech synthesis scheme to generate a large number of speech samples that closely approximate real pathological distributions for enhanced training of automatic speech recognition systems. This scheme, through a multi-stage rhythm prediction mechanism and a two-level style fusion mechanism, effectively simulates the rhythmic breaks, abnormal pauses, and frame-level acoustic variations unique to articulation disorders, greatly improving the adaptability and recognition performance of automatic speech recognition systems to pathological speech. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0015] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the speech synthesis method for articulation disorders provided in this embodiment of the invention includes the following steps: S1, in response to receiving a text sequence, phoneme encoding is performed on the text sequence to obtain a phoneme encoded sequence.

[0016] In specific implementation, in response to receiving a text sequence, the text sequence is input into a phoneme encoder, which performs phoneme encoding on the text sequence to obtain a phoneme-encoded sequence. It should be noted that the phoneme-encoded sequence is a vector representation of natural language text converted into phoneme-level representations, denoted as E, which carries the pronunciation information of the speech to be synthesized.

[0017] S2, input the phoneme encoding sequence into the pause predictor to perform pause prediction and obtain the pause prediction result.

[0018] In practice, the phoneme encoding sequence is input into a pause predictor for pause prediction, and a pause prediction result is obtained. The pause predictor is used to predict whether a pause should occur at each phoneme position in the phoneme encoding sequence, and the specific length and category of the pause, thereby simulating the rhythmic discontinuity characteristics in articulatory disorders.

[0019] For example, see Figure 3 In some preferred embodiments, the step of inputting the phoneme encoding sequence into a pause predictor to perform pause prediction and obtain a pause prediction result includes: S21, inputting the phoneme encoding sequence into the pause predictor, and calculating the probability that each phoneme position in the phoneme encoding sequence belongs to a pause category of different lengths through the pause predictor; S22, using the probability of the pause category of different lengths corresponding to each phoneme position as the pause prediction result.

[0020] In specific implementation, the phoneme encoding sequence E is input into the pause predictor. The pause predictor calculates the probability that each phoneme position in the phoneme encoding sequence belongs to a pause category of different lengths. Further, the output of the pause predictor is defined as: ;in, To pause the prediction results, f s θ Let θ be the pause predictor function. The pause predictor uses the cross-entropy loss function during training, and its formula is: Where N is the total length of the input sequence, i is the phoneme position index, K is the total number of preset pause categories (e.g., K=4, corresponding to no pause, short pause, medium pause, and long pause respectively), k is the category index, and s i For the actual pause label of the i-th phoneme, Let I(·) be the probability that the i-th phoneme belongs to the k-th pause category, and let I(·) be the indicator function. The probability of the pause category of different lengths corresponding to each phoneme position is taken as the pause prediction result.

[0021] It should be noted that the network architecture of the pause predictor consists of a multi-layer one-dimensional convolutional neural network and a self-attention mechanism. The convolutional layers are used to extract local features of the phoneme sequence, and the self-attention layers are used to capture long-range dependencies in the sequence. Finally, a fully connected layer outputs the probability distribution of each phoneme position belonging to different pause categories.

[0022] During training, the pause predictor uses the cross-entropy loss function as a supervision signal to guide the model in learning to accurately predict whether a pause should occur at each phoneme position and the length and category of the pause based on the context. The AdamW optimizer is used for parameter updates, and early stopping is employed to prevent overfitting.

[0023] Pause category labels in the training data were obtained through a combination of manual annotation and acoustic analysis. Silent segments in real speech were extracted using a speech activity detection tool and categorized into four types based on their actual duration: no pause, short pause, medium pause, and long pause. The annotation process was cross-validated by professional speech therapists to ensure accuracy.

[0024] This embodiment introduces a multi-class pause prediction mechanism to achieve refined modeling of irregular pauses in articulation disorder speech. Compared to existing binary classification methods that only determine whether a pause occurs, the multi-class probabilities output in this embodiment provide richer quantitative information for the subsequent generation of pause embeddings. This enables synthesized speech to accurately simulate pause patterns of varying durations and uneven distribution in real pathological speech, such as the alternation of long pauses, short pauses, and no pauses, thus fundamentally solving the problem of existing technologies' inability to simulate pathological rhythmic breaks.

[0025] S3. Based on a preset mapping relationship, obtain the pause embedding corresponding to the pause prediction result, and insert the pause embedding into the phoneme coding sequence to obtain the enhanced phoneme coding sequence.

[0026] In specific implementation, based on a preset mapping relationship, a pause embedding corresponding to the pause prediction result is obtained, and the pause embedding is inserted into the phoneme coding sequence to obtain an enhanced phoneme coding sequence. The pause embedding is a vector carrying pause length information; inserting it between corresponding phoneme positions in the phoneme coding sequence can explicitly construct the context of the pathological rhythm. This output sequence is denoted as E. aug .

[0027] Specifically, firstly, based on the pause prediction results, the optimal pause category corresponding to each phoneme position is determined. The pause prediction results include the probability distribution of each phoneme position belonging to different pause categories of different lengths. Based on this probability distribution, the pause category with the highest probability value is selected as the predicted pause category for that position. Further, the preset mapping relationship can specifically be a pre-constructed pause embedding table; this invention is not specifically limited to this. The pause embedding vector corresponding to the predicted pause category is found from the pause embedding table. The pause embedding table stores a unique vector representation for each preset pause category (e.g., no pause, short pause, medium pause, long pause). Further, an insertion operation is performed: an empty sequence list is initialized, and each phoneme vector in the phoneme encoding sequence is traversed sequentially. The currently traversed phoneme vector is added to the sequence list; simultaneously, it is determined whether the predicted pause category corresponding to the phoneme position is no pause; if the determination result is no (i.e., a pause exists), the found pause embedding vector is also added to the sequence list. Furthermore, after the traversal is completed, the sequence list is the enhanced phoneme encoded sequence, which is longer than the original sequence and explicitly embeds pause information, providing an accurate rhythmic context for subsequent duration prediction.

[0028] S4, input the enhanced phoneme encoding sequence into the duration predictor to predict the duration and obtain the predicted duration.

[0029] In practice, the enhanced phoneme encoding sequence is input into a duration predictor for duration prediction to obtain the predicted duration. The duration predictor is used to predict the duration of each phoneme in the enhanced sequence, and the predicted duration directly controls the speech rate and rhythm of the synthesized speech. In speech with articulation disorders, the predicted duration typically exhibits abnormal features such as prolongation or shortening.

[0030] See Figure 4 In some preferred embodiments, the step of inputting the enhanced phoneme encoding sequence into a duration predictor for duration prediction to obtain a predicted duration includes: S41, inputting the enhanced phoneme encoding sequence into the duration predictor, wherein the duration predictor is jointly trained using a logarithmic scale mean square error loss and a contrast preference optimization loss; S42, outputting the duration of each phoneme based on the enhanced phoneme encoding sequence through the duration predictor, and using the duration of each phoneme as the predicted duration; wherein the contrast preference optimization loss constrains the predicted duration output by the duration predictor to be closer to the distribution of articulation disorders by comparing the actual duration of articulation disorders with the duration of normal speech.

[0031] In specific implementation, the enhanced phoneme encoding sequence E aug Input the duration predictor. The duration predictor employs a log-scaled mean squared error loss L during training. dCompared with the contrast preference optimization loss L cp Joint optimization. Where, L d The formula is: Where N' is the length of the enhanced phoneme coding sequence, and d i Let be the actual duration of the articulation disorder of the i-th phoneme. This is the predicted duration output by the duration predictor. This loss ensures the basic accuracy of the predicted duration.

[0032] The contrast preference optimization loss L cp By comparing the actual duration of articulation disorders with the duration of normal speech, the predicted results are guided to converge towards the pathological distribution. The formula is as follows: Among them, w i Let d be the dynamic weight of the i-th phoneme, D(·,·) be the distance function (e.g., absolute difference), and d be the dynamic weight of the i-th phoneme. i n Let be the normal speech duration of the i-th phoneme, and m be the margin parameter. The dynamic weight w... i The adjustment is adaptive based on the pause prediction results, and its calculation formula is as follows: Where α and β are preset hyperparameters, For the i-th phoneme, it belongs to its true pause category s i The predicted probability, Let be the predicted probability that the i-th phoneme belongs to the no-pause category (k=0).

[0033] This embodiment introduces the contrast preference optimization loss L. cp and the dynamic weight w i This adds a guiding signal for pathological distribution to the traditional duration regression. The L... cp The core issue is punishing those who predict duration. Closer to normal duration d i n Non-pathological duration d i The situation regarding the dynamic weight w. i The system adaptively adjusts based on the pause prediction results, ensuring that within the pause region (s)... i > 0) The penalty for differences in pathological duration is more significant. This mechanism effectively solves the problem that ordinary duration regression models are easily affected by massive amounts of normal speech data during training, causing the prediction results to collapse towards normal speech, ensuring that the predicted duration more accurately reflects the distribution of pathological rhythm (such as duration prolongation), thereby greatly improving the pathological realism of synthesized speech at the rhythm level.

[0034] It should be noted that the duration predictor adopts an architecture combining multiple one-dimensional convolutional layers and fully connected layers. The convolutional layers are used to extract temporal features of the enhanced phoneme coding sequence, and the fully connected layers are used to map the predicted duration value of each phoneme.

[0035] Furthermore, the duration predictor employs a joint training strategy, with a loss function consisting of two parts: the first part is the logarithmic mean squared error loss, used to ensure the basic accuracy of the predicted duration; the second part is the contrast preference optimization loss, which guides the prediction results to approach the pathological distribution by comparing the actual duration of articulation disorders with the duration of normal speech. Specifically, for each phoneme, both its pathological duration and normal duration are provided as references; if the model predicts a duration closer to the normal duration, it will be penalized more severely.

[0036] Furthermore, the AdamW optimizer was used during training, and early stopping was employed to prevent overfitting. In addition, a dynamic weight mechanism was introduced during training, which adaptively adjusted the weights based on the output of the pause predictor, making the penalty for pathological duration differences in pause regions more significant, and further guiding the model to learn pathological rhythm features.

[0037] S5, obtain the reference Mel spectrum, extract the global style vector and the frame-level local style vector from the reference Mel spectrum, and fuse the enhanced phoneme coding sequence with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean.

[0038] In practice, a segment of actual speech from a patient with articulation disorders is obtained as a reference Mel spectrum, denoted as M. ref From the reference Mel spectrum M ref The global style vector and frame-level local style vector are extracted separately. The global style vector is denoted as A. g This is used to characterize the overall macroscopic acoustic features of the speaker, such as average pitch and timbre; the frame-level local style vector is denoted as A. l This is used to capture fine-grained pathological acoustic variations in each frame, such as transient defects like breath instability, voice tremor, or hoarseness. Subsequently, the enhanced phoneme encoding sequence E... aug With the global style vector A g The frame-level local style vector A l Multi-level fusion is performed to obtain the pathological acoustic style conditional mean, denoted as μ. The pathological acoustic style conditional mean μ is a multi-dimensional conditional vector that integrates text semantics, rhythmic features, and acoustic style.

[0039] See Figure 5In some preferred embodiments, the step of fusing the enhanced phoneme coding sequence with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean includes: S51, encoding the enhanced phoneme coding sequence to obtain intermediate features; S52, performing attention fusion between the frame-level local style vector and the intermediate features to obtain fused features; S53, concatenating the global style vector and the fused features to obtain the pathological acoustic style conditional mean.

[0040] In specific implementation, the enhanced phoneme encoding sequence E aug The input is processed by the encoder network to obtain intermediate features, denoted as H. These intermediate features H retain textual and rhythmic information and await interaction with acoustic style information.

[0041] Furthermore, from the reference Mel spectrum M ref Extract the frame-level local style vector A l Specifically, M is quantized through a vector quantization (VQ) bottleneck module. ref Each frame in the process is processed by retrieving the corresponding embedding vector from a pre-constructed discrete codebook (of size C×D, where C is the number of codebook entries, e.g., 512, and D is the embedding vector dimension, which is used as a learnable parameter in joint optimization with the model during training), thereby forming the frame-level local style vector. This mechanism enables the model to capture and reconstruct microscopic acoustic defects that change rapidly over time in pathological speech, such as localized breath instability or vocal tremor.

[0042] Furthermore, from the reference Mel spectrum M ref Extract the global style vector A g Specifically, M is processed through an attention pooling network. ref This yields a fixed global vector that encodes the speaker’s overall acoustic characteristics, such as timbre and pitch range.

[0043] Furthermore, the frame-level local style vector A l Attention fusion is performed with the intermediate feature H to obtain the fused feature. Finally, the global style vector A is... g By concatenating the fusion features, the conditional mean μ of the pathological acoustic style is obtained, and its formula is: Here, Encoder represents the overall functionality of the encoder and the fusion network.

[0044] This embodiment achieves fine-grained modeling of pathological acoustic styles by constructing a two-level style coding and fusion mechanism that combines global and local styles. The frame-level local style vector A... lThe introduction of this approach enables the scheme to learn and simulate subtle acoustic variations at the frame level in pathological speech, solving the problem that existing technologies, which only use global speaker embedding, cannot capture pathological details (such as breath instability). Furthermore, by dynamically associating the local style with the intermediate features through an attention mechanism, the resulting pathological acoustic style conditional mean μ becomes an informative and structurally sound guiding condition, laying a solid foundation for subsequent high-quality speech generation.

[0045] In some preferred embodiments, the step of performing attention fusion on the frame-level local style vector and the intermediate features to obtain fused features includes: using the frame-level local style vector as a query vector and the intermediate features as a key vector and a value vector, calculating attention weights; and performing a weighted summation on the intermediate features based on the attention weights to obtain the fused features.

[0046] In specific implementation, the frame-level local style vector A l As the query vector Q, the intermediate feature H is used as the key vector K and the value vector V. The attention weight is calculated using the following formula: Attention(Q, K, V) = softmax(QK T / √d k V Where, d k The scaling factor is used. The attention weights characterize the correlation strength between each frame-level local style vector and each intermediate feature vector. Based on the attention weights, the intermediate features V are weighted and summed to obtain the fused features. This process enables each frame-level local style to "focus" on the most relevant text / rhythm information from the global intermediate features.

[0047] This embodiment employs an explicit scaling dot product attention mechanism to ensure high-precision, dynamic contextual association between frame-level style information and text-rhythm information. Unlike simple static fusion methods such as concatenation or summation, this attention mechanism adaptively determines which textual information should be combined with which pathological acoustic details based on the characteristics of the input data. This generates a highly coordinated fusion representation of acoustic and rhythmic features, providing optimal guidance for the subsequent flow matching network.

[0048] S6, input the predicted duration and the conditional mean of the pathological acoustic style into the flow matching network, and generate articulation disorder style speech through conditional flow matching.

[0049] In practice, the predicted duration and the conditional mean μ of the pathological acoustic style are input into a flow matching network, and speech in the style of articulation disorder is generated through conditional flow matching. The flow matching network is a generative model based on ordinary differential equations, which can gradually derive the Mel spectrum that conforms to the target pathological characteristics from the noise distribution based on the input conditional information.

[0050] The network architecture of the stream matching network consists of multiple fully connected layers, with conditional layers added between the layers for normalization to enhance the guiding effect of conditional information. The input includes the current state vector, the time step, and the combined conditional vector (composed of the prediction duration and the mean of the pathological acoustic style conditions), and the output is the vector field at the current time step.

[0051] Furthermore, during training, the flow matching network samples an initial noise vector from a standard normal distribution, uses a combined condition vector as a guiding condition, and iteratively solves the problem through an ordinary differential equation solver. In each iteration, the network upsamples and expands the generated intermediate Mel spectrum based on the prediction duration information to ensure that its time dimension is strictly aligned with the prediction duration.

[0052] Furthermore, training employs a conditional flow matching loss function, uses the AdamW optimizer, and adopts a cosine annealing learning rate decay strategy. After training, the network is able to generate Mel spectra with realistic pathological rhythms and acoustic details from noise distributions.

[0053] This embodiment constructs a complete end-to-end speech synthesis framework for articulation disorders by organically combining a multi-stage rhythm prediction mechanism with a two-level style modeling mechanism. The multi-stage rhythm prediction mechanism accurately simulates the rhythmic breaks and irregular pauses unique to articulation disorder speech by first predicting pauses and then their duration. The two-level style modeling mechanism utilizes both global and frame-level local style vectors to not only control the overall timbre but also finely simulate frame-level pathological acoustic details. The combination of these two mechanisms ensures that the synthesized speech closely resembles the pronunciation of real patients in both rhythm distribution and acoustic details, solving the technical problem that existing synthesized speech rhythms are "too normal" to represent real pathological samples. Furthermore, this provides high-quality augmented data for automatic speech recognition systems, significantly improving their performance in recognizing the speech of users with articulation disorders in fintech or healthcare scenarios.

[0054] See Figure 6In some preferred embodiments, the step of inputting the predicted duration and the pathological acoustic style conditional mean into a flow matching network to generate articulation disorder-style speech through conditional flow matching includes: S61, concatenating the predicted duration and the pathological acoustic style conditional mean to obtain a combined conditional vector; S62, inputting the combined conditional vector into the flow matching network, and performing a conditional flow matching generation process based on the combined conditional vector through the flow matching network to generate an articulation disorder-style Mel spectrum from the noise distribution, wherein the conditional flow matching generation process includes: upsampling the predicted duration to align the time dimension of the articulation disorder-style Mel spectrum with the predicted duration; S63, inputting the articulation disorder-style Mel spectrum into a vocoder to synthesize the articulation disorder-style speech.

[0055] In practice, the predicted duration and the mean value μ of the pathological acoustic style conditions are concatenated to obtain a combined condition vector, denoted as C. The combined condition vector C contains both temporal structure information (the predicted duration) and acoustic spatial information (the mean value μ of the pathological acoustic style conditions).

[0056] Further, the combined condition vector C is input into the flow matching network. The flow matching network learns a time-dependent vector field v. t (x|μ;θ), this vector field defines how to smoothly "push" a point x in the noise distribution toward the target data distribution.

[0057] Furthermore, a conditional flow matching generation process is performed through the flow matching network to generate a melanogram of articulation disorder style from the noise distribution. The conditional flow matching generation process includes upsampling the prediction duration to align the time dimension of the melanogram of articulation disorder style with the prediction duration. Finally, the melanogram of articulation disorder style is input into a vocoder to synthesize the articulation disorder style speech.

[0058] This embodiment utilizes the conditional flow matching network to precisely guide the pathological speech generation process by using complex rhythms, styles, and textual features as joint conditions. Compared to traditional autoregressive or diffusion-based models, the flow matching model achieves a better balance between generation speed and quality. By upsampling the predicted duration, it ensures that the generated Mel spectrum is strictly aligned with the predicted pathological rhythm in terms of temporal structure, fundamentally avoiding the problem of rhythmic information loss or misalignment during acoustic mapping.

[0059] In some preferred embodiments, the step of generating a melancholic style Mel spectrum from a noise distribution by performing a conditional flow matching generation process based on the combined conditional vector through the flow matching network includes: S621, sampling an initial noise vector from a standard normal distribution through the flow matching network; S622, using the combined conditional vector as a guiding condition, iteratively solving an ordinary differential equation through a preset ordinary differential equation solver via the flow matching network, wherein, during the iterative solution process, the flow matching network performs upsampling expansion based on the prediction duration, aligning the time dimension of the intermediate Mel spectrum generated during the iterative solution process with the prediction duration; S623, outputting the final Mel spectrum obtained from the iterative solution as the melancholic style Mel spectrum.

[0060] In specific implementation, an initial noise vector x0 is sampled from a standard normal distribution using the flow matching network. The flow matching network then uses the combined condition vector C as a guiding condition and a preset ordinary differential equation solver to iteratively solve the ordinary differential equation. The formula for the ordinary differential equation is: Furthermore, during the iterative solution process, the flow matching network performs upsampling expansion based on the prediction duration, aligning the time dimension of the intermediate Mel spectra generated during the iterative solution process with the prediction duration. Finally, the final Mel spectrum x1 obtained from the iterative solution is used as the Mel spectrum output for the articulation disorder style.

[0061] This embodiment provides a mathematically rigorous and differentiable learning framework for pathological speech generation through conditional flow matching and solving ordinary differential equations. Compared to ordinary diffusion models, the flow matching model allows for more flexible selection of solvers, balancing generation speed and quality. Furthermore, by using the combined conditional vector C as the guiding condition for solving the ordinary differential equations, this embodiment ensures that the generated Mel spectrum accurately matches the target pathological features in both time and spectral dimensions. Furthermore, the upsampling expansion mechanism during the iterative solution process further guarantees that the rhythmic structure of the synthesized speech is strictly consistent with the predicted duration, thereby generating high-quality articulation disorder speech with both correct rhythm and realistic timbre.

[0062] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0063] Specific application examples of this invention are as follows: In remote identity verification scenarios within the fintech field, when users register their voiceprints or confirm transaction instructions via mobile clients, if the user suffers from articulation disorders, the rhythmic breaks and abnormal pauses characteristic of their speech signals will significantly reduce the accuracy of traditional voiceprint recognition or voice command recognition systems. The method provided by this invention can be deployed on the server side of financial services for enhanced training of speech data from users with articulation disorders. Specifically, when the server receives a registration request from a user, it first obtains a small amount of the user's actual speech as a reference Mel spectrum, extracting its pathological global and frame-level local style vectors. Simultaneously, it performs phoneme encoding, pause prediction, and duration prediction based on the user's read identity identifier text sequence to generate an enhanced phoneme encoding sequence. Finally, a large number of articulation disorder-style speech samples simulating the user's pathological characteristics are synthesized through a stream matching network. Furthermore, these synthesized speech samples are used to fine-tune the backend voiceprint recognition or voice command recognition model, enabling the system to accurately identify the user's subsequent actual articulation disorder speech. This effectively completes remote identity verification and transaction confirmation without requiring repeated recordings from the user, significantly improving the accessibility and inclusiveness of financial services.

[0064] Furthermore, in rehabilitation assessments and remote consultations within the healthcare field, the speech characteristics of patients with dysarthria are often used as important indicators for assessing their neurological recovery. However, traditional automatic speech recognition systems struggle to accurately transcribe pathological speech, leading to low efficiency in electronic medical record entry. The method provided by this invention can be integrated into the data processing center of a medical information system. When a doctor needs to assess the speech rehabilitation progress of a patient with dysarthria, the system can retrieve the patient's past pathological speech as a reference Mel spectrum, extract its global and frame-level local style vectors, and combine them with the doctor's input assessment text sequence. Through pause prediction, duration prediction, and conditional flow matching, a large number of synthesized speech samples highly similar to the patient's pathological characteristics are generated. Furthermore, these synthesized speech samples are used to fine-tune a medical-specific speech recognition engine, enabling it to accurately recognize the patient's actual pronunciation, thereby automatically completing rehabilitation assessment scale completion or medical record entry during remote consultations. Furthermore, the generated synthesized speech samples themselves can also serve as auditory material for speech feedback training in rehabilitation therapy, helping patients master clearer pronunciation skills while imitating their own pathological speech, forming a two-way closed loop between medical data collection and rehabilitation training.

[0065] In one embodiment, a speech synthesis device 40 for speech disorders is provided, which corresponds one-to-one with the speech synthesis method for speech disorders described in the above embodiments. For example... Figure 7 As shown, the speech synthesis device 40 for speech disorders includes: Encoding unit 41 is configured to encode the text sequence into phonemes in response to receiving the text sequence, thereby obtaining a phoneme-encoded sequence; The pause prediction unit 42 is used to input the phoneme encoding sequence into the pause predictor to perform pause prediction and obtain the pause prediction result. The insertion unit 43 is used to obtain the pause embedding corresponding to the pause prediction result based on a preset mapping relationship, and insert the pause embedding into the phoneme coding sequence to obtain the enhanced phoneme coding sequence; Duration prediction unit 44 is used to input the enhanced phoneme coding sequence into the duration predictor to predict the duration and obtain the predicted duration; The fusion unit 45 is used to acquire a reference Mel spectrum, extract a global style vector and a frame-level local style vector from the reference Mel spectrum, and fuse the enhanced phoneme coding sequence with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean. The generation unit 46 is used to input the predicted duration and the conditional mean of the pathological acoustic style into the flow matching network, and generate articulation disorder style speech through conditional flow matching.

[0066] In some preferred embodiments, the step of inputting the phoneme-coded sequence into a pause predictor for pause prediction to obtain a pause prediction result includes: The phoneme encoding sequence is input into the pause predictor, and the pause predictor calculates the probability that each phoneme position in the phoneme encoding sequence belongs to a pause category of different lengths for each phoneme position. The probability of the different length pause categories corresponding to each phoneme position is used as the pause prediction result.

[0067] In some preferred embodiments, the step of inputting the enhanced phoneme encoded sequence into a duration predictor for duration prediction to obtain the predicted duration includes: The enhanced phoneme encoding sequence is input into the duration predictor, wherein the duration predictor is obtained by joint training using logarithmic scale mean square error loss and contrast preference optimization loss; The duration predictor outputs the duration of each phoneme based on the enhanced phoneme coding sequence, and uses the duration of each phoneme as the predicted duration. The contrast preference optimization loss, by comparing the actual duration of articulation disorders with the duration of normal speech, constrains the predicted duration output by the duration predictor to be closer to the distribution of articulation disorders.

[0068] In some preferred embodiments, the step of fusing the enhanced phoneme coding sequence with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean includes: The enhanced phoneme coding sequence is encoded to obtain intermediate features; The frame-level local style vector and the intermediate features are fused by attention to obtain the fused features; The global style vector is concatenated with the fused feature to obtain the conditional mean of the pathological acoustic style.

[0069] In some preferred embodiments, the step of performing attention fusion between the frame-level local style vector and the intermediate features to obtain fused features includes: The frame-level local style vector is used as the query vector, and the intermediate features are used as the key vector and value vector to calculate the attention weights. Based on the attention weights, the intermediate features are weighted and summed to obtain the fused features.

[0070] In some preferred embodiments, the step of inputting the prediction duration and the conditional mean of the pathological acoustic style into a flow matching network, and generating articulation disorder style speech through conditional flow matching, includes: The predicted duration and the mean of the pathological acoustic style conditions are concatenated to obtain a combined condition vector; The combined conditional vector is input into the flow matching network. The flow matching network performs a conditional flow matching generation process based on the combined conditional vector to generate a speech disorder style Mel spectrum from the noise distribution. The conditional flow matching generation process includes: upsampling the prediction duration to align the time dimension of the speech disorder style Mel spectrum with the prediction duration. The articulation disorder style Mel spectrum is input into a vocoder to synthesize the articulation disorder style speech.

[0071] In some preferred embodiments, the step of performing a conditional flow matching generation process based on the combined conditional vector through the flow matching network to generate a mell spectrum of articulation disorder style from the noise distribution includes: The initial noise vector is obtained by sampling from the standard normal distribution through the flow matching network; The flow matching network uses the combined condition vector as a guiding condition and a preset ordinary differential equation solver to iteratively solve the ordinary differential equation. During the iterative solution process, the flow matching network performs upsampling expansion based on the prediction duration, so that the time dimension of the intermediate Mel spectrum generated during the iterative solution process is aligned with the prediction duration. The final Mel spectrum obtained by iterative solution is used as the Mel spectrum output of the articulation disorder style.

[0072] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a speech synthesis method for articulation disorders on the server side.

[0073] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a client-side method for speech synthesis in articulation disorders.

[0074] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a speech synthesis method for articulation disorders provided in any of the above embodiments.

[0075] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of a speech synthesis method for articulation disorders provided in any of the above embodiments.

[0076] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0077] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0079] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0080] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for synthesizing speech with articulation disorders, characterized in that, include: In response to receiving a text sequence, the text sequence is phoneme encoded to obtain a phoneme encoded sequence; The phoneme encoding sequence is input into the pause predictor to perform pause prediction and obtain the pause prediction result; Based on a preset mapping relationship, the pause embedding corresponding to the pause prediction result is obtained, and the pause embedding is inserted into the phoneme coding sequence to obtain the enhanced phoneme coding sequence. The enhanced phoneme coding sequence is input into the duration predictor to predict the duration and obtain the predicted duration. A reference Mel spectrum is obtained, a global style vector and a frame-level local style vector are extracted from the reference Mel spectrum, and the enhanced phoneme coding sequence is fused with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean. The predicted duration and the conditional mean of the pathological acoustic style are input into a flow matching network, and dysarthmic style speech is generated through conditional flow matching.

2. The speech synthesis method for articulation disorders according to claim 1, characterized in that, The step of inputting the phoneme-encoded sequence into a pause predictor for pause prediction to obtain a pause prediction result includes: The phoneme encoding sequence is input into the pause predictor, and the pause predictor calculates the probability that each phoneme position in the phoneme encoding sequence belongs to a pause category of different lengths for each phoneme position. The probability of the different length pause categories corresponding to each phoneme position is used as the pause prediction result.

3. The speech synthesis method for articulation disorders according to claim 1, characterized in that, The step of inputting the enhanced phoneme encoded sequence into a duration predictor for duration prediction to obtain the predicted duration includes: The enhanced phoneme encoding sequence is input into the duration predictor, wherein the duration predictor is obtained by joint training using logarithmic scale mean square error loss and contrast preference optimization loss; The duration predictor outputs the duration of each phoneme based on the enhanced phoneme coding sequence, and uses the duration of each phoneme as the predicted duration. The contrast preference optimization loss, by comparing the actual duration of articulation disorders with the duration of normal speech, constrains the predicted duration output by the duration predictor to be closer to the distribution of articulation disorders.

4. The speech synthesis method for articulation disorders according to claim 1, characterized in that, The step of fusing the enhanced phoneme coding sequence with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean includes: The enhanced phoneme coding sequence is encoded to obtain intermediate features; The frame-level local style vector and the intermediate features are fused by attention to obtain the fused features; The global style vector is concatenated with the fused feature to obtain the conditional mean of the pathological acoustic style.

5. The speech synthesis method for articulation disorders according to claim 4, characterized in that, The step of performing attention fusion between the frame-level local style vector and the intermediate features to obtain fused features includes: The frame-level local style vector is used as the query vector, and the intermediate features are used as the key vector and value vector to calculate the attention weights. Based on the attention weights, the intermediate features are weighted and summed to obtain the fused features.

6. The speech synthesis method for articulation disorders according to claim 1, characterized in that, The step of inputting the predicted duration and the conditional mean of the pathological acoustic style into a flow matching network, and generating articulation disorder style speech through conditional flow matching, includes: The predicted duration and the mean of the pathological acoustic style conditions are concatenated to obtain a combined condition vector; The combined conditional vector is input into the flow matching network. The flow matching network performs a conditional flow matching generation process based on the combined conditional vector to generate a speech disorder style Mel spectrum from the noise distribution. The conditional flow matching generation process includes: upsampling the prediction duration to align the time dimension of the speech disorder style Mel spectrum with the prediction duration. The articulation disorder style Mel spectrum is input into a vocoder to synthesize the articulation disorder style speech.

7. The speech synthesis method for articulation disorders according to claim 6, characterized in that, The step of generating a conditional flow matching generation process based on the combined conditional vector through the flow matching network to generate a mellitus spectrum of articulation disorder style from the noise distribution includes: The initial noise vector is obtained by sampling from the standard normal distribution through the flow matching network; The flow matching network uses the combined condition vector as a guiding condition and a preset ordinary differential equation solver to iteratively solve the ordinary differential equation. During the iterative solution process, the flow matching network performs upsampling expansion based on the prediction duration, so that the time dimension of the intermediate Mel spectrum generated during the iterative solution process is aligned with the prediction duration. The final Mel spectrum obtained by iterative solution is used as the Mel spectrum output of the articulation disorder style.

8. A speech synthesis device for articulation disorders, characterized in that, include: An encoding unit is configured to encode a text sequence into phonemes in response to receiving a text sequence, thereby obtaining a phoneme-encoded sequence. The pause prediction unit is used to input the phoneme encoding sequence into the pause predictor to perform pause prediction and obtain the pause prediction result. An insertion unit is used to obtain a pause embedding corresponding to the pause prediction result based on a preset mapping relationship, and insert the pause embedding into the phoneme coding sequence to obtain an enhanced phoneme coding sequence. The duration prediction unit is used to input the enhanced phoneme coding sequence into the duration predictor to predict the duration and obtain the predicted duration. The fusion unit is used to acquire a reference Mel spectrum, extract a global style vector and a frame-level local style vector from the reference Mel spectrum, and fuse the enhanced phoneme coding sequence with the global style vector and the frame-level local style vector to obtain the pathological acoustic style conditional mean. The generation unit is used to input the predicted duration and the conditional mean of the pathological acoustic style into the flow matching network, and generate articulation disorder style speech through conditional flow matching.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the speech synthesis method for articulation disorders as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the speech synthesis method for articulation disorders as described in any one of claims 1 to 7.

Images