Model training method, speech synthesis method, electronic equipment and storage medium

Abstract

The invention discloses a model training method, a speech synthesis method, electronic equipment and a storage medium, and relates to the technical field of speech synthesis and large models. The method comprises the steps that a disturbance audio signal and a disturbance audio clip are acquired, and the disturbance audio clip is an audio clip intercepted from the disturbance audio signal; encoding the disturbance audio clip to obtain a first encoding result, and performing clip extraction on the disturbance audio signal to obtain an extraction result corresponding to the disturbance audio clip; based on the first coding result and the extraction result, consistency constraint training is carried out on the initial speech synthesis model, a target speech synthesis model is generated, and the target speech synthesis model is used for carrying out speech synthesis processing on the target text prompt and the target audio prompt to generate a target audio. According to the method and the device, the technical problems that the generated voice audio is unstable and the overall quality of a voice synthesis model is relatively low due to the fact that a one-to-many phenomenon exists in an audio word segmentation device in the related technology are solved.

Description

Technical Field

[0001] This application relates to speech synthesis technology and large model technology, specifically to a model training method, a speech synthesis method, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, the field of speech generation is undergoing a revolution. In recent years, large language models have been widely used in speech synthesis systems to improve the accuracy and naturalness of speech synthesis. The core of these speech synthesis systems lies in using audio segmenters to quantize speech signals. Audio segmenters are used to convert continuous audio signals into a series of discrete units as the basic representation of speech, so as to facilitate the training and generation of large speech synthesis models.

[0003] However, existing audio segmenters suffer from a one-to-many phenomenon, meaning that the same audio segment may be encoded into different discrete units under different circumstances. This uncertainty not only increases the learning burden of large speech synthesis models, but also leads to unstable phenomena such as repetition and omission of words in the generated speech audio, which seriously reduces the overall quality of speech synthesis.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a model training method, a speech synthesis method, an electronic device, and a storage medium to at least solve the technical problem in related technologies where audio segmenters exhibit a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0006] According to one aspect of the embodiments of this application, a model training method is provided, comprising: acquiring a perturbed audio signal and a perturbed audio segment, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal; encoding the perturbed audio segment to obtain a first encoding result; and extracting segments from the perturbed audio signal to obtain extraction results corresponding to the perturbed audio segments; and performing consistency constraint training on an initial speech synthesis model based on the first encoding result and the extraction result to generate a target speech synthesis model, wherein the target speech synthesis model is used to perform speech synthesis processing on target text prompts and target audio prompts to generate target audio.

[0007] According to another aspect of the embodiments of this application, a speech synthesis method is also provided, comprising: acquiring a target text prompt and a target audio prompt, wherein the target text prompt is used to provide speech synthesis text content, and the target audio prompt is used to provide acoustic features of a target speaking object; and using a target speech synthesis model to perform speech synthesis processing on the target text prompt and the target audio prompt to generate target audio, wherein the target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on a first encoding result and an extraction result corresponding to a perturbed audio segment, the first encoding result is obtained by encoding the perturbed audio segment, and the extraction result is obtained by segment extraction from the perturbed audio signal, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0008] According to another aspect of the embodiments of this application, a speech synthesis method is also provided, comprising: acquiring video narration text prompts and video narration audio prompts, wherein the video narration text prompts are used to provide video narration speech synthesis text content, and the video narration audio prompts are used to provide the acoustic features of the video narration object; and employing a target speech synthesis model to perform speech synthesis processing on the video narration text prompts and video narration audio prompts to generate narration audio, wherein the target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on a first encoding result and an extraction result corresponding to a perturbed audio segment, the first encoding result is obtained by encoding the perturbed audio segment, and the extraction result is obtained by segment extraction from the perturbed audio signal, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0009] According to another aspect of the embodiments of this application, a speech synthesis method is also provided, comprising: obtaining a speech synthesis request through a first application programming interface, wherein the request data carried in the speech synthesis request includes: a target text prompt and a target audio prompt, wherein the target text prompt is used to provide speech synthesis text content, and the target audio prompt is used to provide acoustic features of a target speaking object; and returning a speech synthesis response through a second application programming interface, wherein the response data carried in the speech synthesis response includes: a target audio, wherein the target audio is generated by processing the target text prompt and the target audio prompt using a target speech synthesis model, the target speech synthesis model is generated by training an initial speech synthesis model with consistency constraints based on a first encoding result and an extraction result corresponding to a perturbed audio segment, the first encoding result is obtained by encoding the perturbed audio segment, and the extraction result is obtained by extracting segments from the perturbed audio signal, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0010] According to another aspect of the embodiments of this application, a speech synthesis method is also provided, which provides a graphical user interface through a terminal device. The speech synthesis method includes: responding to a first control operation performed on the graphical user interface, uploading a target text prompt and a target audio prompt, wherein the target text prompt is used to provide speech synthesis text content, and the target audio prompt is used to provide acoustic features of a target speaker; responding to a second control operation performed on the graphical user interface, confirming that the target text prompt and the target audio prompt are processed for speech synthesis to generate target audio, wherein the target audio is generated by processing the target text prompt and the target audio prompt using a target speech synthesis model, the target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on a first encoding result and an extraction result corresponding to a perturbed audio segment, the first encoding result is obtained by encoding the perturbed audio segment, the extraction result is obtained by extracting segments from the perturbed audio signal, and the perturbed audio segment is an audio segment extracted from the perturbed audio signal; and playing the target audio within the graphical user interface.

[0011] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any of the above-described model training methods or speech synthesis methods during runtime.

[0012] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to execute any of the above-described model training method or speech synthesis method.

[0013] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described model training methods or speech synthesis methods.

[0014] In this embodiment, a perturbation audio signal and a perturbation audio segment extracted from the perturbation audio signal are acquired. The perturbation audio segment is then encoded to obtain a first encoding result. Segment extraction is performed on the perturbation audio signal to obtain the extraction result corresponding to the perturbation audio segment. Finally, based on the first encoding result and the extraction result, a consistency constraint training is performed on the initial speech synthesis model to ensure the consistency of the acoustic representation between the first encoding result and the extraction result. This generates a target speech synthesis model for processing the target text prompts and target audio prompts to generate target audio. This achieves the goal of training and generating a high-quality speech synthesis model, effectively improving the consistency performance of the audio segmenter, enhancing the stability of the speech generation model, and improving the quality of the generated speech audio. It effectively avoids unstable phenomena such as repetition and missing words in the generated speech audio, thus solving the technical problem in related technologies where the audio segmenter exhibits a one-to-many phenomenon, leading to unstable generated speech audio and low overall quality of the speech synthesis model.

[0015] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a schematic diagram of the one-to-many phenomenon in an audio word segmenter;

[0018] Figure 2 This is a schematic diagram illustrating an application scenario of a model training method according to an embodiment of this application;

[0019] Figure 3 This is a flowchart of a model training method according to an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of consistency constraints according to an embodiment of this application;

[0021] Figure 5 This is a flowchart of a speech synthesis model according to an embodiment of this application;

[0022] Figure 6 This is a flowchart of a speech synthesis method according to an embodiment of this application;

[0023] Figure 7 This is a flowchart of a speech synthesis method according to an embodiment of this application;

[0024] Figure 8 This is a flowchart of a speech synthesis method according to an embodiment of this application;

[0025] Figure 9 This is a flowchart of a speech synthesis method according to an embodiment of this application;

[0026] Figure 10 This is a schematic diagram of the structure of a model training device according to an embodiment of this application;

[0027] Figure 11 This is a schematic diagram of another speech synthesis device according to an embodiment of this application;

[0028] Figure 12 This is a schematic diagram of another speech synthesis device according to an embodiment of this application;

[0029] Figure 13 This is a schematic diagram of another speech synthesis device according to an embodiment of this application;

[0030] Figure 14 This is a schematic diagram of another speech synthesis device according to an embodiment of this application;

[0031] Figure 15 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] The technical solution provided in this application is mainly implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models can also be called foundation models. They are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0035] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely used in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, speech synthesis is used as an example to illustrate the process.

[0036] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0037] A neural audio codec, also known as an audio segmenter, is an algorithm used to compress and encode audio signals. It uses neural networks to learn and represent the features of audio signals, then encodes them into digital signals for transmission and storage in computers.

[0038] Vector quantization (VQ) maps a continuous input space to a finite, discrete codebook, thereby achieving an approximate representation of the input signal. In VQ, a multidimensional vector is mapped to a vector in the codebook; this process is called quantization.

[0039] Recurrent Vector Quantization (RVQ) is an algorithm used for time series forecasting and pattern recognition. RVQ combines the features of vector quantization and recurrent neural networks. In machine learning and signal processing, RVQ is used to capture dynamic features and temporal dependencies in data.

[0040] Discrete units: These are units that discretize continuous data into discrete values. In audio coding, discrete units are typically used to represent the codebook number after the audio signal has been encoded and quantized.

[0041] Codebook: In a neural audio encoder, the codebook is used to store the continuous representation of the encoded audio signal.

[0042] A convolutional layer identifier (Conv ID) represents the sequence number or number of a convolutional layer, used to distinguish between different convolutional layers. A convolutional block (Conv Block) represents a collection of convolutional layers, typically including convolutional layers, pooling layers, etc., used to construct the basic structure of a neural network.

[0043] Mean Squared Error (MSE) function: A commonly used statistic used to measure the difference between an estimate and the true value. The MSE function works by calculating the average of the squared differences between the estimate and the true value, and is typically used to evaluate the performance of regression models.

[0044] VALL-E is a deep learning-based speech synthesis technology, an advanced speech synthesis method capable of generating natural, fluent speech that sounds almost indistinguishable from that of a real person. The core of VALL-E lies in using deep neural networks to learn and simulate the complex features of human speech, including pitch, rhythm, and emotion.

[0045] Large-scale speech synthesis (LSS) is an advanced artificial intelligence technology that converts text into speech, enabling machines to output audio that mimics human speech. This technology has wide applications in fields such as intelligent assistants, voice broadcasting, and speech recognition. The typical structure of a large-scale speech synthesis model includes a text analyzer, a text-to-speech mapping, an audio segmenter, an acoustic model, and a vocoder. The text analyzer breaks down text into smaller units, such as words, syllables, or phonemes, and performs linguistic analysis, including part-of-speech tagging and grammatical analysis. The text-to-speech mapping maps text units to corresponding speech units. This typically involves a large database containing various speech samples and their corresponding text representations. By learning these mappings, the large-scale speech synthesis model can predict how text should be pronounced. The audio segmenter is used to segment continuous speech signals into discrete units, which can be phonemes, syllables, or even smaller acoustic units. The purpose of the audio segmenter is to quantize the speech signal so that the model can process and synthesize speech more accurately. Acoustic models are used to generate acoustic features of speech, such as pitch, duration, and intensity. These features describe the physical properties of speech and are key information in the speech synthesis process. A vocoder is a component used to convert these acoustic features into audible audio signals.

[0046] An audio word segmentation module typically includes convolutional layers, activation functions, convolutional blocks, pooling layers, quantization layers, post-processing layers, and an output layer. The convolutional layer is the core of the audio word segmentation module, used to extract features from the audio signal through convolutional operations. A convolutional layer usually contains multiple convolutional kernels, each responsible for extracting specific features from the audio signal. These features can be frequencies, patterns in time series, etc. After the convolutional layers, activation functions are typically used to increase the non-linearity of the large-scale speech synthesis model, enabling it to capture more complex audio features. A convolutional block is a module composed of multiple convolutional layers and activation functions, which can be stacked together to extract deeper levels of audio features. Pooling layers are used to reduce the spatial dimensionality of the feature map while preserving important feature information. This helps reduce the computational complexity of the large-scale speech synthesis model and improve its generalization ability. After extracting the audio features, the large-scale speech synthesis model needs to convert these continuous features into discrete units. The quantization layer is responsible for this conversion process, mapping continuous feature values ​​to a predefined set of discrete units. After quantization, some post-processing operations may be required, such as removing duplicate units and smoothing, to optimize the final discrete unit sequence. Finally, the large speech synthesis model outputs a discrete unit sequence through the output layer. These sequences can be directly used for subsequent audio processing tasks or further converted into text, tags, etc.

[0047] It can be seen that current large-scale speech synthesis models typically use audio segmenters to quantize speech and use the quantized discrete units as speech representations. However, audio segmenters exhibit a one-to-many phenomenon, meaning that they encode the same audio content into different latent spaces and discretize it into different discrete units. This results in the same audio segment being mapped to multiple acoustic representations, that is, the same audio can be represented as different discrete units. Figure 1 This is a diagram illustrating the one-to-many phenomenon in an audio word segmenter, such as... Figure 1 As shown, audio A is taken, and a portion of audio segment B is extracted from audio A. Audio A and audio segment B are input into an audio segmenter, which encodes the entire audio A and the extracted audio segment B.

[0048] In the process of encoding using an audio word segmenter, the input audio is segmented through convolutional layers and convolutional blocks to obtain discrete units. Then, the obtained discrete units are quantized, so it can be seen that the same audio sample is encoded into different units by the audio word segmenter.

[0049] Ideally, the discrete units encoded from the entire audio A and audio segment B would have the same value at corresponding positions because they contain the same content. However, because speech models cannot reliably map the same audio content to a consistent discrete acoustic representation, therefore... Figure 1 The discrete unit values ​​at corresponding positions in the two audio segments are significantly different. The "one-to-many" relationship between the audio segmenter and the discrete units increases the difficulty for the speech model to understand and decode the audio content.

[0050] Therefore, speech synthesis systems based on large speech synthesis models in related technologies have the following drawbacks: the audio segmenter of the speech synthesis system exhibits a one-to-many phenomenon, meaning that the same audio segment may be encoded into different discrete units under different circumstances. This increases the learning burden of the large speech synthesis model and also leads to instability in the generated speech audio, such as repetition and missing words, which in turn results in unstable speech audio and seriously reduces the overall quality of speech synthesis.

[0051] To address the aforementioned deficiencies, no effective solution has been proposed prior to this application.

[0052] According to an embodiment of this application, a model training method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0053] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the model training method and speech synthesis method provided in this application can be applied to, for example... Figure 2 The application scenarios shown are not limited to these. In, for example... Figure 2 In the application scenario shown, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to access the large model, thereby implementing the method provided in this embodiment.

[0054] In this embodiment, the system consisting of a client device and a server can perform the following steps: When executing the model training method, the client device performs steps such as acquiring the perturbation audio signal and perturbation audio segment input by the user, and transmitting the acquired perturbation audio signal and perturbation audio segment to the server; the server performs steps such as acquiring the perturbation audio signal and perturbation audio segment sent by the client, encoding the perturbation audio segment to obtain a first encoding result, and extracting segments from the perturbation audio signal to obtain extraction results corresponding to the perturbation audio segment; finally, based on the first encoding result and extraction result, the initial speech synthesis model is trained with consistency constraints to generate a target speech synthesis model. When executing the speech synthesis method, the client device performs steps such as acquiring the target text prompt and target audio prompt input by the user, and transmitting the acquired target text prompt and target audio prompt to the server; the server performs steps such as using the target speech synthesis model to perform speech synthesis processing on the acquired target text prompt and target audio prompt to generate target audio, and returning the generated target audio to the client device. It should be noted that, provided that the operating resources of the client device can meet the deployment and operation conditions of a large model, this embodiment can be performed on the client device.

[0055] Under the aforementioned operating environment, this application provides the following: Figure 3 The model training method shown. Figure 3 This is a flowchart of a model training method according to an embodiment of this application. Figure 3 As shown, the method may include the following steps:

[0056] Step S31: Obtain the perturbed audio signal and the perturbed audio segment, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal;

[0057] Step S32: Encode the disturbed audio segment to obtain the first encoding result, and extract segments from the disturbed audio signal to obtain the extraction result corresponding to the disturbed audio segment;

[0058] Step S33: Based on the first encoding result and the extraction result, the initial speech synthesis model is trained with consistency constraints to generate the target speech synthesis model. The target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate the target audio.

[0059] In this embodiment, the perturbation audio signal can be understood as a segment of speech audio, and more specifically, as a segment of speech audio with applied perturbation. Exemplarily, during the training of a speech synthesis model, the perturbation audio signal can select audio clips with different speakers, different speech styles, and different speaking speeds to ensure that the model can adapt to various different speech inputs, thereby ensuring the diversity of the trained speech synthesis model. The perturbation audio signal can also select audio clips containing various different speech content to ensure that the model can generate various different speech outputs. Furthermore, the perturbation audio signal can also select high-quality audio clips, avoiding audio clips with noise interference or poor recording quality to improve the training effect of the model; this is not limited here.

[0060] A perturbed audio segment is an audio segment extracted from a perturbed audio signal; it can be understood as a portion of the speech audio extracted from the speech audio. For example, any audio segment within a given time interval can be selected from the perturbed audio signal as the perturbed audio segment; there is no limitation here.

[0061] For example, the perturbed audio signal can be denoted as X1 = [x1,...,x t Extracting a portion of the audio from X1 yields a perturbed audio segment, which can be denoted as X2 = [x ts ,...,x te [This is not restricted.]

[0062] After acquiring the perturbed audio signal and the perturbed audio segment, the perturbed audio segment is encoded to obtain the first encoding result. This can be understood as encoding the audio content of the perturbed audio segment to obtain the first encoding result.

[0063] For example, when encoding a perturbed audio segment, an audio segmenter E can be used to encode the perturbed audio segment, mapping it into a latent space to obtain a latent vector of the perturbed audio segment in the latent space. This latent vector is then discretized into discrete units, resulting in the discrete units of the perturbed audio segment. Here, the downsampling rate f of the audio segmenter...ds You can set it according to your actual needs; there are no restrictions here.

[0064] While encoding the perturbed audio segment, the perturbed audio signal is also segmented to obtain the extraction result corresponding to the perturbed audio segment. This can be understood as extracting the encoded result of the audio segment corresponding to the perturbed audio segment from the perturbed audio signal, thereby obtaining the extraction result.

[0065] It is important to note that the audio segment corresponding to the extracted result and the perturbed audio segment are audio segments located at the same position in the perturbed audio signal. For example, when the initial speech synthesis model encodes the perturbed audio segment and extracts segments from the perturbed audio signal, it first needs to determine the truncation time of the perturbed audio segment, that is, determine the start and end times of the perturbed audio segment extracted from the perturbed audio signal. This truncation time can be preset or randomly determined by the initial speech synthesis model; there is no restriction here. After determining the truncation time, the extraction result corresponding to that truncation time, that is, the extraction result corresponding to the perturbed audio segment, is extracted from the encoding result of encoding the perturbed audio signal according to the truncation time. Simultaneously, the perturbed audio segment is encoded to obtain the first encoding result.

[0066] After obtaining the first encoding result and the extraction result, the initial speech synthesis model is trained with consistency constraints based on the first encoding result and the extraction result to generate the target speech synthesis model. Here, the initial speech synthesis model can be understood as the speech synthesis model before training, and the target speech synthesis model can be understood as the speech synthesis model obtained after training the initial speech synthesis model, that is, the speech synthesis model finally trained in this embodiment of the application.

[0067] Based on the first encoding result and the extraction result, the initial speech synthesis model is trained with consistency constraints. This can be understood as training the initial speech synthesis model with consistency constraints based on the first encoding result corresponding to the audio segment extracted from the perturbed audio signal and the extraction result of the segment extraction from the perturbed audio signal, so that the first encoding result and the extraction result remain close, ensuring the consistency of the acoustic representation of the first encoding result and the extraction result, thereby enabling the training of the target speech synthesis model.

[0068] For example, during consistency constraint training, a loss function can be designed to measure the difference between the first encoded result and the extracted result. For example, the MSE function or other similarity measures can be used to calculate the difference between the first encoded result and the extracted result, which is not limited here.

[0069] In this embodiment, the target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate the target audio. The target text prompt provides the text content of the target audio to be output. Based on the target text prompt, linguistic features can be determined, including phonemes, phoneme sequences, prosodic features, etc., which are not limited here.

[0070] The target audio cue is used to provide the acoustic features of the target audio to be output. For example, the acoustic features provided by the target audio cue are usually acoustic features that imitate human voice, including pitch, timbre, intonation, etc., which are not limited here.

[0071] The target audio can be understood as the audio used to read the text content provided by the target text prompt, based on the acoustic features provided by the target audio prompt.

[0072] In this embodiment, target text prompts and target audio prompts can be input into the target speech synthesis model. The target speech synthesis model can further determine the pronunciation and intonation of the target audio based on the linguistic features determined by the target text prompts. At the same time, the target speech synthesis model can quickly adapt to and imitate the voice of a specific speaker based on the target audio prompts, thereby outputting the target audio. This can effectively ensure the consistency of the target audio in terms of content and listening experience, making the generated target audio more accurate and significantly improving word error rate and speaker timbre similarity.

[0073] As can be seen from the observation of the one-to-many relationship between audio segmenters and discrete units in related technologies, this application finds that the one-to-many relationship is mainly caused by the receptive field of convolution in the audio segmenter. The receptive field of convolution causes the encoded acoustic representation to carry contextual information, resulting in the discrete units changing with the contextual information. Therefore, this application proposes a consistency constraint scheme to constrain the audio segmenter. A segment of audio is randomly sliced ​​from the input audio signal. The consistency constraint requires that the acoustic representation encoded from this audio segment closely approximate the acoustic representation encoded at the corresponding position in the entire input audio signal. This enables the audio segmenter to encode audio with the same content into the same discrete units, thus ensuring that adding contextual information does not change the acoustic representation and discrete units. Simultaneously, this application uses perturbated audio signals for training and requires that the acoustic representation encoded from an audio signal closely align with its acoustic representation after adding a slight perturbation and encoding. This ensures that adding a slight perturbation does not change the acoustic representation and discrete units. By imposing consistency constraints on the audio segmenter, the consistency of audio content and listening experience can be ensured, thereby improving the audio generation effect of the speech synthesis model. This effectively solves the one-to-many problem of the audio segmenter, significantly enhances the stability and naturalness of the speech synthesis system, and avoids unstable phenomena such as repetition and missing words in the speech audio generated by the target speech synthesis model.

[0074] The model training method provided in this application embodiment can be applied, but is not limited to, to application scenarios involving the training of speech synthesis models in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, it can be applied to the training of speech synthesis models related to e-commerce services, education services, and legal services. No limitation is imposed here.

[0075] By employing the embodiments of this application, a perturbation audio signal and a perturbation audio segment extracted from the perturbation audio signal are acquired. The perturbation audio segment is then encoded to obtain a first encoding result. Furthermore, segments are extracted from the perturbation audio signal to obtain extraction results corresponding to the perturbation audio segments. Finally, based on the first encoding result and the extraction result, a consistency constraint training is performed on the initial speech synthesis model to ensure the consistency of the acoustic representation between the first encoding result and the extraction result. This generates a target speech synthesis model for processing target text prompts and target audio prompts to generate target audio. This achieves the goal of training and generating a high-quality speech synthesis model, effectively improving the consistency performance of the audio segmenter, enhancing the stability of the speech generation model, and improving the quality of the generated speech audio. It effectively avoids unstable phenomena such as repetition and missing words in the generated speech audio, thus solving the technical problem in related technologies where the audio segmenter exhibits a one-to-many phenomenon, leading to unstable generated speech audio and low overall quality of the speech synthesis model.

[0076] In an optional embodiment, step S31 involves acquiring the perturbation audio signal, including the following method steps:

[0077] Step S311: Obtain the original audio signal;

[0078] Step S312: Perturb the original audio signal according to the preset perturbation amplitude to obtain the perturbed audio signal, wherein the preset perturbation amplitude is lower than the perturbation threshold.

[0079] In this embodiment of the application, when obtaining the perturbed audio signal, the original audio signal, i.e., the unperturbed speech audio, can be obtained, and then the original audio signal can be perturbed according to a preset perturbing amplitude to obtain the perturbed audio signal.

[0080] It is important to note that the preset perturbation amplitude is lower than the perturbation threshold, which indicates a very small perturbation amplitude. The original audio signal is perturbed according to the preset perturbation amplitude, meaning a negligible, small perturbation is applied to the original audio signal to obtain a perturbed audio signal.

[0081] For example, in this embodiment of the application, in order to ensure perturbation consistency, the perturbed audio signal X1 obtained after applying a negligible small perturbation P to the original audio signal X should be perceptually similar to X. The perturbation P can be applied by various methods, such as slightly adjusting the phase of the audio signal without significantly changing the waveform structure. Considering that the perturbation consistency requirement requires that the changes to the audio be imperceptible to the human ear, and to avoid severe perturbations from damaging the reconstruction effect of the audio segmenter, no new audio samples are created in this embodiment of the application. Since the human ear has limited direct perception of phase changes, phase changes are usually difficult to detect. Therefore, mean square error can be used to constrain the encoded representation before and after the perturbation, as shown in Equations (1) and (2), employing phase perturbation to enhance the perturbation consistency within the audio segmenter.

[0082] X1= P(X) Formula (1)

[0083] Loss perception = MSE(E(X),E(X1)) Formula (2)

[0084] In this context, formula (1) indicates that the perceptual similarity between the perturbed audio signal X1 and the original audio signal X is as follows: E(X) represents the encoding result of the original audio signal X, E(X1) represents the encoding result of the perturbed audio signal X1, and formula (2) indicates that the perception loss is the mean square error between the encoding results of the original audio signal X and the encoding results of the perturbed audio signal X1.

[0085] In an optional embodiment, step S31, acquiring the perturbed audio segment, includes the following method steps:

[0086] Step S313: Select the target time interval within the time range corresponding to the disturbed audio signal;

[0087] Step S314: Perform a segmentation operation on the audio segment located within the target time interval to obtain the perturbed audio segment.

[0088] In this embodiment, when acquiring the perturbed audio segment, a target time interval can be selected within the time range corresponding to the perturbed audio signal. The target time interval can be understood as a period of time; more specifically, the target time interval is a period of time within the time range corresponding to the perturbed audio signal. Then, the audio segment located within the target time interval is truncated to obtain the perturbed audio segment, thereby ensuring that the perturbed audio segment has the same audio content as the perturbed audio signal.

[0089] For example, after acquiring the perturbation audio signal X1, a target time interval I is selected from X1. t =[ts , t e The target time interval I is extracted from the perturbed audio signal X1. t =[t s , t e The audio segment within the target time interval I is used to obtain the perturbed audio segment X2. X2 can be understood as the perturbed audio signal X1 within the target time interval I. t The subsequence within the time domain, this operation is called slicing in the time domain, denoted as S. t .

[0090] In an optional embodiment, in step S32, segment extraction is performed on the perturbed audio signal to obtain the extraction result corresponding to the perturbed audio segment, including the following method steps:

[0091] Step S321: Perform audio encoding on the disturbed audio signal to obtain the second encoding result;

[0092] Step S322: Determine the target feature interval corresponding to the perturbed audio segment in the second encoding result;

[0093] Step S323: Extract the feature subsequence located within the target feature interval from the feature sequence corresponding to the second encoding result to obtain the extraction result.

[0094] In this embodiment of the application, when extracting segments from the perturbed audio signal and obtaining the extraction result, the perturbed audio signal can be audio encoded to obtain a second encoding result. For example, this can be achieved using an audio segmenter at a sampling rate f. ds Factor downsampling is used to obtain the second encoding result.

[0095] Then, the target feature interval corresponding to the perturbed audio segment is determined from the second encoding result. This can be understood as selecting the target feature interval corresponding to the perturbed audio segment within the target time interval of the perturbed audio signal. For example, the target time interval I can be selected. t The target feature interval I corresponding to the audio samples (i.e., the perturbed audio segments) within. f =[t s / f ds , t e / f ds ].

[0096] Finally, feature subsequences located within the target feature interval are extracted from the feature sequence corresponding to the second encoding result to obtain the extraction result corresponding to the perturbed audio segment. For example, the target feature interval I is extracted from the feature sequence corresponding to the second encoding result. f The feature subsequence within is obtained by extracting the encoding result corresponding to the target feature interval from the second encoding result of encoding the perturbed audio signal, thereby obtaining the extraction result corresponding to the perturbed audio segment.

[0097] It is understood that audio coding is a process of converting audio signals into a more compact, easier-to-store, and easier-to-transmit format. In this process, the encoder extracts key features from the audio signal and converts them into an encoded result. In this embodiment, the feature sequence corresponding to the second encoding result refers to a series of values ​​representing the features of the audio signal in the second encoding result. The target feature interval is the position of the feature subsequence corresponding to the encoded perturbed audio segment within the feature sequence corresponding to the second encoding result. The feature subsequence within the target feature interval is also the feature subsequence of the encoded result of the perturbed audio segment within the feature sequence corresponding to the second encoding result.

[0098] It is important to note that the boundary t of the target feature interval corresponding to the selected audio sample (i.e., the perturbed audio segment) is... s and t e It can be the downsampling rate f ds To avoid perturbing the temporal alignment issue between the latent vectors of the audio segments and the extracted results, this slicing operation on the feature sequence is denoted as S. f .

[0099] For example, in this embodiment of the application, to ensure segmentation consistency, the acoustic representations of the same audio content encoded by the audio segmenter should be close to each other so that the resulting discrete unit sequences after quantization remain consistent. Further, given a perturbed audio signal X1 = [x1,...,x...],... t Then randomly select a target time interval I. t =[t s , t e ], and trim the audio samples within this target time interval to generate a new audio segment, i.e., the perturbation audio segment X2 = [x ts ,...,x te Subsequently, the audio segmenter E maps X1 to latent vector Z1 and X2 to latent vector Z2. Clearly, Z1 is within the target feature interval I. f =[t s / f ds , t e / f ds The subsequence Z1' on the [ ] encodes the same audio sample as Z2. To enforce consistency between Z1' and Z2, mean squared error is used as a constraint, as expressed in Equation (3).

[0100] Loss slice = MSE(Z1', Z2) Formula (3)

[0101] Wherein, formula (3) represents the slice loss as the latent vector Z1 of the perturbed audio signal X1 at the target feature interval I. f =[t s / f ds , t e / f ds The mean square error between the subsequence Z1' on the [] and the latent vector Z2 of the perturbed audio segment X2.

[0102] In an optional embodiment, in step S33, based on the first encoding result and the extraction result, the initial speech synthesis model is trained with consistency constraints to generate the target speech synthesis model, including the following method steps:

[0103] Step S331: Calculate the consistency constraint loss based on the first encoding result and the extraction result;

[0104] Step S332: Based on the consistency constraint loss, perform a preset number of consistency constraint training rounds on the initial speech synthesis model to generate the target speech synthesis model.

[0105] This application's embodiments claim that an ideal audio segmenter should simultaneously satisfy slice consistency and perturbation consistency, meaning that adding contextual information and slight perturbations should not change the acoustic representation and discrete units. This application's embodiments ensure that within the target feature interval I... f =[t s / f ds , t e / f ds The perturbation on the signal X is represented by E(X1) being close to Z2 to achieve consistency constraints. That is, for the original audio signal X, slice consistency constraints and perturbation consistency constraints are applied by formula (4).

[0106]

[0107] in, This represents the result of extracting the perturbed audio signal after applying a perturbation to the original audio signal X. The first encoding result is represented by the encoding of the perturbed audio segment extracted from the perturbed audio signal. Formula (4) represents the consistency constraint loss as the mean square error between the extraction result and the first encoding result.

[0108] After calculating the consistency constraint loss, the initial speech synthesis model is trained with consistency constraints for a preset number of rounds based on the consistency constraint loss to generate the target speech synthesis model. The preset number of rounds can be understood as a pre-defined number of training rounds. The initial speech synthesis model is trained with consistency constraints for the preset number of rounds based on the consistency constraint loss, iteratively updating the initial speech synthesis model until the preset number of training rounds is reached, ultimately yielding the target speech synthesis model.

[0109] In an optional embodiment, step S331 involves calculating the consistency constraint loss based on the first encoding result and the extraction result, including the following method steps:

[0110] Step S3311: Using a preset numerical difference degree calculation function, calculate the degree of difference between the first encoding result and the extraction result to obtain the consistency constraint loss.

[0111] In this embodiment of the application, when calculating the consistency constraint loss based on the first encoding result and the extraction result, a preset numerical difference degree calculation function can be used to calculate the degree of difference between the first encoding result and the extraction result to obtain the consistency constraint loss.

[0112] For example, the preset numerical difference calculation function can be the mean squared error (MSE) function, that is, the MSE function is used to calculate the degree of difference between the extraction result and the first encoding result to obtain the consistency constraint loss, which is not limited here.

[0113] In an optional embodiment, in step S332, the initial speech synthesis model is trained with consistency constraints for a preset number of rounds based on the consistency constraint loss to generate the target speech synthesis model, including the following method steps:

[0114] Step S3321: Calculate the target loss based on the consistency constraint loss and the known loss. The known loss is a variety of losses obtained by the initial speech synthesis model during historical training. The known losses include: reconstruction loss, adversarial loss, feature matching loss and residual vector quantization loss.

[0115] Step S3322: Adjust the model parameters of the audio encoder and audio decoder in the initial speech synthesis model for a preset number of rounds based on the target loss to generate the target speech synthesis model.

[0116] In this embodiment of the application, the known loss refers to the multiple types of losses obtained by the initial speech synthesis model during the historical training process. The known losses include: reconstruction loss, adversarial loss, feature matching loss, and residual vector quantization loss (RVQ commit loss).

[0117] When generating a target speech synthesis model by performing a preset number of consistency constraint training rounds on the initial speech synthesis model based on the consistency constraint loss, the target loss can be calculated based on the consistency constraint loss and the known loss. For example, the target loss L is represented by formula (5).

[0118] L = L rec + λ adv L adv + λ fm L fm + λ rvq L rvq + λ con L consistency Formula (5)

[0119] Among them, L rec L represents the reconstruction loss. adv Indicating resistance to loss, L fm L represents the feature matching loss. rvq L represents the residual vector quantization loss. consistency λ represents the consistency constraint loss. adv , λ fm , λ rvq and λ con L respectively adv L fm , λ rvq and L consistency The corresponding coefficient.

[0120] After calculating the target loss, the model parameters of the audio encoder and audio decoder in the initial speech synthesis model are adjusted for a preset number of rounds based on the target loss, thereby generating the target speech synthesis model.

[0121] As can be seen, the embodiments of this application add consistency constraint loss on the basis of four common known losses, and after obtaining the target loss, ensure that the model parameters of other components of the initial speech synthesis model remain unchanged, and only update the parameters of the audio encoder and decoder of the audio segmenter in the initial speech synthesis model, thereby generating the target speech synthesis model. This can effectively improve the consistency performance of the audio segmenter and enhance the audio generation effect of the target speech synthesis model.

[0122] In summary, this application proposes an innovative consistency constraint method to address the one-to-many problem encountered by audio segmenters in speech synthesis. This method primarily addresses two aspects: slice consistency and perturbation consistency, thereby ensuring that the audio segmenter can generate consistent discrete unit representations for the same or similar audio content. Slice consistency ensures that for audio segments with the same content, the audio segmenter outputs the same discrete units, thus contributing to coding consistency at the local information level. Perturbation consistency considers the human ear's tolerance to minor perturbations; even with slight changes in the audio signal, the audio segmenter should maintain stable encoding of similar audio content, ensuring the coherence and naturalness of speech synthesis in complex environments.

[0123] In the experiment, this application effectively improved the consistency performance of the audio segmenter by constructing segments containing the same audio samples for each audio segment and applying the two consistency constraints mentioned above, thereby improving the audio generation effect of the speech synthesis model.

[0124] Figure 4 This is a schematic diagram of consistency constraints according to an embodiment of this application, such as... Figure 4 As shown, given an original audio signal C, a small perturbation is applied to C to obtain a perturbed audio signal D. Audio segments E are extracted from the perturbed signal D, ensuring that segment E and the perturbed signal D contain the same audio content. Then, an audio segmenter encodes segment E and the perturbed signal D, generating their respective acoustic features. During training, consistency constraints are applied to these acoustic features. Further, segment E and the perturbed signal D are processed through convolutional layers and convolutional block encoding, followed by consistency constraints, quantization, and decoding to obtain the output audio. Simultaneously, a discriminator distinguishes the output audio from the original audio signal C. Training stops after a specified number of epochs, resulting in a speech synthesis model with consistency constraints.

[0125] Figure 5 This is a flowchart of a speech synthesis model according to an embodiment of this application, such as... Figure 5As shown, when the speech synthesis model obtained in this application is applied in the field of public speaking, it can input synthesized text as text prompts and recorded audio as audio prompts into the speech synthesis model, and then output personalized speech audio through the speech synthesis model. Further, the text prompts are converted into phonemes and then input into a neural codec language model, while the audio prompts are encoded by an audio encoder and then input into the neural codec language model. The neural codec language model performs speech synthesis processing on the text prompts and audio prompts, and then decodes them through an audio decoder to output personalized speech audio. For example, the speech synthesis model can be based on VALL-E speech synthesis technology, which is not limited here.

[0126] It should be noted that this application focuses on optimizing the encoding and decoding processes performed by the audio segmenter, that is, optimizing the audio encoder and audio decoder.

[0127] It is easy to understand that the beneficial effects of the model training method provided in this application include the following points.

[0128] Beneficial effect (1): It noted the existence of one-to-many phenomenon in audio word segmenters and the potential impact of this one-to-many phenomenon on speech synthesis models.

[0129] Beneficial effect (2): By introducing consistency constraints in the audio segmenter, the one-to-many problem is solved, which significantly enhances the stability and naturalness of the speech synthesis system.

[0130] Beneficial effect (3): After applying consistency constraints, the speech synthesis model showed a significant improvement in word error rate and speaker timbre similarity.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0132] Furthermore, it should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0134] According to embodiments of this application, the following are also provided: Figure 6 The speech synthesis method shown is as follows: Figure 6 This is a flowchart of a speech synthesis method according to an embodiment of this application, such as... Figure 6 As shown, the method includes:

[0135] Step S61: Obtain target text prompts and target audio prompts, wherein the target text prompts are used to provide speech synthesis text content, and the target audio prompts are used to provide the acoustic features of the target speaker;

[0136] Step S62: The target speech synthesis model is used to process the target text prompt and the target audio prompt into speech, and the target audio is generated. The target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

[0137] In this embodiment, the target text prompt is used to provide the text content for speech synthesis. Based on the target text prompt, linguistic features can be determined, including phonemes, phoneme sequences, prosodic features, etc., which are not limited here. The target audio prompt is used to provide the acoustic features of the target speaker. For example, the acoustic features provided by the target audio prompt are typically acoustic features that imitate human voices, including pitch, timbre, intonation, etc., which are not limited here. It is understood that the target audio prompt can help the speech synthesis model retain the unique vocal characteristics of the target speaker when synthesizing speech, improving the naturalness of the synthesized speech and its similarity to the target speaker. The target audio can be understood as the audio of reading the text content provided by the target text prompt using the acoustic features provided by the target audio prompt.

[0138] In this embodiment, target text prompts and target audio prompts can be input into the target speech synthesis model. The target speech synthesis model can further determine the pronunciation and intonation of the target audio to be generated based on the linguistic features determined by the target text prompts. At the same time, the target speech synthesis model can quickly adapt to and imitate the voice of a specific speaker based on the target audio prompts, thereby outputting the target audio. This can effectively ensure the consistency of the target audio in terms of content and listening experience, making the generated target audio more accurate and significantly improving the word error rate and the similarity of the speaker's timbre.

[0139] In this embodiment, the training and generation method of the target speech synthesis model can be found in the foregoing description of the model training method, and will not be repeated here.

[0140] The speech synthesis method provided in this application can be applied, but is not limited to, to speech synthesis application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, speech synthesis for e-commerce services, speech synthesis for education services, and speech synthesis for medical services are not limited here.

[0141] In this embodiment, target text prompts and target audio prompts are obtained. The target text prompts provide the text content for speech synthesis, and the target audio prompts provide the acoustic features of the target speaker. Then, a target speech synthesis model is used to process the target text prompts and target audio prompts to generate target audio. The target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal. This achieves the goal of generating high-quality audio based on the speech synthesis model, thereby effectively improving the consistency performance of the audio segmenter, improving the stability of the speech generation model, improving the quality of the speech audio generated by the speech generation model, and effectively avoiding the technical effects of instability such as repetition and missing words in the generated speech audio. This solves the technical problem in related technologies where the audio segmenter has a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0142] In an optional embodiment, the target speech synthesis model includes: a phoneme converter, an audio encoder, a neural codec language model, and an audio decoder. In step S62, the target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate the target audio, including the following method steps:

[0143] Step S621: Use a phoneme converter to perform phoneme conversion on the target text prompt to obtain the phoneme conversion result;

[0144] Step S622: Use an audio encoder to perform audio encoding processing on the target audio prompt to obtain the audio encoding result;

[0145] Step S623: Use a neural codec language model to perform contextual learning on the phoneme conversion results and audio encoding results to obtain the learning results;

[0146] Step S624: Use an audio decoder to perform audio decoding processing on the learning results to generate the target audio.

[0147] like Figure 5 As shown in the embodiments of this application, the target speech synthesis model includes: a phoneme converter, an audio encoder, a neural codec language model, and an audio decoder. When using the target speech synthesis model to perform speech synthesis processing on the target text prompt and the target audio prompt to generate the target audio, a phoneme converter can be used to perform phoneme conversion processing on the target text prompt to obtain the phoneme conversion result, and simultaneously, an audio encoder can be used to perform audio encoding processing on the target audio prompt to obtain the audio encoding result. Then, the neural codec language model is used to perform context learning on the phoneme conversion result and the audio encoding result to obtain the learning result. Finally, the audio decoder is used to perform audio decoding processing on the learning result to generate the target audio.

[0148] According to embodiments of this application, this application also provides, for example... Figure 7 This illustrates a speech synthesis method. Figure 7 This is a flowchart of a speech synthesis method according to an embodiment of this application, such as... Figure 7 As shown, the method includes:

[0149] Step S71: Obtain video narration text prompts and video narration audio prompts, wherein the video narration text prompts are used to provide the text content of the video narration speech synthesis, and the video narration audio prompts are used to provide the acoustic features of the video narration object;

[0150] Step S72: The target speech synthesis model is used to perform speech synthesis processing on the video narration text prompts and video narration audio prompts to generate narration audio. The target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

[0151] In this embodiment of the application, the speech synthesis method can be applied to the field of video narration, wherein the video narration text prompt is used to provide the video narration speech synthesis text content, and the video narration audio prompt is used to provide the acoustic features of the video narration object.

[0152] In this embodiment, video narration text prompts and video narration audio prompts can be input into a target speech synthesis model. Based on the linguistic features determined by the video narration text prompts, the target speech synthesis model can further determine the pronunciation and intonation of the narration audio to be generated. At the same time, based on the video narration audio prompts, the target speech synthesis model can quickly adapt to and imitate the voice of the video narrator, thereby outputting the narration audio. This effectively ensures the consistency of the narration audio in terms of content and listening experience, making the generated narration audio more accurate and significantly improving the word error rate and the similarity of the video narrator's timbre.

[0153] In this embodiment, the training and generation method of the target speech synthesis model can be found in the foregoing description of the model training method, and will not be repeated here.

[0154] The speech synthesis method provided in this application can be applied, but is not limited to, to speech synthesis application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, speech synthesis for e-commerce services, speech synthesis for education services, and speech synthesis for medical services are not limited here.

[0155] By employing the embodiments of this application, video narration text prompts and video narration audio prompts are obtained. The video narration text prompts provide the text content for speech synthesis, and the video narration audio prompts provide the acoustic features of the target speaker. Then, a target speech synthesis model is used to process the video narration text prompts and video narration audio prompts to generate narration audio. The target speech synthesis model is generated after training an initial speech synthesis model based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal. This achieves the goal of generating high-quality audio based on the speech synthesis model, thereby effectively improving the consistency performance of the audio segmenter, improving the stability of the speech generation model, improving the quality of the speech audio generated by the speech generation model, and effectively avoiding the technical effects of instability such as repetition and missing words in the generated speech audio. This solves the technical problem in related technologies where the audio segmenter has a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0156] According to embodiments of this application, this application also provides, for example... Figure 8 This illustrates a speech synthesis method. Figure 8 This is a flowchart of a speech synthesis method according to an embodiment of this application, such as... Figure 8 As shown, the method includes:

[0157] Step S81: Obtain a speech synthesis request through the first application programming interface. The request data carried in the speech synthesis request includes: a target text prompt and a target audio prompt. The target text prompt is used to provide the speech synthesis text content, and the target audio prompt is used to provide the acoustic features of the target speaker.

[0158] Step S82: Return a speech synthesis response through the second application programming interface. The response data carried in the speech synthesis response includes: target audio, which is generated by the target speech synthesis model after processing the target text prompt and the target audio prompt into speech. The target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment. The extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

[0159] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0160] In this embodiment, the speech synthesis request carries a target text prompt and a target audio prompt. The speech synthesis request is used to request the generation of a corresponding target audio based on the carried target text prompt and target audio prompt. The speech synthesis response is the response content of the speech synthesis request, and the speech synthesis response carries the target audio.

[0161] In this embodiment, the training and generation method of the target speech synthesis model can be found in the foregoing description of the model training method, and will not be repeated here.

[0162] The speech synthesis method provided in this application can be applied, but is not limited to, to speech synthesis application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, speech synthesis for e-commerce services, speech synthesis for education services, and speech synthesis for medical services are not limited here.

[0163] In this embodiment, a speech synthesis request is obtained through a first application programming interface (API). The request data carried in the speech synthesis request includes a target text prompt and a target audio prompt. The target text prompt provides the text content for speech synthesis, and the target audio prompt provides the acoustic features of the target speaker. Then, a speech synthesis response is returned through a second API. The response data carried in the speech synthesis response includes a target audio. The target audio is generated by processing the target text prompt and the target audio prompt using a target speech synthesis model. The target speech synthesis model performs consistency constraint training on the initial speech synthesis model based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The training process generates the first encoding result by encoding the perturbed audio segment, and the extraction result by extracting segments from the perturbed audio signal. The perturbed audio segment is an audio fragment extracted from the perturbed audio signal. This achieves the goal of generating high-quality audio based on the speech synthesis model, thereby effectively improving the consistency performance of the audio segmenter, enhancing the stability of the speech generation model, improving the quality of the speech audio generated by the speech generation model, and effectively avoiding unstable phenomena such as repetition and missing words in the generated speech audio. This solves the technical problem in related technologies where the audio segmenter has a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0164] According to embodiments of this application, this application also provides, for example... Figure 9 This illustrates a speech synthesis method. Figure 9 This is a flowchart of a speech synthesis method according to an embodiment of this application, such as... Figure 9 As shown, the method includes:

[0165] Step S91: In response to the first control operation performed on the graphical user interface, upload the target text prompt and the target audio prompt, wherein the target text prompt is used to provide the speech-synthesized text content and the target audio prompt is used to provide the acoustic features of the target speaking object;

[0166] Step S92: In response to the second control operation performed on the graphical user interface, confirm that the target text prompt and the target audio prompt are processed by speech synthesis to generate target audio. The target audio is generated by processing the target text prompt and the target audio prompt by speech synthesis using a target speech synthesis model. The target speech synthesis model is generated by training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbed audio segment. The first encoding result is obtained by encoding the perturbed audio segment. The extraction result is obtained by extracting segments from the perturbed audio signal. The perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0167] Step S93: Play the target audio within the graphical user interface.

[0168] The graphical user interface in this application embodiment displays at least a speech synthesis scenario, and the user can perform control operations within the speech synthesis scenario displayed in the graphical user interface. It is understood that the aforementioned speech synthesis scenario can be, but is not limited to, scenarios involving speech synthesis in fields such as e-commerce, education, healthcare, conferencing, social networks, financial products, logistics, and navigation.

[0169] The aforementioned graphical user interface also includes a first control (or a first touch area). When a first touch operation is detected acting on the first control (or the first touch area), target text prompts and target audio prompts can be uploaded. It is understood that the target text prompts are used to provide the text content for speech synthesis, and the target audio prompts are used to provide the acoustic characteristics of the target speaker. The aforementioned first touch operation can be a point selection, box selection, checkmark selection, conditional filtering, etc., and is not limited here.

[0170] The graphical user interface described above also includes a second control (or a second touch area). When a second touch operation is detected on the second control (or the second touch area), the target speech synthesis model can be controlled to perform speech synthesis processing on the target text prompt and the target audio prompt to generate target audio. The target audio is generated by the target speech synthesis model after performing speech synthesis processing on the target text prompt and the target audio prompt. The target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbed audio segment. The first encoding result is obtained by encoding the perturbed audio segment, and the extraction result is obtained by extracting segments from the perturbed audio signal. The perturbed audio segment is an audio segment extracted from the perturbed audio signal. The second touch operation can be a point selection, box selection, checkmark selection, conditional filtering, etc., and is not limited here.

[0171] Once the target audio is obtained, it can be displayed within the graphical user interface.

[0172] It should be noted that the first, second, and third touch operations described above can all be operations performed by a user touching the display screen of the terminal device with their finger and interacting with the device. These touch operations can include single-point touch and multi-point touch, where each touch point can be interacted with by clicking, long-pressing, pressing hard, or swiping. The first, second, and third touch operations can also be performed using input devices such as a mouse or keyboard; this is not a limitation.

[0173] In this embodiment, in response to a first control operation performed on the graphical user interface, a target text prompt and a target audio prompt are uploaded. Then, in response to a second control operation performed on the graphical user interface, it is confirmed that the target text prompt and the target audio prompt are processed by speech synthesis to generate target audio. Finally, the target audio is played in the graphical user interface to provide feedback to the user.

[0174] In this embodiment, the training and generation method of the target speech synthesis model can be found in the foregoing description of the model training method, and will not be repeated here.

[0175] The speech synthesis method provided in this application can be applied, but is not limited to, to speech synthesis application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, speech synthesis for e-commerce services, speech synthesis for education services, and speech synthesis for medical services are not limited here.

[0176] In this embodiment, by responding to a first control operation performed on the graphical user interface, a target text prompt and a target audio prompt are uploaded. The target text prompt provides the text content for speech synthesis, and the target audio prompt provides the acoustic features of the target speaker. Then, in response to a second control operation performed on the graphical user interface, speech synthesis processing of the target text prompt and target audio prompt is confirmed to generate target audio. The target audio is generated after the target text prompt and target audio prompt have undergone speech synthesis processing using a target speech synthesis model. The target speech synthesis model is generated after training an initial speech synthesis model based on consistency constraints using the first encoding result and the extraction result corresponding to the perturbed audio segment. The first encoding result is... After encoding the perturbation audio segment, the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal. Finally, the target audio is played in the graphical user interface. This achieves the goal of generating high-quality audio based on the speech synthesis model, thereby effectively improving the consistency performance of the audio segmenter, improving the stability of the speech generation model, improving the quality of the speech audio generated by the speech generation model, and effectively avoiding the technical effects of instability such as repetition and missing words in the generated speech audio. In addition, it solves the technical problem in related technologies where the audio segmenter has a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0177] According to an embodiment of this application, an apparatus embodiment for implementing the above-described model training method is also provided. Figure 10 This is a schematic diagram of a model training device according to an embodiment of this application, as shown below. Figure 10 As shown, the device includes:

[0178] The first acquisition module 1001 is used to acquire the perturbed audio signal and the perturbed audio segment, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal;

[0179] The encoding module 1002 is used to encode the disturbed audio segment to obtain a first encoding result, and to extract segments from the disturbed audio signal to obtain the extraction result corresponding to the disturbed audio segment;

[0180] Training module 1003 is used to perform consistency constraint training on the initial speech synthesis model based on the first encoding result and the extraction result to generate the target speech synthesis model. The target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate the target audio.

[0181] Optionally, the first acquisition module 1001 is further configured to: acquire the original audio signal; and perturb the original audio signal according to a preset perturbation amplitude to obtain a perturbed audio signal, wherein the preset perturbation amplitude is lower than a perturbation threshold.

[0182] Optionally, the first acquisition module 1001 is further configured to: select a target time interval within the time range corresponding to the perturbed audio signal; and perform a segmentation operation on the audio segment located within the target time interval to obtain the perturbed audio segment.

[0183] Optionally, the first acquisition module 1001 is further configured to: perform audio encoding on the disturbed audio signal to obtain a second encoding result; determine the target feature interval corresponding to the disturbed audio segment in the second encoding result; and extract the feature subsequence located within the target feature interval from the feature sequence corresponding to the second encoding result to obtain the extraction result.

[0184] Optionally, the training module 1003 is further configured to: calculate the consistency constraint loss based on the first encoding result and the extraction result; and perform a preset number of consistency constraint training rounds on the initial speech synthesis model based on the consistency constraint loss to generate the target speech synthesis model.

[0185] Optionally, the training module 1003 is further configured to: use a preset numerical difference calculation function to calculate the degree of difference between the first encoding result and the extraction result, and obtain the consistency constraint loss.

[0186] Optionally, the training module 1003 is further configured to: calculate the target loss based on the consistency constraint loss and the known loss, wherein the known loss is a variety of losses obtained by the initial speech synthesis model during the historical training process, including: reconstruction loss, adversarial loss, feature matching loss and residual vector quantization loss; and adjust the model parameters of the audio encoder and audio decoder in the initial speech synthesis model for a preset number of rounds based on the target loss to generate the target speech synthesis model.

[0187] By employing the embodiments of this application, a perturbation audio signal and a perturbation audio segment extracted from the perturbation audio signal are acquired. The perturbation audio segment is then encoded to obtain a first encoding result. Furthermore, segments are extracted from the perturbation audio signal to obtain extraction results corresponding to the perturbation audio segments. Finally, based on the first encoding result and the extraction result, a consistency constraint training is performed on the initial speech synthesis model to ensure the consistency of the acoustic representation between the first encoding result and the extraction result. This generates a target speech synthesis model for processing target text prompts and target audio prompts to generate target audio. This achieves the goal of training and generating a high-quality speech synthesis model, effectively improving the consistency performance of the audio segmenter, enhancing the stability of the speech generation model, and improving the quality of the generated speech audio. It effectively avoids unstable phenomena such as repetition and missing words in the generated speech audio, thus solving the technical problem in related technologies where the audio segmenter exhibits a one-to-many phenomenon, leading to unstable generated speech audio and low overall quality of the speech synthesis model.

[0188] It should be noted that the acquisition module 1001, encoding module 1002, and training module 1003 mentioned above correspond to steps S31 to S33 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0189] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described speech synthesis method is also provided. Figure 11 This is a schematic diagram of another speech synthesis device according to an embodiment of this application, such as... Figure 11 As shown, the device includes:

[0190] The second acquisition module 1101 is used to acquire target text prompts and target audio prompts, wherein the target text prompts are used to provide speech-synthesized text content, and the target audio prompts are used to provide the acoustic features of the target speaker;

[0191] The first processing module 1102 is used to perform speech synthesis processing on the target text prompt and the target audio prompt using the target speech synthesis model to generate target audio. The target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

[0192] Optionally, the target speech synthesis model includes: a phoneme converter, an audio encoder, a neural codec language model, and an audio decoder. The first processing module 1102 is further configured to: perform phoneme conversion processing on the target text prompt using the phoneme converter to obtain the phoneme conversion result; perform audio encoding processing on the target audio prompt using the audio encoder to obtain the audio encoding result; perform context learning on the phoneme conversion result and the audio encoding result using the neural codec language model to obtain the learning result; and perform audio decoding processing on the learning result using the audio decoder to generate the target audio.

[0193] In this embodiment, target text prompts and target audio prompts are obtained. The target text prompts provide the text content for speech synthesis, and the target audio prompts provide the acoustic features of the target speaker. Then, a target speech synthesis model is used to process the target text prompts and target audio prompts to generate target audio. The target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal. This achieves the goal of generating high-quality audio based on the speech synthesis model, thereby effectively improving the consistency performance of the audio segmenter, improving the stability of the speech generation model, improving the quality of the speech audio generated by the speech generation model, and effectively avoiding the technical effects of instability such as repetition and missing words in the generated speech audio. This solves the technical problem in related technologies where the audio segmenter has a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0194] It should be noted that the second acquisition module 1201 and the first processing module 1202 mentioned above correspond to steps S61 and S62 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0195] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described speech synthesis method is also provided. Figure 12 This is a schematic diagram of another speech synthesis device according to an embodiment of this application, such as... Figure 12 As shown, the device includes:

[0196] The third acquisition module 1201 is used to acquire video narration text prompts and video narration audio prompts, wherein the video narration text prompts are used to provide the video narration speech synthesis text content, and the video narration audio prompts are used to provide the acoustic features of the video narration object;

[0197] The second processing module 1202 is used to perform speech synthesis processing on the video narration text prompts and video narration audio prompts using a target speech synthesis model to generate narration audio. The target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

[0198] By employing the embodiments of this application, video narration text prompts and video narration audio prompts are obtained. The video narration text prompts provide the text content for speech synthesis, and the video narration audio prompts provide the acoustic features of the target speaker. Then, a target speech synthesis model is used to process the video narration text prompts and video narration audio prompts to generate narration audio. The target speech synthesis model is generated after training an initial speech synthesis model based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal. This achieves the goal of generating high-quality audio based on the speech synthesis model, thereby effectively improving the consistency performance of the audio segmenter, improving the stability of the speech generation model, improving the quality of the speech audio generated by the speech generation model, and effectively avoiding the technical effects of instability such as repetition and missing words in the generated speech audio. This solves the technical problem in related technologies where the audio segmenter has a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0199] It should be noted that the third acquisition module 1201 and the second processing module 1202 mentioned above correspond to steps S71 and S72 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0200] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described speech synthesis method is also provided. Figure 13This is a schematic diagram of another speech synthesis device according to an embodiment of this application, such as... Figure 13 As shown, the device includes:

[0201] The fourth acquisition module 1301 is used to acquire a speech synthesis request through the first application programming interface. The request data carried in the speech synthesis request includes: a target text prompt and a target audio prompt. The target text prompt is used to provide the speech synthesis text content, and the target audio prompt is used to provide the acoustic features of the target speaker.

[0202] The return module 1302 is used to return a speech synthesis response through a second application programming interface. The response data carried in the speech synthesis response includes: target audio, which is generated by the target speech synthesis model after processing the target text prompt and the target audio prompt into speech. The target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

[0203] In this embodiment, a speech synthesis request is obtained through a first application programming interface (API). The request data carried in the speech synthesis request includes a target text prompt and a target audio prompt. The target text prompt provides the text content for speech synthesis, and the target audio prompt provides the acoustic features of the target speaker. Then, a speech synthesis response is returned through a second API. The response data carried in the speech synthesis response includes a target audio. The target audio is generated by processing the target text prompt and the target audio prompt using a target speech synthesis model. The target speech synthesis model performs consistency constraint training on the initial speech synthesis model based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The training process generates the first encoding result by encoding the perturbed audio segment, and the extraction result by extracting segments from the perturbed audio signal. The perturbed audio segment is an audio fragment extracted from the perturbed audio signal. This achieves the goal of generating high-quality audio based on the speech synthesis model, thereby effectively improving the consistency performance of the audio segmenter, enhancing the stability of the speech generation model, improving the quality of the speech audio generated by the speech generation model, and effectively avoiding unstable phenomena such as repetition and missing words in the generated speech audio. This solves the technical problem in related technologies where the audio segmenter has a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0204] It should be noted that the fourth acquisition module 1301 and return module 1302 mentioned above correspond to steps S81 and S82 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0205] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described speech synthesis method is also provided. Figure 14 This is a schematic diagram of another speech synthesis device according to an embodiment of this application, such as... Figure 14 As shown, the device includes:

[0206] Upload module 1401 is used to respond to a first control operation performed on the graphical user interface and upload target text prompts and target audio prompts, wherein the target text prompts are used to provide speech-synthesized text content and the target audio prompts are used to provide the acoustic features of the target speaking object;

[0207] The confirmation module 1402 is used to respond to a second control operation performed on the graphical user interface, confirming that the target text prompt and the target audio prompt are processed by speech synthesis to generate target audio. The target audio is generated by processing the target text prompt and the target audio prompt by speech synthesis using a target speech synthesis model. The target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbed audio segment. The first encoding result is obtained by encoding the perturbed audio segment, and the extraction result is obtained by extracting segments from the perturbed audio signal. The perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0208] The playback module 1402 is used to play target audio within a graphical user interface.

[0209] In this embodiment of the application, by responding to a first control operation performed on the graphical user interface, a target text prompt and a target audio prompt are uploaded. The target text prompt provides the text content for speech synthesis, and the target audio prompt provides the acoustic features of the target speaker. Then, in response to a second control operation performed on the graphical user interface, speech synthesis processing of the target text prompt and the target audio prompt is confirmed to generate target audio. The target audio is generated after the target text prompt and the target audio prompt have undergone speech synthesis processing using a target speech synthesis model. The target speech synthesis model is generated after training an initial speech synthesis model based on consistency constraints using the first encoding result and the extraction result corresponding to the perturbed audio segment. The first encoding result is obtained through… After encoding the perturbation audio segment, the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal. Finally, the target audio is played in the graphical user interface. This achieves the goal of generating high-quality audio based on the speech synthesis model, thereby effectively improving the consistency performance of the audio segmenter, improving the stability of the speech generation model, improving the quality of the speech audio generated by the speech generation model, and effectively avoiding the technical effects of instability such as repetition and missing words in the generated speech audio. In addition, it solves the technical problem in related technologies where the audio segmenter has a one-to-many phenomenon, resulting in unstable generated speech audio and low overall quality of the speech synthesis model.

[0210] It should be noted that the upload module 1401, confirmation module 1402, and playback module 1402 mentioned above correspond to steps S91 to S93 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0211] It should be noted that the preferred embodiments involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.

[0212] Embodiments of this application may provide an electronic device, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a mobile terminal or other terminal device.

[0213] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0214] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the model training method: acquiring a perturbed audio signal and a perturbed audio segment, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal; encoding the perturbed audio segment to obtain a first encoding result, and extracting segments from the perturbed audio signal to obtain an extraction result; based on the first encoding result and the extraction result, performing consistency constraint training on the initial speech synthesis model to generate a target speech synthesis model, wherein the target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate the target audio.

[0215] In this embodiment, the aforementioned electronic device can also execute program code for the following steps in the speech synthesis method: obtaining target text prompts and target audio prompts, wherein the target text prompts are used to provide speech synthesis text content, and the target audio prompts are used to provide acoustic features of the target speaking object; using a target speech synthesis model to perform speech synthesis processing on the target text prompts and target audio prompts to generate target audio, wherein the target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on a first encoding result and an extraction result, the first encoding result is obtained by encoding a perturbed audio segment, and the extraction result is obtained by extracting segments from the perturbed audio signal, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0216] Optionally, Figure 15 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 15 As shown, taking electronic device A as an example, electronic device A may include: one or more (only one is shown in the figure) processors 1502, memory 1504, memory controller, and peripheral interfaces, wherein the peripheral interfaces are connected to radio frequency modules, audio modules and displays.

[0217] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model training method or speech synthesis method and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the stored software programs and modules, thereby realizing the aforementioned model training method or speech synthesis method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to electronic device A via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0218] The processor can invoke information and applications stored in the memory via a transmission device to perform the following steps: acquiring a perturbed audio signal and a perturbed audio segment, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal; encoding the perturbed audio segment to obtain a first encoding result, and extracting segments from the perturbed audio signal to obtain extraction results corresponding to the perturbed audio segments; and performing consistency constraint training on an initial speech synthesis model based on the first encoding result and the extraction result to generate a target speech synthesis model, wherein the target speech synthesis model is used to perform speech synthesis processing on target text prompts and target audio prompts to generate target audio.

[0219] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: acquiring target text prompts and target audio prompts, wherein the target text prompts are used to provide speech synthesis text content, and the target audio prompts are used to provide the acoustic features of the target speaker; using a target speech synthesis model to perform speech synthesis processing on the target text prompts and target audio prompts to generate target audio, wherein the target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbed audio segment, the first encoding result is obtained by encoding the perturbed audio segment, and the extraction result is obtained by segment extraction from the perturbed audio signal, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0220] Those skilled in the art will understand that Figure 15 The structure shown is for illustrative purposes only. Electronic device A can also be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, handheld computer, mobile internet device (MID), PAD and other terminal devices. Figure 15 This does not limit the structure of the aforementioned electronic device. For example, electronic device A may also include components that are more... Figure 15 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 15 The different configurations shown.

[0221] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0222] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the aforementioned computer-readable storage medium can be used to store the program code executed by the model training method or speech synthesis method provided in Embodiment 1 above.

[0223] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0224] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring a perturbed audio signal and a perturbed audio segment, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal; encoding the perturbed audio segment to obtain a first encoding result, and extracting segments from the perturbed audio signal to obtain extraction results corresponding to the perturbed audio segment; and performing consistency constraint training on an initial speech synthesis model based on the first encoding result and the extraction result to generate a target speech synthesis model, wherein the target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate target audio.

[0225] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: obtaining target text prompts and target audio prompts, wherein the target text prompts are used to provide speech-synthesized text content, and the target audio prompts are used to provide acoustic features of the target speaking object; using a target speech synthesis model to perform speech synthesis processing on the target text prompts and target audio prompts to generate target audio, wherein the target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbed audio segment, the first encoding result is obtained by encoding the perturbed audio segment, and the extraction result is obtained by segment extraction from the perturbed audio signal, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0226] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described model training methods or speech synthesis methods.

[0227] Optionally, in this embodiment, when the computer program product is executed by the processor, the program code is used to perform the following steps: acquiring a perturbed audio signal and a perturbed audio segment, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal; encoding the perturbed audio segment to obtain a first encoding result, and extracting segments from the perturbed audio signal to obtain extraction results corresponding to the perturbed audio segment; based on the first encoding result and the extraction result, performing consistency constraint training on the initial speech synthesis model to generate a target speech synthesis model, wherein the target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate target audio.

[0228] Optionally, in this embodiment, when the computer program product is executed by the processor, the program code is used to perform the following steps: obtaining target text prompts and target audio prompts, wherein the target text prompts are used to provide speech synthesis text content, and the target audio prompts are used to provide acoustic features of the target speaking object; using a target speech synthesis model to perform speech synthesis processing on the target text prompts and target audio prompts to generate target audio, wherein the target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbed audio segment, the first encoding result is obtained by encoding the perturbed audio segment, and the extraction result is obtained by segment extraction from the perturbed audio signal, wherein the perturbed audio segment is an audio segment extracted from the perturbed audio signal.

[0229] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0230] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0231] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0232] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0233] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0234] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0235] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: Acquire a disturbed audio signal and a disturbed audio segment, wherein the disturbed audio segment is an audio segment extracted from the disturbed audio signal; The disturbed audio segment is encoded to obtain a first encoding result, and the disturbed audio signal is segment extracted to obtain the extraction result corresponding to the disturbed audio segment; Based on the first encoding result and the extraction result, the initial speech synthesis model is trained with consistency constraints to generate a target speech synthesis model, wherein the target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate the target audio.

2. The model training method according to claim 1, characterized in that, Acquiring the disturbed audio signal includes: Acquire the raw audio signal; The original audio signal is perturbed according to a preset perturbation amplitude to obtain the perturbed audio signal, wherein the preset perturbation amplitude is lower than the perturbation threshold.

3. The model training method according to claim 1, characterized in that, Obtaining the disturbed audio segment includes: Select a target time interval within the time range corresponding to the disturbed audio signal; The audio segment located within the target time interval is truncated to obtain the perturbed audio segment.

4. The model training method according to claim 3, characterized in that, The extraction of segments from the disturbed audio signal, and the extraction results corresponding to the disturbed audio segments, include: The disturbed audio signal is encoded to obtain a second encoding result; The target feature interval corresponding to the perturbed audio segment is determined from the second encoding result; The feature subsequence located within the target feature interval is extracted from the feature sequence corresponding to the second encoding result to obtain the extraction result.

5. The model training method according to claim 1, characterized in that, Based on the first encoding result and the extraction result, the initial speech synthesis model is trained with consistency constraints to generate the target speech synthesis model, including: Calculate the consistency constraint loss based on the first encoding result and the extraction result; Based on the consistency constraint loss, the initial speech synthesis model is trained with consistency constraints for a preset number of rounds to generate the target speech synthesis model.

6. The model training method according to claim 5, characterized in that, The consistency constraint loss is calculated based on the first encoding result and the extraction result, including: The degree of difference between the first encoding result and the extraction result is calculated using a preset numerical difference calculation function to obtain the consistency constraint loss.

7. The model training method according to claim 5, characterized in that, Based on the consistency constraint loss, the initial speech synthesis model is trained with the consistency constraint for the preset number of rounds to generate the target speech synthesis model, including: The target loss is calculated based on the consistency constraint loss and the known loss, wherein the known loss is a multi-class loss obtained by the initial speech synthesis model during the historical training process, and the known loss includes: reconstruction loss, adversarial loss, feature matching loss and residual vector quantization loss; Based on the target loss, the model parameters of the audio encoder and audio decoder in the initial speech synthesis model are adjusted for the preset number of rounds to generate the target speech synthesis model.

8. A speech synthesis method, characterized in that, include: Obtain target text prompts and target audio prompts, wherein the target text prompts are used to provide speech-synthesized text content, and the target audio prompts are used to provide the acoustic features of the target speaker; A target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate target audio. The target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

9. The speech synthesis method according to claim 8, characterized in that, The target speech synthesis model includes: a phoneme converter, an audio encoder, a neural codec language model, and an audio decoder. The target speech synthesis model is used to perform speech synthesis processing on the target text prompt and the target audio prompt to generate the target audio, including: The target text prompt is processed by the phoneme converter to obtain the phoneme conversion result. The target audio prompt is encoded using the audio encoder described above to obtain the audio encoding result. The neural codec language model is used to perform context learning on the phoneme conversion result and the audio encoding result to obtain the learning result; The learning results are processed using the audio decoder to generate the target audio.

10. A speech synthesis method, characterized in that, include: The method acquires video narration text prompts and video narration audio prompts, wherein the video narration text prompts are used to provide the synthesized text content of the video narration speech, and the video narration audio prompts are used to provide the acoustic features of the video narration object; A target speech synthesis model is used to perform speech synthesis processing on the video narration text prompts and the video narration audio prompts to generate narration audio. The target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment, and the extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

11. A speech synthesis method, characterized in that, include: A speech synthesis request is obtained through a first application programming interface, wherein the request data carried in the speech synthesis request includes: a target text prompt and a target audio prompt, wherein the target text prompt is used to provide the speech synthesis text content, and the target audio prompt is used to provide the acoustic features of the target speaker; The speech synthesis response is returned through the second application programming interface. The response data carried in the speech synthesis response includes: target audio, which is generated by the target speech synthesis model after processing the target text prompt and the target audio prompt into speech. The target speech synthesis model is generated after training the initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbation audio segment. The first encoding result is obtained by encoding the perturbation audio segment. The extraction result is obtained by extracting segments from the perturbation audio signal. The perturbation audio segment is an audio segment extracted from the perturbation audio signal.

12. A speech synthesis method, characterized in that, The speech synthesis method includes providing a graphical user interface via a terminal device: In response to a first control operation performed on the graphical user interface, a target text prompt and a target audio prompt are uploaded, wherein the target text prompt is used to provide speech-synthesized text content, and the target audio prompt is used to provide the acoustic features of the target speaking object; In response to a second control operation performed on the graphical user interface, it is confirmed that speech synthesis processing is performed on the target text prompt and the target audio prompt to generate target audio. The target audio is generated by performing speech synthesis processing on the target text prompt and the target audio prompt using a target speech synthesis model. The target speech synthesis model is generated after training an initial speech synthesis model with consistency constraints based on the first encoding result and the extraction result corresponding to the perturbed audio segment. The first encoding result is obtained by encoding the perturbed audio segment. The extraction result is obtained by extracting segments from the perturbed audio signal. The perturbed audio segment is an audio segment extracted from the perturbed audio signal. Play the target audio within the graphical user interface.

13. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, executes the model training method according to any one of claims 1 to 7 or the speech synthesis method according to any one of claims 8 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the model training method according to any one of claims 1 to 7 or the speech synthesis method according to any one of claims 8 to 12.

15. A computer program product, characterized in that, It includes a computer program that is executed by a processor using the model training method of any one of claims 1 to 7 or the speech synthesis method of any one of claims 8 to 12.

Images