Audio synthesis method and device, processor and readable storage medium

Abstract

The invention discloses an audio synthesis method and device, a processor and a readable storage medium. The method comprises the steps that basic data of an audio to be synthesized are acquired, and the basic data are used for indicating conditions which need to be met when the audio to be synthesized is synthesized; inputting the basic data into a target acoustic model for analysis to obtain spectrum data, the spectrum data being used for indicating audio features of the to-be-synthesized audio, the target acoustic model being obtained by training a basic data sample and a spectrum data sample, the basic data sample being used for indicating conditions required to be satisfied for synthesizing historical to-be-synthesized audio, and the spectrum data sample being used for indicating conditions required to be satisfied for synthesizing the historical to-be-synthesized audio; the spectrum data samples are used for indicating audio features of historical to-be-synthesized audios; and synthesizing the basic data into the target audio based on the spectrum data. According to the invention, the technical problem of low quality of the synthesized audio is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio synthesis, and in particular to an audio synthesis method, device, processor and readable storage medium. Background Art

[0002] At present, audio synthesis is an important technology that can merge multiple audio clips into a complete audio file. Through audio synthesis, various types of music, sound effects, and speech can be created. Audio synthesis technology is also widely used in film, television, radio and other fields, providing richer and more diverse sound expressions for various media content. With the development of artificial intelligence and machine learning technology, audio synthesis technology is also constantly innovating and improving. In the future, audio synthesis technology is expected to play a more important role in fields such as virtual reality, augmented reality, and speech recognition.

[0003] Under normal circumstances, by sharing text content encoding and distinguishing between speaker embedding and language embedding, any speaker in the set can speak any language in the set. However, when synthesizing across languages, if the input is a speaker and a language representation that has been seen in the set (rather than a target language representation), the target language will not be spoken authentically enough; but if the input is a speaker and a target language representation that has not been seen in the set, the speaker similarity during synthesis is not good enough because the modeling lacks generalization. Therefore, there is a technical problem of low quality of synthesized audio.

[0004] Currently, no effective solution has been proposed to the above-mentioned technical problem of low quality of synthesized audio. Summary of the invention

[0005] Embodiments of the present invention provide an audio synthesis method, device, processor, and readable storage medium to at least solve the technical problem of low quality of synthesized audio.

[0006] According to one aspect of an embodiment of the present invention, a method for synthesizing audio is provided. The method may include: obtaining basic data of audio to be synthesized, wherein the basic data is used to indicate conditions that need to be met in synthesizing the audio to be synthesized; inputting the basic data into a target acoustic model for analysis to obtain spectrum data, wherein the spectrum data is used to indicate audio features of the audio to be synthesized, the target acoustic model is trained using basic data samples and spectrum data samples, the basic data samples are used to indicate conditions that need to be met in synthesizing historical audio to be synthesized, and the spectrum data samples are used to indicate audio features of historical audio to be synthesized; based on the spectrum data, synthesizing the basic data into the target audio.

[0007] Optionally, before the basic data is input into the target acoustic model for analysis to obtain the spectral data, the audio synthesis method also includes: extracting fundamental frequency features from the basic data, wherein the fundamental frequency features are used to indicate audio information from the pronunciation object in the basic data; and removing the fundamental frequency features from the fundamental frequency features.

[0008] Optionally, the audio synthesis method further includes: using basic data samples and spectrum data samples to train an initial acoustic model to obtain a target acoustic model.

[0009] Optionally, the initial acoustic model is trained using the basic data samples and the spectral data samples to obtain a target acoustic model, including: determining language feature samples and style feature samples in the basic data samples, wherein the language feature samples are used to indicate semantic samples and pronunciation feature samples of different languages ​​in the basic data samples, and the style feature samples are used to indicate language styles of different languages ​​in the basic data samples; and the initial acoustic model is trained using the language feature samples and the style feature samples to obtain a target acoustic model.

[0010] Optionally, the language feature samples and the style feature samples are used to train the initial acoustic model to obtain the target acoustic model, including: integrating the language feature samples and the style feature samples to obtain integrated feature samples; and training the initial acoustic model using the integrated feature samples to obtain the target acoustic model.

[0011] Optionally, the audio synthesis method also includes: obtaining a joint loss model, wherein the joint loss model is used to indicate a target that needs to be optimized for the initial acoustic model during training; and training the initial acoustic model using the joint loss model and basic data samples.

[0012] According to another aspect of an embodiment of the present invention, there is also provided an audio synthesis device. The device may include: an acquisition unit, used to acquire basic data of the audio to be synthesized, wherein the basic data is used to indicate the conditions that need to be met for synthesizing the audio to be synthesized; an analysis unit, used to input the basic data into a target acoustic model for analysis to obtain spectrum data, wherein the target acoustic model is trained using basic data samples and spectrum data samples, the basic data samples are used to indicate the conditions that need to be met for synthesizing historical audio to be synthesized, and the spectrum data samples are used to indicate the audio features of the historical audio to be synthesized; a synthesis unit, used to synthesize the basic data into the target audio based on the spectrum data.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein when the program is executed by a processor, the device where the storage medium is located is controlled to execute the audio synthesis method in the embodiment of the present invention.

[0014] According to another aspect of the embodiment of the present invention, a processor is further provided. The processor is used to run a program, wherein the audio synthesis method in the embodiment of the present invention is executed when the program is run.

[0015] According to another aspect of the embodiments of the present invention, a computer program product is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the audio synthesis method in the embodiments of the present invention is implemented.

[0016] In an embodiment of the present invention, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into a target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio features of the audio to be synthesized, and the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. That is, the embodiment of the present invention obtains spectral data by analyzing the basic data of the audio to be synthesized using a target acoustic model, thereby synthesizing audio according to the spectral data. Since the target acoustic model of the embodiment of the present invention can decouple a variety of timbres and styles, the technical problem of low quality of the synthesized audio is solved, and the technical effect of improving the quality of the synthesized audio is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of an audio synthesis method according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a cross-language synthesis method based on language style fusion features according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of an acoustic model structure according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a language style composite feature predictor in a training and reasoning process according to an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of an audio synthesis device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, functional component or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, functional components or devices.

[0025] According to an embodiment of the present invention, an embodiment of a method for synthesizing audio is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] Figure 1 is a flow chart of an audio synthesis method according to an embodiment of the present invention, such as Figure 1 As shown, the method may include the following steps:

[0027] Step S101, obtaining basic data of the audio to be synthesized.

[0028] In the technical solution provided in the above step S101 of the present invention, the basic data is used to indicate the conditions that need to be met for synthesizing the audio to be synthesized. The basic data can also be called audio data.

[0029] In this embodiment, basic data of the audio to be synthesized is obtained, for example, by an input module according to a user's input or selection, and the basic data of the audio to be synthesized is obtained. This is only an illustrative example and does not limit the specific method of obtaining the basic data of the audio to be synthesized.

[0030] Optionally, the basic data at least includes, but is not limited to: the text to be synthesized, identification information of the target speaker, style, and target language.

[0031] Step S102: input the basic data into the target acoustic model for analysis to obtain spectrum data.

[0032] In the technical solution provided in the above step S102 of the present invention, the spectral data is used to indicate the audio features of the audio to be synthesized, the target acoustic model is trained using the basic data samples and the spectral data samples, the basic data samples are used to indicate the conditions that need to be met to synthesize the historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of the historical audio to be synthesized.

[0033] In this embodiment, after the basic data of the audio to be synthesized is obtained in step S101, the basic data is input into the target acoustic model for analysis to obtain spectrum data, where the target acoustic model may also be referred to as an acoustic model, and the spectrum data may also be a Mel spectrum.

[0034] Optionally, by inputting basic data for analysis, detailed spectrum data can be obtained, including spectrum graphs, spectrum distribution and other information, to help users gain a deeper understanding of the characteristics and structure of sound signals.

[0035] Step S103: synthesize the basic data into target audio based on the spectrum data.

[0036] In the technical solution provided in the above step S103 of the present invention, after the spectrum data is obtained in step S102, the basic data is synthesized into the target audio according to the spectrum data.

[0037] In this embodiment, the basic data is synthesized into the target audio according to the spectrum data. For example, the basic data is synthesized into the target audio by a vocoder, which is only an illustrative example and does not limit the specific method of synthesizing the target audio.

[0038] Optionally, the target audio can be accurately reconstructed based on the spectral data by synthesizing the base data, thereby improving the quality and clarity of the audio.

[0039] It should be noted that the above embodiments may be executed by an audio synthesis device.

[0040] In the above steps S101 to S103 of the present invention, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into the target acoustic model for analysis to obtain spectrum data, wherein the spectrum data is used to indicate the audio features of the audio to be synthesized, the target acoustic model is trained using basic data samples and spectrum data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectrum data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectrum data, the basic data is synthesized into the target audio. That is, the embodiment of the present invention obtains spectrum data by analyzing the basic data of the audio to be synthesized using the target acoustic model, thereby synthesizing audio according to the spectrum data. Since the target acoustic model of the embodiment of the present invention can decouple multiple timbres and styles, the technical problem of low quality of the synthesized audio is solved, and the technical effect of improving the quality of the synthesized audio is achieved.

[0041] The above method of this embodiment is further introduced below.

[0042] As an optional implementation method, before the basic data is input into the target acoustic model for analysis to obtain the spectrum data, the audio synthesis method also includes: extracting fundamental frequency features from the basic data, wherein the fundamental frequency features are used to indicate audio information from the pronunciation object in the basic data; and removing the fundamental frequency features from the basic data.

[0043] In this embodiment, fundamental frequency features are extracted from basic data, wherein the fundamental frequency features may include: phoneme identification sequence features, speaker identification features, phoneme-level language identification sequence features, and style identification features.

[0044] Optionally, after extracting the fundamental frequency feature, the fundamental frequency feature in the basic data is determined and removed. For example, the fundamental frequency feature is normalized. Because different speakers may have different fundamental frequency data due to physiological structure, pronunciation habits, etc. Normalization can smooth these differences, thereby more accurately determining the fundamental frequency feature in the basic data, and then removing the normalized fundamental frequency feature in the basic data.

[0045] For example, frame-level fundamental frequency features are extracted from the basic data and normalized according to the mean variance of each speaker, and the speaker information is removed from the fundamental frequency features.

[0046] As an optional implementation manner, the audio synthesis method includes: using basic data samples and spectrum data samples to train an initial acoustic model to obtain a target acoustic model.

[0047] In this embodiment, the basic data samples and the spectrum data samples are input into the initial acoustic model for training to obtain the target acoustic model, wherein the basic data samples are sample data in a data set composed of multiple basic data.

[0048] For example, audio data is obtained to create a multi-language (taking Chinese and English as examples), multi-timbre, and multi-style dataset. In addition to the phoneme identifier sequence and speaker identifier, the phoneme-level language identifier sequence and style identifier are input into the initial acoustic model. Language embedding and style embedding are constructed, and a frame-level discrete speech representation predictor module is constructed, whose structure is similar to the duration predictor and fundamental frequency predictor. During training, its learning target is the wav2vec2.0 frame-level discrete codebook of the current audio. It is generally believed that the discrete representation of Wav2Vec2.0 mainly contains content and rhythm information, and contains less speaker timbre information than other self-supervised audio representations. After predicting the discrete codebook, an additional discrete codebook embedding layer is used to obtain the final audio discrete representation. The shape used here is [B, T, 80]. The purpose of this module is to uniformly map the content corresponding to all different styles and language combinations of the training set to the same discrete representation space, which enhances the generalization of the content representation, and thus enhances the synthesis effect of the speaker speaking an unfamiliar language during cross-language transfer. The final input of the CFM (Conditional Flow Matching) decoder is: random noise [B, T, 80], speaker representation [B, T, 80], discrete speech representation [B, T, 80], and the output of the decoder is a Mel spectrum of shape [B, T, 80], that is, the target acoustic model is obtained.

[0049] As an optional implementation method, the initial acoustic model is trained using basic data samples and spectral data samples to obtain a target acoustic model, including: determining language feature samples and style feature samples in the basic data samples, wherein the language feature samples are used to indicate semantic samples and pronunciation feature samples of different languages ​​in the basic data samples, and the style feature samples are used to indicate language styles of different languages ​​in the basic data samples; and the initial acoustic model is trained using the language feature samples and the style feature samples to obtain a target acoustic model.

[0050] In this embodiment, a language feature sample and a style feature sample are determined in the basic data sample, wherein the language feature sample can also be called a language representation, and the style feature sample can also be called a style representation.

[0051] For example, construct a language embedding of dimension (denoted as d), and use the language identifier to convert it into a language representation through a lookup operation. This representation is added to the phoneme representation of the same dimension and input into the encoder to model the context representation. Since the language representation is at the phoneme level, it can flexibly support a variety of situations such as Chinese, English, and Chinese-English mixed text.

[0052] For another example, construct a style embedding of dimension (denoted as D), and use the style to convert it into a style representation through a table lookup operation.

[0053] Optionally, after the language feature samples and the style feature samples are determined, the language feature samples and the style feature samples are input into an initial acoustic model for training to obtain a target acoustic model.

[0054] As an optional implementation method, the initial acoustic model is trained using language feature samples and style feature samples to obtain a target acoustic model, including: integrating the language feature samples and style feature samples to obtain integrated feature samples; and using the integrated feature samples to train the initial acoustic model to obtain a target acoustic model.

[0055] In this embodiment, the language feature sample and the style feature sample are integrated to obtain an integrated feature sample. For example, the language feature sample and the style feature sample are spliced ​​to obtain an integrated feature sample. The integrated feature sample can also be called a composite representation.

[0056] For example, the style embedding at the input of the variable predictor is first expanded to the phoneme level, that is, expanded to the number of phonemes (denoted as T) through a broadcast operation. This representation is concatenated with the language representation at the phoneme level in D dimensions to form a composite representation of 2D dimensions.

[0057] Optionally, after obtaining the integrated feature samples, the integrated feature samples are input into an initial acoustic model for training to obtain a target acoustic model.

[0058] For example, a language style composite representation predictor module is constructed. The main body of this module is the global style tokens (GST) based on the attention mechanism, where the Query of the attention mechanism is the input, that is, the concatenated phoneme-level style representation and language representation, with a shape of [B, T, 2D], where B is the batch size; Key and Value are N randomly initialized global style tokens, with a shape of [N, 2D], and the output is the phoneme-level, weighted average GST representation [B, T, 2D], which is finally mapped to the [B, T, D] dimension as the final language style composite representation. This representation is added to the context representation of the encoder as the conditional input of the fundamental frequency predictor and the duration predictor. The original intention of the language style composite representation predictor module is to perform global unsupervised clustering of the composite representation of language and style on the training set, and map it to a continuous, normalized attention weight space, so that the composite representation is more generalizable and the transfer effect is more natural.

[0059] As an optional implementation method, the audio synthesis method also includes: obtaining a joint loss model, wherein the joint loss model is used to indicate a target that needs to be optimized for the initial acoustic model during training; and training the initial acoustic model using the joint loss model and basic data.

[0060] In this embodiment, a joint loss model is obtained, and the initial acoustic model is trained using the joint loss model and basic data. The joint loss model includes at least duration prediction loss, fundamental frequency prediction loss, CFM prediction loss, prior loss and discrete codebook prediction loss (also called cross entropy loss).

[0061] For example, when constructing training loss, in addition to duration prediction loss, fundamental frequency prediction loss, CFM prediction loss, and prior loss, discrete codebook prediction loss (cross entropy loss) is also added. Based on the above joint loss training model, the model parameters are updated through back propagation.

[0062] It should be noted that the above embodiments may be executed by an audio synthesis device.

[0063] In this embodiment, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into the target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio features of the audio to be synthesized, and the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. That is, the embodiment of the present invention obtains spectral data by analyzing the basic data of the audio to be synthesized using the target acoustic model, thereby synthesizing audio according to the spectral data. Since the target acoustic model of the embodiment of the present invention can decouple multiple timbres and styles, the technical problem of low quality of the synthesized audio is solved, and the technical effect of improving the quality of the synthesized audio is achieved.

[0064] The technical solution of the embodiment of the present invention is illustrated below in conjunction with preferred implementation modes.

[0065] At present, audio synthesis is an important technology that can merge multiple audio clips into a complete audio file. Through audio synthesis, various types of music, sound effects, and speech can be created. Audio synthesis technology is also widely used in film, television, radio and other fields, providing richer and more diverse sound expressions for various media content. With the development of artificial intelligence and machine learning technology, audio synthesis technology is also constantly innovating and improving. In the future, audio synthesis technology is expected to play a more important role in virtual reality, augmented reality, and speech recognition. Cross-lingual speech synthesis (Cross-lingual TTS) technology generally refers to the data of each speaker speaking a single language during training, but during reasoning, it is hoped that the speaker can synthesize any target language and speak it authentically.

[0066] Under normal circumstances, by sharing text content encoding and distinguishing between speaker embedding and language embedding, any speaker in the set can speak any language in the set. However, if the input speaker and the language representation seen in the set (rather than the target language representation) are used for cross-language synthesis, the target language is not authentic enough; but if the input speaker and the target language representation not seen in the set are used, the speaker similarity is not good enough during synthesis due to the lack of generalization in modeling. At the same time, in recent years, synthesis solutions based on pre-trained speech large models have emerged, which can support cross-language synthesis functions by using the powerful context learning ability of self-supervised autoregressive large models. However, due to the serious time-consuming nature of autoregressive models, it is difficult to meet business deployment requirements. There are also some works that use the generalization of discrete representations of pre-trained speech large models to decouple content and timbre. However, due to the use of authentic speakers to model rhythmic information, the style of the target speaker is lost. Therefore, there is a technical problem of low quality of synthesized audio. For the above-mentioned technical problem of low quality of synthesized audio, no effective solution has been proposed yet.

[0067] However, an embodiment of the present invention proposes a cross-language synthesis method based on language style fusion features, which establishes an acoustic model structure, obtains the Mel spectrum predicted by the acoustic model by inputting the text to be synthesized, the target speaker, the style, and the target language, and then synthesizes the audio through a vocoder, thereby solving the technical problem of low quality of the synthesized audio and achieving the technical effect of improving the quality of the synthesized audio.

[0068] The embodiments of the present invention are further described below.

[0069] Figure 2 1 is a flow chart of a cross-language synthesis method based on language style fusion features according to an embodiment of the present invention. The cross-language synthesis method based on language style fusion features comprises the following steps:

[0070] Step S201, obtaining audio data.

[0071] In this embodiment, audio data is obtained to create a multi-language (for example, Chinese and English), multi-timbre, and multi-style data set. Frame-level fundamental frequency features are extracted and normalized according to the mean variance of each speaker, and speaker information is removed from the fundamental frequency features.

[0072] Optionally, each speaker specifies at least one style name. Here, we consider style to be each speaker's unique pronunciation habits, including speaking speed, intonation, emotion, etc. Different speakers do not share styles. Each speaker specifies one timbre name. Table 1 is a schematic diagram of a sample composition of a multi-speaker, multi-style, and multi-language sound library dataset. As shown in Table 1, a sample of a multi-language, multi-timbre, and multi-style dataset is shown.

[0073] Table 1 Schematic diagram of the composition of a multi-speaker, multi-style, and multi-language sound library dataset

[0074]

[0075] Step S202: construct an acoustic model.

[0076] In this embodiment, Figure 3 is a schematic diagram of an acoustic model structure according to an embodiment of the present invention, such as Figure 3 As shown, in addition to the phoneme identification sequence and the speaker identification, the input also includes a phoneme-level language identification sequence and a style identification.

[0077] Optionally, the present invention constructs a base model based on a non-autoregressive FastPitch encoder, a variable predictor, and a CFM (Conditional Flow Matching) decoder of Matcha-TTS, and introduces discrete speech coding extracted by an open source model as input to the decoder.

[0078] Optionally, language embeddings and style embeddings of the same dimension as the phoneme embedding sequence (assuming the dimension is d) are added to the model input. The input language embeddings are added to the corresponding phoneme embeddings in units of phonemes to facilitate the support of multi-language mixing. The style embeddings input to the variable predictor are first expanded to the phoneme level and concatenated with the language embedding sequence to form a 2d-dimensional composite representation.

[0079] Optionally, in the variable predictor part, add a language style composite representation predictor module. The main body of this module is GST (Global Style Tokens) based on the attention mechanism, where the Query of the attention mechanism is the input, that is, the concatenated phoneme-level style representation and language representation, with a shape of [B, T, 2d], where B is the batch size; Key and Value are N randomly initialized global style tokens, with a shape of [N, 2d], and the output is the phoneme-level, weighted average GST representation [B, T, 2d], which is finally mapped to the [B, T, d] dimension as the final language style fusion feature. This representation is added to the context representation of the encoder as the conditional input of the fundamental frequency predictor and the duration predictor.

[0080] Optionally, based on the sample data set shown in Table 1, Figure 4 FIG. 1 is a schematic diagram of a language style composite feature predictor in the training and reasoning process according to an embodiment of the present invention. Figure 4As shown in the figure, during training, the language style composite representation predictor has seen the phoneme-level style and language concatenation vectors. The predictor predicts the weight corresponding to each case through the attention mechanism, and finally generates a language style fusion feature vector. In this process, it is equivalent to clustering all language and style combinations in the training set and mapping them to a globally shared weight space.

[0081] For example, when inferring, suppose that speaker D is expected to speak English in a happy (S6) style. Through the prediction of the language style composite representation predictor, its weight may be close to the weight corresponding to speaking English in an excited (S4) style in the training set, because both the S4 excited style and the S6 happy style have large intonation fluctuations, and the corresponding language is English. Figure 4 Another example is shown in the figure. When speaker A speaks English in a neutral (S1) style, the weights predicted by the model may be very similar to the weights corresponding to speaker B speaking English in a calm (S3) style. Therefore, the language style composite representation predictor performs global unsupervised clustering of the composite representation of language and style on the training set and maps it to a continuous, normalized attention weight space, making the composite representation more generalizable and more natural in cross-language transfer.

[0082] Optionally, the attention mechanism is a machine learning method that mimics the human attention allocation mechanism and is used to weight different parts of information when processing large-scale data to improve the performance and efficiency of the model. By introducing the attention mechanism, the model can focus more on information related to the current task, thereby improving the understanding and processing capabilities of the input data.

[0083] Alternatively, the attention mechanism is often applied to neural network models, such as recurrent neural networks, long short-term memory networks, etc. In these models, the attention mechanism can assign different weights according to the importance of the input data, so that the model can better capture the key information of the input data.

[0084] Alternatively, through the attention mechanism, the model can automatically learn which information should be paid attention to in different tasks and scenarios, thereby improving the generalization ability and adaptability of the model. Therefore, the attention mechanism has become an important and effective technical means in the current field of deep learning.

[0085] Optionally, the original intention of the language-style composite representation predictor module is to perform global unsupervised clustering of the composite representation of language and style on the training set, and map it to a continuous, normalized attention weight space, so that the composite representation is more generalizable and the transfer effect is more natural.

[0086] Optionally, a frame-level discrete speech representation predictor module is constructed, and its structure includes both a length predictor and a fundamental frequency predictor. During training, its learning target is the wav2vec 2.0 frame-level discrete codebook of the current audio.

[0087] Optionally, it is generally believed that the discrete representation of Wav2Vec 2.0 mainly contains content and prosody information, and contains less speaker timbre information than other self-supervised audio representations. After the discrete codebook is predicted, it is quantized through an additional discrete codebook embedding layer to obtain the final audio discrete representation.

[0088] Optionally, the purpose of this module is to uniformly map the contents corresponding to different styles and language combinations in the training set to the same discrete representation space, thereby enhancing the generalization of the content representation and further enhancing the synthesis effect of the speaker speaking an unfamiliar language during cross-language transfer.

[0089] Optionally, the unnaturalness caused by the unfamiliar combination of speakers in the set and unseen language representations during cross-language transfer can be alleviated, and the similarity of the speakers' style and rhythm can be improved while still being able to speak the target language.

[0090] Optionally, the input of the final CFM decoder is: random noise [B, T, 80], speaker representation [B, T, 80], discrete speech representation [B, T, 80], and the output of the decoder is a Mel spectrum of shape [B, T, 80].

[0091] Alternatively, Mel-frequency cepstrum (MFCC) is a feature representation method widely used in speech signal processing and audio feature extraction. It is obtained by performing a series of processing steps on the audio signal and is mainly used for tasks such as speech recognition, speaker recognition, and audio classification.

[0092] Step S203, constructing training loss.

[0093] In this embodiment, in addition to the duration prediction loss, the fundamental frequency prediction loss, the prediction loss (Conditional Flow Matching, referred to as CFM), and the prior loss, a discrete codebook prediction loss (cross entropy loss) is also added. Based on the above joint loss training model, the model parameters are updated through back propagation.

[0094] Optionally, duration prediction loss refers to the difference between the model prediction result and the actual duration when the model is used for duration prediction. This difference is usually measured by different loss functions, and common loss functions include mean squared error (MSE), mean absolute error (MAE), etc.

[0095] Optionally, the magnitude of the duration prediction loss can reflect the prediction accuracy of the model, and the smaller the loss, the more accurate the prediction of the model. By minimizing the duration prediction loss, the performance of the model can be optimized and the accuracy and stability of the duration prediction can be improved.

[0096] Optionally, the fundamental frequency prediction loss refers to an indicator used to evaluate the accuracy of the fundamental frequency prediction algorithm in speech signal processing. Fundamental frequency prediction refers to predicting the fundamental frequency information in speech, that is, the basic frequency of the sound, based on the waveform characteristics of the speech signal. The fundamental frequency prediction loss is the difference between the predicted fundamental frequency value and the actual fundamental frequency value.

[0097] Optionally, in general, the fundamental frequency prediction loss can be evaluated by calculating the mean square error or other distance metrics between the predicted value and the true value. The smaller the fundamental frequency prediction loss, the higher the accuracy of the prediction algorithm, and vice versa, it means that the prediction algorithm has a large error.

[0098] Optionally, the fundamental frequency prediction loss plays an important role in speech signal processing, which can help evaluate the performance of the fundamental frequency prediction algorithm and have an impact on applications such as speech synthesis and speech recognition. By continuously optimizing the fundamental frequency prediction algorithm, the accuracy and efficiency of speech signal processing can be improved.

[0099] Optionally, CFM's methods of predicting losses mainly include using historical data for trend analysis, developing cash flow forecasting models, considering market and industry factors, etc. Through these methods, the company can predict possible cash flow problems in the future and take timely measures to reduce losses and ensure the company's financial stability.

[0100] Optionally, the prior loss refers to the loss function calculated based on the prior distribution and observed data when performing Bayesian statistical inference. In Bayesian statistical inference, we first give a prior distribution to represent the uncertainty of the parameters, and then update this distribution based on the observed data to obtain the posterior distribution. Prior loss can help us take into account the uncertainty of parameters during the inference process, so as to estimate parameter values ​​or make decisions more accurately. Common prior losses include mean square loss, absolute loss, etc.

[0101] Optionally, the discrete codebook prediction loss, also known as the cross entropy loss, is a commonly used loss function to evaluate the gap between the prediction results of the classification model and the true label. In the discrete codebook prediction task, the cross entropy loss is usually used to measure the degree of match between the model's predicted probability distribution for each category and the true label.

[0102] Alternatively, in discrete codebook prediction tasks, the output of the model is usually converted into a probability distribution using a softmax function, and then the cross entropy loss is calculated. The goal of the model is to minimize the cross entropy loss to improve prediction accuracy.

[0103] Optionally, the softmax function is a commonly used activation function, which is often used in the output layer in multi-classification problems. The softmax function can transform a real number vector into a probability distribution so that each element in the vector is between 0 and 1, and the sum of all elements is 1.

[0104] Step S204, synthesizing audio.

[0105] In this embodiment, the Mel-spectrogram predicted by the acoustic model is obtained by inputting the text to be synthesized, the target speaker, the style, and the target language, and then the audio is synthesized through the HiFiGan vocoder.

[0106] Alternatively, HiFiGan is a deep learning-based vocoder that can generate high-fidelity audio waveforms. By training a large amount of audio data, HiFiGan can generate realistic sounds, including speech, music, and other sound effects. This vocoder can be used in applications such as audio synthesis, speech synthesis, and audio enhancement. Its advantage is that the generated audio quality is very high and sounds almost the same as real recordings. HiFiGan has achieved great success in the field of audio processing and is being widely used in various audio-related projects.

[0107] Optionally, the present invention constructs a non-autoregressive acoustic model based on CFM and pre-trained discrete speech representation, which supports cross-language transfer and does not require a target language reference speaker. This model provides a new acoustic model structure for cross-language transfer, and is much better than the autoregressive large speech model in terms of inference speed.

[0108] Optionally, in the acoustic model structure of cross-language transfer, decoupling timbre and style representation can not only support timbre and style controllability and better preserve the speaker's specified style during cross-language synthesis, but also eliminate timbre leakage or redundancy during discrete speech codebook prediction.

[0109] Optionally, by unsupervised clustering of style and language composite representations, a prosodic representation with a higher degree of integration of a specific style with representations of seen or unseen languages ​​and stronger robustness is obtained. Using this representation modeling, cross-language transfer can be performed more naturally while retaining the unique style of the speakers in the set.

[0110] In this embodiment, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into the target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio features of the audio to be synthesized, the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. That is to say, the present invention obtains spectral data by analyzing the basic data of the audio to be synthesized using the target acoustic model, and then synthesizes audio according to the spectral data. Since the target acoustic model of the present invention can decouple multiple timbres and styles, the technical problem of low quality of the synthesized audio is solved, and the technical effect of improving the quality of the synthesized audio is achieved.

[0111] According to an embodiment of the present invention, there is also provided an audio synthesis device. It should be noted that the audio synthesis device can be used to execute the audio synthesis method in the method embodiment.

[0112] Figure 5 is a schematic diagram of an audio synthesis device according to an embodiment of the present invention. Figure 5 As shown, the audio synthesis device 500 may include: an acquisition unit 501, an analysis unit 502 and a synthesis unit 503.

[0113] The acquisition unit 501 is used to acquire basic data of the audio to be synthesized, wherein the basic data is used to indicate conditions that need to be met for synthesizing the audio to be synthesized.

[0114] The analysis unit 502 is used to input the basic data into the target acoustic model for analysis to obtain spectral data, wherein the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of the historical audio to be synthesized.

[0115] The synthesis unit 503 is used to synthesize the basic data into target audio based on the spectrum data.

[0116] Optionally, the audio synthesis device 500 may further include: an extraction unit for extracting fundamental frequency features from basic data, wherein the fundamental frequency features are used to indicate audio information from a pronunciation object in the basic data; and a removal unit for removing fundamental frequency features from the fundamental frequency features.

[0117] Optionally, the audio synthesis device 500 may further include: a training unit, configured to train an initial acoustic model using basic data samples and spectrum data samples to obtain a target acoustic model.

[0118] Optionally, the input unit may include: a determination module, used to determine language feature samples and style feature samples in the basic data samples, wherein the language feature samples are used to indicate semantic samples and pronunciation feature samples of different languages ​​in the basic data samples, and the style feature samples are used to indicate language styles of different languages ​​in the basic data samples; and a training module, used to train the initial acoustic model using the language feature samples and the style feature samples to obtain a target acoustic model.

[0119] Optionally, the training module may include: an integration submodule for integrating the language feature samples and the style feature samples to obtain an integrated feature sample; and a training submodule for training the initial acoustic model using the integrated feature sample to obtain a target acoustic model.

[0120] Optionally, the audio synthesis device 400 may further include: a first acquisition unit, used to acquire a joint loss model, wherein the joint loss model is used to indicate a target that needs to be optimized for the initial acoustic model during training; and a training unit, used to train the initial acoustic model using the joint loss model and basic data.

[0121] In this embodiment, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into the target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio features of the audio to be synthesized, and the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. That is, the embodiment of the present invention obtains spectral data by analyzing the basic data of the audio to be synthesized using the target acoustic model, thereby synthesizing audio according to the spectral data. Since the target acoustic model of the embodiment of the present invention can decouple multiple timbres and styles, the technical problem of low quality of the synthesized audio is solved, and the technical effect of improving the quality of the synthesized audio is achieved.

[0122] According to an embodiment of the present invention, a computer-readable storage medium is further provided. The storage medium includes a stored program, wherein the program executes the audio synthesis method in the method embodiment.

[0123] According to an embodiment of the present invention, a processor is further provided. The processor is used to run a program, wherein the program executes the audio synthesis method in the method embodiment when running.

[0124] According to an embodiment of the present invention, a computer program product is further provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the audio synthesis method in the method embodiment is implemented.

[0125] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0126] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0129] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent functional component, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software functional component, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program codes.

[0131] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for synthesizing audio, characterized in that: include: Acquire basic data of the audio to be synthesized, wherein the basic data is used to indicate conditions that need to be met in synthesizing the audio to be synthesized; The basic data is input into a target acoustic model for analysis to obtain spectrum data, wherein the spectrum data is used to indicate the audio features of the audio to be synthesized, the target acoustic model is trained using basic data samples and spectrum data samples, the basic data samples are used to indicate the conditions that need to be met for synthesizing historical audio to be synthesized, and the spectrum data samples are used to indicate the audio features of the historical audio to be synthesized; Based on the frequency spectrum data, the basic data is synthesized into target audio.

2. The method according to claim 1, characterized in that Before inputting the basic data into the target acoustic model for analysis to obtain the spectrum data, the method further includes: Extracting fundamental frequency features from the basic data, wherein the fundamental frequency features are used to indicate audio information from a pronunciation object in the basic data; The fundamental frequency feature is removed from the basic data.

3. The method according to claim 1, characterized in that The method further comprises: The initial acoustic model is trained using the basic data samples and the spectrum data samples to obtain the target acoustic model.

4. The method according to claim 3, characterized in that Using the basic data samples and the spectrum data samples, training the initial acoustic model to obtain the target acoustic model includes: Determine language feature samples and style feature samples in the basic data samples, wherein the language feature samples are used to indicate semantic samples and pronunciation feature samples of different languages ​​in the basic data samples, and the style feature samples are used to indicate language styles of different languages ​​in the basic data samples; The initial acoustic model is trained using the language feature samples and the style feature samples to obtain the target acoustic model.

5. The method according to claim 4, characterized in that Using the language feature samples and the style feature samples, training the initial acoustic model to obtain the target acoustic model includes: Integrate the language feature sample and the style feature sample to obtain an integrated feature sample; The initial acoustic model is trained using the integrated feature samples to obtain the target acoustic model.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire a joint loss model, wherein the joint loss model is used to indicate a target that needs to be optimized during the training of the initial acoustic model; The initial acoustic model is trained using the joint loss model and the basic data samples.

7. An audio synthesis device, characterized in that: include: An acquisition unit, used to acquire basic data of the audio to be synthesized, wherein the basic data is used to indicate conditions that need to be met in synthesizing the audio to be synthesized; an analysis unit, configured to input the basic data into a target acoustic model for analysis to obtain spectrum data, wherein the target acoustic model is trained using basic data samples and spectrum data samples, the basic data samples are used to indicate conditions that need to be met for synthesizing historical audio to be synthesized, and the spectrum data samples are used to indicate audio features of the historical audio to be synthesized; A synthesis unit is used to synthesize the basic data into target audio based on the spectrum data.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 6 when running.

10. A computer program product, characterized in that The computer program product comprises computer instructions, which implement the method according to any one of claims 1 to 6 when executed by a processor.

Images