Audio synthesis method, device, processor and readable storage medium

Abstract

The present invention discloses an audio synthesis method, device, processor, and readable storage medium. The method includes: obtaining basic data of the audio to be synthesized, wherein the basic data indicates the conditions required to synthesize the audio to be synthesized; inputting the basic data into a target acoustic model for analysis to obtain spectral data, wherein the spectral data indicates the audio characteristics of the audio to be synthesized; the target acoustic model is trained using basic data samples and spectral data samples, wherein the basic data samples indicate the conditions required to synthesize historical audio to be synthesized, and the spectral data samples indicate the audio characteristics of historical audio to be synthesized; and synthesizing the basic data into the target audio based on the spectral data. The present invention solves the technical problem of low synthesized audio quality.

Description

Technical Field

[0001] The present invention relates to the technical field of audio synthesis, and in particular to an audio synthesis method, device, processor and readable storage medium. Background Art

[0002] Audio synthesis is currently a crucial technology, combining multiple audio clips into a complete audio file. Through audio synthesis, a wide variety of music, sound effects, and speech can be created. Audio synthesis technology is also widely used in film, television, and broadcasting, providing richer and more diverse sound representations for various media content. With the advancement of artificial intelligence and machine learning, audio synthesis technology is constantly innovating and improving. In the future, audio synthesis technology is expected to play an even more important role in fields such as virtual reality, augmented reality, and speech recognition.

[0003] Typically, by sharing text content encodings and distinguishing between speaker embeddings and language embeddings, any speaker in a set can speak any language in the set. However, when synthesizing cross-language audio, if the input speaker and a language representation already seen in the set (rather than the target language representation) are used, the target language may not sound authentic. However, if the input speaker and a target language representation not seen in the set are used, the lack of generalization in the modeling results in poor speaker similarity during synthesis. This results in low synthesized audio quality.

[0004] Currently, no effective solution has been proposed to the above-mentioned technical problem of low quality of synthesized audio. Summary of the Invention

[0005] Embodiments of the present invention provide an audio synthesis method, apparatus, processor, and readable storage medium to at least solve the technical problem of low quality of synthesized audio.

[0006] According to one aspect of an embodiment of the present invention, a method for synthesizing audio is provided. The method may include: obtaining basic data of audio to be synthesized, wherein the basic data indicates conditions required to synthesize the audio to be synthesized; inputting the basic data into a target acoustic model for analysis to obtain spectrum data, wherein the spectrum data indicates audio features of the audio to be synthesized; the target acoustic model is trained using basic data samples and spectrum data samples, wherein the basic data samples indicate conditions required to synthesize historical audio to be synthesized, and the spectrum data samples indicate audio features of historical audio to be synthesized; and synthesizing the basic data into the target audio based on the spectrum data.

[0007] Optionally, before inputting the basic data into the target acoustic model for analysis to obtain spectral data, the audio synthesis method also includes: extracting fundamental frequency features from the basic data, wherein the fundamental frequency features are used to indicate audio information from the pronunciation object in the basic data; and removing the fundamental frequency features from the fundamental frequency features.

[0008] Optionally, the audio synthesis method further includes: using basic data samples and spectrum data samples to train an initial acoustic model to obtain a target acoustic model.

[0009] Optionally, the initial acoustic model is trained using the basic data samples and the spectral data samples to obtain a target acoustic model, including: determining language feature samples and style feature samples in the basic data samples, wherein the language feature samples are used to indicate the semantic samples and pronunciation feature samples of different languages ​​in the basic data samples, and the style feature samples are used to indicate the language styles of different languages ​​in the basic data samples; and the initial acoustic model is trained using the language feature samples and the style feature samples to obtain a target acoustic model.

[0010] Optionally, the language feature samples and the style feature samples are used to train the initial acoustic model to obtain the target acoustic model, including: integrating the language feature samples and the style feature samples to obtain integrated feature samples; and using the integrated feature samples to train the initial acoustic model to obtain the target acoustic model.

[0011] Optionally, the audio synthesis method also includes: obtaining a joint loss model, wherein the joint loss model is used to indicate a target that needs to be optimized for the initial acoustic model during training; and training the initial acoustic model using the joint loss model and basic data samples.

[0012] According to another aspect of an embodiment of the present invention, an audio synthesis device is provided. The device may include: an acquisition unit for acquiring basic data of audio to be synthesized, wherein the basic data indicates conditions required for synthesizing the audio to be synthesized; an analysis unit for inputting the basic data into a target acoustic model for analysis to obtain spectrum data, wherein the target acoustic model is trained using basic data samples and spectrum data samples, the basic data samples indicate conditions required for synthesizing historical audio to be synthesized, and the spectrum data samples indicate audio features of historical audio to be synthesized; and a synthesis unit for synthesizing the basic data into target audio based on the spectrum data.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the device where the storage medium is located is controlled to execute the audio synthesis method in the embodiment of the present invention.

[0014] According to another aspect of an embodiment of the present invention, a processor is provided, which is configured to run a program, wherein the program, when running, executes the audio synthesis method of the embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer program product is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the audio synthesis method according to the embodiment of the present invention is implemented.

[0016] In an embodiment of the present invention, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into a target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio features of the audio to be synthesized; the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. That is, the embodiment of the present invention analyzes the basic data of the audio to be synthesized using the target acoustic model to obtain spectral data, thereby synthesizing audio based on the spectral data. Since the target acoustic model of the embodiment of the present invention can decouple multiple timbres and styles, it solves the technical problem of low quality of the synthesized audio and achieves the technical effect of improving the quality of the synthesized audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of an audio synthesis method according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a cross-language synthesis method based on language style fusion features according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of an acoustic model structure according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of the training and inference process of a language style composite feature predictor according to an embodiment of the present invention;

[0022] Figure 5 2 is a schematic diagram of an audio synthesis device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, functional component or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, functional components or devices.

[0025] According to an embodiment of the present invention, an embodiment of a method for synthesizing audio is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] Figure 1 is a flow chart of an audio synthesis method according to an embodiment of the present invention, such as Figure 1 As shown, the method may include the following steps:

[0027] Step S101: Obtain basic data of the audio to be synthesized.

[0028] In the technical solution provided in step S101 of the present invention, the basic data is used to indicate the conditions that need to be met for synthesizing the audio to be synthesized. The basic data can also be called audio data.

[0029] In this embodiment, basic data of the audio to be synthesized is obtained, for example, by an input module based on user input or selection. This is merely an example and does not limit the specific method of obtaining the basic data of the audio to be synthesized.

[0030] Optionally, the basic data includes at least but is not limited to: the text to be synthesized, identification information of the target speaker, style, and target language.

[0031] Step S102: input the basic data into the target acoustic model for analysis to obtain spectrum data.

[0032] In the technical solution provided in the above step S102 of the present invention, the spectral data is used to indicate the audio features of the audio to be synthesized, the target acoustic model is trained using the basic data samples and the spectral data samples, the basic data samples are used to indicate the conditions that need to be met to synthesize the historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of the historical audio to be synthesized.

[0033] In this embodiment, after obtaining the basic data of the audio to be synthesized in step S101, the basic data is input into the target acoustic model for analysis to obtain spectrum data. The target acoustic model can also be called an acoustic model, and the spectrum data can also be a Mel spectrum.

[0034] Optionally, by inputting basic data for analysis, detailed spectrum data can be obtained, including spectrum graphs, spectrum distribution and other information, to help users gain a deeper understanding of the characteristics and structure of sound signals.

[0035] Step S103: synthesize the basic data into target audio based on the spectrum data.

[0036] In the technical solution provided in the above step S103 of the present invention, after the spectrum data is obtained in step S102, the basic data is synthesized into the target audio according to the spectrum data.

[0037] In this embodiment, the basic data is synthesized into the target audio according to the spectrum data. For example, the basic data is synthesized into the target audio by a vocoder. This is only an example and does not limit the specific method of synthesizing the target audio.

[0038] Optionally, the target audio can be accurately reconstructed based on the spectral data by synthesizing the base data, thereby improving the quality and clarity of the audio.

[0039] It should be noted that the above embodiment can be executed by an audio synthesis device.

[0040] In the above steps S101 to S103 of the present invention, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met for synthesizing the audio to be synthesized; the basic data is input into the target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio characteristics of the audio to be synthesized; the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met for synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio characteristics of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. In other words, the embodiment of the present invention analyzes the basic data of the audio to be synthesized using the target acoustic model to obtain spectral data, and thus synthesizes audio based on the spectral data. Since the target acoustic model of the embodiment of the present invention can decouple multiple timbres and styles, it solves the technical problem of low quality of the synthesized audio and achieves the technical effect of improving the quality of the synthesized audio.

[0041] The above method of this embodiment is further introduced below.

[0042] As an optional embodiment, before inputting the basic data into the target acoustic model for analysis to obtain spectral data, the audio synthesis method also includes: extracting fundamental frequency features from the basic data, wherein the fundamental frequency features are used to indicate audio information from the pronunciation object in the basic data; and removing the fundamental frequency features from the basic data.

[0043] In this embodiment, fundamental frequency features are extracted from the basic data, wherein the fundamental frequency features may include: phoneme identification sequence features, speaker identification features, phoneme-level language identification sequence features, and style identification features.

[0044] Optionally, after extracting the fundamental frequency features, the fundamental frequency features in the base data are determined and removed. For example, the fundamental frequency features can be normalized. Because different speakers may have significantly different fundamental frequency data due to factors such as physiological structure and pronunciation habits, normalization can smooth out these differences, thereby more accurately determining the fundamental frequency features in the base data, and then removing the normalized fundamental frequency features from the base data.

[0045] For example, frame-level fundamental frequency features are extracted from the basic data and normalized according to the mean variance of each speaker to remove speaker information from the fundamental frequency features.

[0046] As an optional embodiment, the audio synthesis method includes: using basic data samples and spectrum data samples to train an initial acoustic model to obtain a target acoustic model.

[0047] In this embodiment, basic data samples and spectrum data samples are input into an initial acoustic model for training to obtain a target acoustic model, wherein the basic data samples are sample data in a data set composed of multiple basic data.

[0048] For example, audio data is acquired to create a multilingual (using Chinese and English as examples), multi-timbre, and multi-style dataset. In addition to the phoneme identifier sequence and speaker identifier, the initial acoustic model is fed with a phoneme-level language identifier sequence and a style identifier. Language and style embeddings are constructed, and a frame-level discrete speech representation predictor module is constructed, similar in structure to the duration and fundamental frequency predictors. During training, the learning target is the wav2vec2.0 frame-level discrete codebook of the current audio. It is generally believed that the Wav2vec2.0 discrete representation primarily contains content and prosody information and contains less speaker timbre information than other self-supervised audio representations. After predicting the discrete codebook, an additional discrete codebook embedding layer is applied to obtain the final discrete audio representation, using the shape [B, T, 80]. This module aims to uniformly map all content corresponding to different styles and language combinations in the training set into the same discrete representation space, enhancing the generalizability of the content representation and, in turn, improving the synthesis of unfamiliar speakers during cross-lingual transfer. The final input of the CFM (Conditional Flow Matching) decoder is: random noise [B, T, 80], speaker representation [B, T, 80], discrete speech representation [B, T, 80]. The output of the decoder is a Mel spectrum of shape [B, T, 80], that is, the target acoustic model is obtained.

[0049] As an optional implementation method, the initial acoustic model is trained using basic data samples and spectral data samples to obtain a target acoustic model, including: determining language feature samples and style feature samples in the basic data samples, wherein the language feature samples are used to indicate semantic samples and pronunciation feature samples of different languages ​​in the basic data samples, and the style feature samples are used to indicate the language styles of different languages ​​in the basic data samples; and using the language feature samples and style feature samples to train the initial acoustic model to obtain a target acoustic model.

[0050] In this embodiment, language feature samples and style feature samples are determined in the basic data sample, wherein the language feature samples may also be referred to as language representations, and the style feature samples may also be referred to as style representations.

[0051] For example, a language embedding of dimension (denoted as d) is constructed, and a language identifier is used to convert it into a language representation through a lookup operation. This representation is then added to a phoneme representation of the same dimension and fed into the encoder to model the contextual representation. Because the language representation is at the phoneme level, it can flexibly support a variety of scenarios, such as Chinese, English, and mixed Chinese and English text.

[0052] For another example, construct a style embedding of dimension (denoted as D), and use the style to convert it into a style representation through a table lookup operation.

[0053] Optionally, after the language feature samples and the style feature samples are determined, the language feature samples and the style feature samples are input into an initial acoustic model for training to obtain a target acoustic model.

[0054] As an optional implementation method, the initial acoustic model is trained using language feature samples and style feature samples to obtain a target acoustic model, including: integrating the language feature samples and style feature samples to obtain an integrated feature sample; and using the integrated feature sample to train the initial acoustic model to obtain a target acoustic model.

[0055] In this embodiment, the language feature sample and the style feature sample are integrated to obtain an integrated feature sample. For example, the language feature sample and the style feature sample are concatenated to obtain an integrated feature sample. The integrated feature sample can also be called a composite representation.

[0056] For example, the style embedding at the input of the variable predictor is first expanded to the phoneme level, that is, expanded to the number of phonemes (denoted as T) through a broadcast operation. This representation is then concatenated with the phoneme-level language representation in D dimensions to form a 2D composite representation.

[0057] Optionally, after obtaining the integrated feature samples, the integrated feature samples are input into the initial acoustic model for training to obtain a target acoustic model.

[0058] For example, a language style composite representation predictor module is constructed. The main body of this module is the Global Style Tokens (GST) based on the attention mechanism. The attention mechanism's query is the input, which is the concatenated phoneme-level style and language representations, with a shape of [B, T, 2D], where B is the batch size. The key and value are N randomly initialized global style tokens, with a shape of [N, 2D]. The output is the phoneme-level, weighted average GST representation [B, T, 2D], which is finally mapped to the [B, T, D] dimension as the final language style composite representation. This representation is added to the encoder's contextual representation and serves as the conditional input for the fundamental frequency predictor and duration predictor. The original intention of the language style composite representation predictor module is to perform global unsupervised clustering on the training set of the concatenated language and style representations and map them to a continuous, normalized attention weight space, making the composite representation more generalizable and the transfer effect more natural.

[0059] As an optional embodiment, the audio synthesis method also includes: obtaining a joint loss model, wherein the joint loss model is used to indicate the target that needs to be optimized for the initial acoustic model during the training process; and training the initial acoustic model using the joint loss model and basic data.

[0060] In this embodiment, a joint loss model is obtained and the initial acoustic model is trained using the joint loss model and basic data. The joint loss model includes at least duration prediction loss, fundamental frequency prediction loss, CFM prediction loss, prior loss, and discrete codebook prediction loss (also known as cross entropy loss).

[0061] For example, when constructing training losses, in addition to duration prediction loss, fundamental frequency prediction loss, CFM prediction loss, and prior loss, a discrete codebook prediction loss (cross entropy loss) is also added. Based on this combined loss training model, the model parameters are updated through backpropagation.

[0062] It should be noted that the above embodiment can be executed by an audio synthesis device.

[0063] In this embodiment, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into a target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio features of the audio to be synthesized; the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. In other words, the embodiment of the present invention uses the target acoustic model to analyze the basic data of the audio to be synthesized to obtain spectral data, thereby synthesizing audio based on the spectral data. Since the target acoustic model of the embodiment of the present invention can decouple multiple timbres and styles, it solves the technical problem of low quality of the synthesized audio and achieves the technical effect of improving the quality of the synthesized audio.

[0064] The technical solutions of the embodiments of the present invention are described below with reference to preferred implementation methods.

[0065] Currently, audio synthesis is an important technology that can merge multiple audio clips into a complete audio file. Through audio synthesis, various types of music, sound effects, and speech can be created. Audio synthesis technology is also widely used in film, television, radio and other fields, providing richer and more diverse sound expressions for various media content. With the development of artificial intelligence and machine learning technology, audio synthesis technology is also constantly innovating and improving. In the future, audio synthesis technology is expected to play a more important role in fields such as virtual reality, augmented reality, and speech recognition. Cross-lingual speech synthesis (Cross-lingual TTS) technology generally refers to data where only a single language is spoken by each speaker during training, but during inference, it is hoped that the speaker can synthesize any target language and speak it authentically.

[0066] Typically, by sharing text content encodings and distinguishing between speaker embeddings and language embeddings, any speaker in a set can speak any language in the set. However, when synthesizing across languages, if the input speaker and a representation of the language already seen in the set (rather than the target language representation) are used, the target language may not sound authentic. However, if the input speaker and a representation of the target language not seen in the set are used, the model lacks generalizability, resulting in poor speaker similarity during synthesis. Furthermore, synthesis solutions based on pre-trained large speech models have emerged in recent years. These solutions leverage the powerful contextual learning capabilities of self-supervised autoregressive large models to support cross-language synthesis. However, due to the high time consumption of autoregressive models, they struggle to meet deployment requirements. Other work leverages the generalizability of the discrete representations of pre-trained large speech models to decouple content and timbre. However, using a native speaker to model prosodic information can result in a loss of the target speaker's style. Consequently, the synthesized audio quality suffers from low quality. Currently, no effective solution has been proposed to address this issue.

[0067] However, an embodiment of the present invention proposes a cross-language synthesis method based on language style fusion features. By establishing an acoustic model structure, the mel spectrum predicted by the acoustic model is obtained by inputting the text to be synthesized, the target speaker, the style, and the target language, and then the audio is synthesized through a vocoder. This solves the technical problem of low quality of the synthesized audio and achieves the technical effect of improving the quality of the synthesized audio.

[0068] The following is a further introduction to the embodiments of the present invention.

[0069] Figure 2 1 is a flowchart of a cross-language synthesis method based on language style fusion features according to an embodiment of the present invention. The cross-language synthesis method based on language style fusion features includes the following steps:

[0070] Step S201: Acquire audio data.

[0071] In this embodiment, audio data is acquired to create a multi-language (using Chinese and English as examples), multi-timbre, and multi-style dataset. Frame-level fundamental frequency features are extracted and normalized based on the mean-variance of each speaker, removing speaker information from the fundamental frequency features.

[0072] Optionally, each speaker specifies at least one style name. Here, we consider style to be the unique pronunciation habits of each speaker, including speaking speed, intonation, and emotion. Styles are not shared across speakers. Each speaker specifies one timbre name. Table 1 shows an example of a multi-speaker, multi-style, and multi-language sound library dataset. This example illustrates a multi-language, multi-timbre, and multi-style dataset.

[0073] Table 1 Schematic diagram of the composition of a multi-speaker, multi-style, and multi-language sound library dataset

[0074]

[0075] Step S202: construct an acoustic model.

[0076] In this embodiment, Figure 3 is a schematic diagram of an acoustic model structure according to an embodiment of the present invention, such as Figure 3 As shown, in addition to the phoneme identification sequence and speaker identification, the input also includes a phoneme-level language identification sequence and a style identification.

[0077] Optionally, the present invention constructs a base model based on a non-autoregressive FastPitch encoder, a variable predictor, and a CFM (Conditional Flow Matching) decoder of Matcha-TTS, and introduces discrete speech coding extracted by an open source model as input to the decoder.

[0078] Optionally, language and style embeddings of the same dimension as the phoneme embedding sequence (assuming dimension d) are added to the model input. The input language embeddings are added to the corresponding phoneme embeddings in units of phonemes to facilitate support for multilingual mixing. The style embeddings input to the variable predictor are first expanded to the phoneme level and concatenated with the language embedding sequence to form a 2D composite representation.

[0079] Optionally, a language style composite representation predictor module is added to the variable predictor. This module is based on the attention mechanism's GST (Global Style Tokens). The attention mechanism's query is the input, which is the concatenated phoneme-level style and language representations, with a shape of [B, T, 2d], where B is the batch size. The key and value are N randomly initialized global style tokens, with a shape of [N, 2d]. The output is a phoneme-level, weighted average GST representation [B, T, 2d], which is finally mapped to the [B, T, d] dimension as the final language style fusion feature. This representation is added to the encoder's contextual representation and serves as the conditional input for the fundamental frequency predictor and duration predictor.

[0080] Optionally, based on the sample data set shown in Table 1, Figure 4 FIG. 1 is a schematic diagram of the training and inference process of a language style composite feature predictor according to an embodiment of the present invention. Figure 4As shown in the figure, during training, the language style composite representation predictor represents the phoneme-level style and language concatenation vectors seen by the predictor. Using an attention mechanism, the predictor predicts the weight corresponding to each case and ultimately generates a language style fusion feature vector. This process is equivalent to clustering all language and style combinations in the training set and mapping them to a globally shared weight space.

[0081] For example, during inference, if we assume that speaker D is expected to speak English in a happy (S6) style, the weight predicted by the language style composite representation predictor may be close to the weight corresponding to speaking English in an excited (S4) style in the training set, because both the S4 excited style and the S6 happy style have large intonation fluctuations, and the corresponding language is English. Figure 4 Another example is shown in the previous section. For example, if speaker A speaks English in a "Zhongxing" (S1) style, the model's predicted weights are likely to be very similar to those for speaker B speaking English in a "Calm" (S3) style. Therefore, the language style composite representation predictor performs global unsupervised clustering on the training set of composite representations of language and style, and maps them to a continuous, normalized attention weight space. This makes the composite representation more generalizable and more natural in cross-language transfer.

[0082] Alternatively, the attention mechanism is a machine learning method that mimics how humans allocate attention. It is used to weight different parts of information when processing large amounts of data to improve model performance and efficiency. By introducing the attention mechanism, the model can focus more closely on information relevant to the current task, thereby improving its understanding and processing of input data.

[0083] Alternatively, the attention mechanism is often applied to neural network models, such as recurrent neural networks, long short-term memory networks, etc. In these models, the attention mechanism can assign different weights according to the importance of the input data, so that the model can better capture the key information of the input data.

[0084] Alternatively, through the attention mechanism, the model can automatically learn which information to focus on in different tasks and scenarios, thereby improving the model's generalization and adaptability. Therefore, the attention mechanism has become an important and effective technical means in the field of deep learning.

[0085] Optionally, the original intention of the language-style composite representation predictor module is to perform global unsupervised clustering of the composite representation of language and style on the training set, and map it to a continuous, normalized attention weight space, so that the composite representation has stronger generalization and the transfer effect is more natural.

[0086] Optionally, a frame-level discrete speech representation predictor module is constructed, which includes both a length predictor and a fundamental frequency predictor. During training, its learning target is the wav2vec 2.0 frame-level discrete codebook of the current audio.

[0087] Alternatively, it is generally believed that the discrete representation of Wav2Vec 2.0 primarily contains content and prosody information, and contains less speaker timbre information than other self-supervised audio representations. After the discrete codebook is predicted, it is quantized through an additional discrete codebook embedding layer to obtain the final discrete audio representation.

[0088] Optionally, the purpose of this module is to uniformly map the content corresponding to different styles and language combinations in the training set to the same discrete representation space, thereby enhancing the generalization of content representation and thereby enhancing the synthesis effect of the speaker speaking an unfamiliar language during cross-language transfer.

[0089] Optionally, the unnaturalness caused by the unfamiliar combination of speakers in the set and unseen language representations during cross-language transfer can be alleviated, and the similarity of the speakers' style and rhythm can be improved while still being able to speak the target language.

[0090] Optionally, the input of the final CFM decoder is: random noise [B, T, 80], speaker representation [B, T, 80], discrete speech representation [B, T, 80], and the output of the decoder is a Mel spectrum of shape [B, T, 80].

[0091] Alternatively, Mel-frequency cepstrum (MFCC) is a feature representation method widely used in speech signal processing and audio feature extraction. It is obtained by performing a series of processing steps on the audio signal and is mainly used for tasks such as speech recognition, speaker recognition, and audio classification.

[0092] Step S203: construct training loss.

[0093] In this embodiment, in addition to the duration prediction loss, fundamental frequency prediction loss, prediction loss (Conditional Flow Matching, CFM), and prior loss, a discrete codebook prediction loss (cross entropy loss) is also added. Based on this joint loss training model, the model parameters are updated through backpropagation.

[0094] Optionally, duration prediction loss refers to the difference between the model's prediction and the actual duration when using the model for duration prediction. This difference is usually measured using different loss functions, including the mean squared error (MSE) and mean absolute error (MAE).

[0095] Optionally, the duration prediction loss can reflect the model's prediction accuracy, with smaller losses indicating more accurate predictions. By minimizing the duration prediction loss, the model's performance can be optimized, improving the accuracy and stability of duration predictions.

[0096] Optionally, Pitch Prediction Loss is a metric used to evaluate the accuracy of pitch prediction algorithms in speech signal processing. Pitch prediction involves predicting the pitch information (i.e., the fundamental frequency) of a speech signal based on its waveform characteristics. Pitch Prediction Loss is the difference between the predicted pitch and the actual pitch.

[0097] Alternatively, the fundamental frequency prediction loss can be evaluated by calculating the mean square error (MSE) or other distance metrics between the predicted value and the true value. A smaller fundamental frequency prediction loss indicates a higher accuracy of the prediction algorithm, whereas a smaller MSE indicates a larger error in the prediction algorithm.

[0098] Alternatively, fundamental frequency prediction loss plays an important role in speech signal processing. It can help evaluate the performance of fundamental frequency prediction algorithms and have an impact on applications such as speech synthesis and speech recognition. By continuously optimizing the fundamental frequency prediction algorithm, the accuracy and efficiency of speech signal processing can be improved.

[0099] Alternatively, CFM loss forecasting methods include using historical data for trend analysis, developing cash flow forecasting models, and considering market and industry factors. Through these methods, companies can anticipate potential future cash flow problems and take timely measures to mitigate losses and safeguard their financial stability.

[0100] Alternatively, a prior loss refers to a loss function calculated based on a prior distribution and observed data during Bayesian statistical inference. In Bayesian statistical inference, we first give a prior distribution to represent parameter uncertainty, then update this distribution based on the observed data to obtain the posterior distribution. Prior loss can help us account for parameter uncertainty during inference, leading to more accurate parameter value estimates or decision making. Common prior losses include mean squared loss and absolute loss.

[0101] Alternatively, the discrete codebook prediction loss, also known as cross-entropy loss, is a commonly used loss function to evaluate the gap between the prediction results of a classification model and the true label. In discrete codebook prediction tasks, cross-entropy loss is often used to measure how well the model's predicted probability distribution for each class matches the true label.

[0102] Alternatively, in discrete codebook prediction tasks, the softmax function is often used to convert the model's output into a probability distribution, and then the cross-entropy loss is calculated. The goal of the model is to minimize the cross-entropy loss to improve prediction accuracy.

[0103] Alternatively, the softmax function is a commonly used activation function, often used in the output layer of multi-classification problems. The softmax function can transform a real number vector into a probability distribution, so that each element in the vector is between 0 and 1, and the sum of all elements is 1.

[0104] Step S204: synthesize audio.

[0105] In this embodiment, the text to be synthesized, the target speaker, the style, and the target language are input to obtain the mel spectrum predicted by the acoustic model, and then the audio is synthesized through the HiFiGan vocoder.

[0106] Alternatively, HiFiGan is a deep learning-based vocoder that generates high-fidelity audio waveforms. By training on large amounts of audio data, HiFiGan can generate realistic sounds, including speech, music, and other sound effects. This vocoder can be used in applications such as audio synthesis, speech synthesis, and audio enhancement. Its advantage lies in the very high quality of the generated audio, which sounds almost identical to real recordings. HiFiGan has achieved great success in the field of audio processing and is being widely used in various audio-related projects.

[0107] Alternatively, the present invention constructs a non-autoregressive acoustic model based on CFM and pre-trained discrete speech representations that supports cross-language transfer and does not require a reference speaker in the target language. This model provides a new acoustic model structure for cross-language transfer and significantly outperforms large autoregressive speech models in inference speed.

[0108] Optionally, in the acoustic model structure of cross-language transfer, decoupling timbre and style representation can not only support timbre and style controllability and better preserve the speaker's specified style during cross-language synthesis, but also eliminate timbre leakage or redundancy during discrete speech codebook prediction.

[0109] Optionally, unsupervised clustering of composite representations of style and language can yield a more robust prosodic representation that better integrates a specific style with representations of previously seen or unseen languages. Modeling with this representation enables more natural cross-lingual transfer while preserving the unique style of the speakers within the set.

[0110] In this embodiment, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into a target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio features of the audio to be synthesized; the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. In other words, the present invention uses the target acoustic model to analyze the basic data of the audio to be synthesized to obtain spectral data, thereby synthesizing audio based on the spectral data. Since the target acoustic model of the present invention can decouple multiple timbres and styles, it solves the technical problem of low quality of the synthesized audio and achieves the technical effect of improving the quality of the synthesized audio.

[0111] According to an embodiment of the present invention, an audio synthesis device is further provided. It should be noted that the audio synthesis device can be used to execute the audio synthesis method in the method embodiment.

[0112] Figure 5 FIG is a schematic diagram of an audio synthesis device according to an embodiment of the present invention. Figure 5 As shown, the audio synthesis device 500 may include: an acquisition unit 501, an analysis unit 502 and a synthesis unit 503.

[0113] The acquisition unit 501 is configured to acquire basic data of the audio to be synthesized, wherein the basic data is used to indicate conditions that need to be met for synthesizing the audio to be synthesized.

[0114] The analysis unit 502 is used to input the basic data into the target acoustic model for analysis to obtain spectral data, wherein the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met for synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of the historical audio to be synthesized.

[0115] The synthesis unit 503 is configured to synthesize the basic data into target audio based on the spectrum data.

[0116] Optionally, the audio synthesis device 500 may further include: an extraction unit for extracting fundamental frequency features from basic data, wherein the fundamental frequency features are used to indicate audio information from a pronunciation object in the basic data; and a removal unit for removing the fundamental frequency features from the fundamental frequency features.

[0117] Optionally, the audio synthesis device 500 may further include: a training unit, configured to train an initial acoustic model using basic data samples and spectrum data samples to obtain a target acoustic model.

[0118] Optionally, the input unit may include: a determination module for determining language feature samples and style feature samples in the basic data samples, wherein the language feature samples are used to indicate semantic samples and pronunciation feature samples of different languages ​​in the basic data samples, and the style feature samples are used to indicate the language styles of different languages ​​in the basic data samples; a training module for using the language feature samples and style feature samples to train the initial acoustic model to obtain a target acoustic model.

[0119] Optionally, the training module may include: an integration submodule for integrating the language feature samples and the style feature samples to obtain an integrated feature sample; and a training submodule for training the initial acoustic model using the integrated feature sample to obtain a target acoustic model.

[0120] Optionally, the audio synthesis device 400 may further include: a first acquisition unit for acquiring a joint loss model, wherein the joint loss model is used to indicate a target that needs to be optimized for the initial acoustic model during the training process; and a training unit for training the initial acoustic model using the joint loss model and basic data.

[0121] In this embodiment, basic data of the audio to be synthesized is obtained, wherein the basic data is used to indicate the conditions that need to be met in synthesizing the audio to be synthesized; the basic data is input into a target acoustic model for analysis to obtain spectral data, wherein the spectral data is used to indicate the audio features of the audio to be synthesized; the target acoustic model is trained using basic data samples and spectral data samples, the basic data samples are used to indicate the conditions that need to be met in synthesizing historical audio to be synthesized, and the spectral data samples are used to indicate the audio features of historical audio to be synthesized; based on the spectral data, the basic data is synthesized into the target audio. In other words, the embodiment of the present invention uses the target acoustic model to analyze the basic data of the audio to be synthesized to obtain spectral data, thereby synthesizing audio based on the spectral data. Since the target acoustic model of the embodiment of the present invention can decouple multiple timbres and styles, it solves the technical problem of low quality of the synthesized audio and achieves the technical effect of improving the quality of the synthesized audio.

[0122] According to an embodiment of the present invention, a computer-readable storage medium is further provided. The storage medium includes a stored program, wherein the program executes the audio synthesis method in the method embodiment.

[0123] According to an embodiment of the present invention, a processor is further provided. The processor is used to run a program, wherein the program executes the audio synthesis method in the method embodiment when running.

[0124] According to an embodiment of the present invention, a computer program product is further provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the audio synthesis method in the method embodiment is implemented.

[0125] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0126] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0128] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0129] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent functional component, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software functional component, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0131] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for synthesizing audio, characterized in that: include: Acquiring basic data of the audio to be synthesized, wherein the basic data is used to indicate conditions that must be met to synthesize the audio to be synthesized, and the basic data includes at least but is not limited to: the text to be synthesized, identification information of the target speaker, style, and target language; Inputting the basic data into a target acoustic model for analysis to obtain spectrum data, wherein the spectrum data is used to indicate audio features of the audio to be synthesized; the target acoustic model is trained using basic data samples and spectrum data samples; the basic data samples are used to indicate conditions that must be met for synthesizing historical audio to be synthesized; and the spectrum data samples are used to indicate audio features of the historical audio to be synthesized; Based on the spectrum data, synthesizing the basic data into target audio; The step of inputting the basic data into a target acoustic model for analysis to obtain spectral data includes: constructing an initial acoustic model, the initial acoustic model including an encoder, a variable predictor, and a decoder, the variable predictor including a language style composite representation predictor module, a fundamental frequency predictor, and a duration predictor, the language style composite representation predictor module being a global style marker based on an attention mechanism; and training the initial acoustic model to obtain the target acoustic model. The method also includes: splicing the language representation with the phoneme representation of the same dimension, inputting the splicing result into the encoder to model the context representation; adding the context representation with the language style fusion feature, inputting the addition result into the variable predictor to obtain a language style fusion feature vector, wherein the language style fusion feature is obtained by mapping the spliced ​​phoneme-level style representation and the language representation; and inputting random noise, speaker representation, and discrete speech representation into the decoder, obtaining the output of the decoder as a Mel spectrum.

2. The method according to claim 1, characterized in that Before inputting the basic data into the target acoustic model for analysis to obtain spectrum data, the method further includes: Extracting a fundamental frequency feature from the basic data, wherein the fundamental frequency feature is used to indicate audio information from a pronunciation object in the basic data; The fundamental frequency feature is removed from the basic data.

3. The method according to claim 1, characterized in that Training the initial acoustic model to obtain the target acoustic model includes: The initial acoustic model is trained using the basic data samples and the spectrum data samples to obtain the target acoustic model.

4. The method according to claim 3, characterized in that The initial acoustic model is trained using the basic data samples and the spectrum data samples to obtain the target acoustic model, including: Determining language feature samples and style feature samples in the basic data samples, wherein the language feature samples are used to indicate semantic samples and pronunciation feature samples of different languages ​​in the basic data samples, and the style feature samples are used to indicate language styles of different languages ​​in the basic data samples; The initial acoustic model is trained using the language feature samples and the style feature samples to obtain the target acoustic model.

5. The method according to claim 4, characterized in that The initial acoustic model is trained using the language feature samples and the style feature samples to obtain the target acoustic model, including: Integrating the language feature sample and the style feature sample to obtain an integrated feature sample; The initial acoustic model is trained using the integrated feature samples to obtain the target acoustic model.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtaining a joint loss model, wherein the joint loss model is used to indicate a target that needs to be optimized during the training of the initial acoustic model; The initial acoustic model is trained using the joint loss model and the basic data samples.

7. An audio synthesis device, characterized in that: include: an acquisition unit, configured to acquire basic data of the audio to be synthesized, wherein the basic data is used to indicate conditions required to synthesize the audio to be synthesized, and the basic data includes at least but is not limited to: the text to be synthesized, identification information of the target speaker, style, and target language; an analysis unit, configured to input the basic data into a target acoustic model for analysis to obtain spectrum data, wherein the target acoustic model is trained using basic data samples and spectrum data samples, the basic data samples are used to indicate conditions that must be met for synthesizing historical audio to be synthesized, and the spectrum data samples are used to indicate audio features of the historical audio to be synthesized; a synthesis unit, configured to synthesize the basic data into target audio based on the spectrum data; The analysis unit is further configured to perform the following steps: constructing an initial acoustic model, the initial acoustic model comprising an encoder, a variable predictor, and a decoder, the variable predictor comprising a language style composite representation predictor module, a fundamental frequency predictor, and a duration predictor, the language style composite representation predictor module being a global style marker based on an attention mechanism; training the initial acoustic model to obtain the target acoustic model; The device is also used to perform the following steps: splicing the language representation with the phoneme representation of the same dimension, inputting the splicing result into the encoder to model the context representation; adding the context representation with the language style fusion feature, inputting the addition result into the variable predictor to obtain a language style fusion feature vector, wherein the language style fusion feature is obtained by mapping the spliced ​​phoneme-level style representation and the language representation; inputting random noise, speaker representation, and discrete speech representation into the decoder, and obtaining the output of the decoder as a Mel spectrum.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A processor, characterized in that: The processor is configured to run a program, wherein the program executes the method according to any one of claims 1 to 6 when running.

10. A computer program product, characterized in that The computer program product comprises computer instructions, which implement the method according to any one of claims 1 to 6 when executed by a processor.

Images