Audio synthesis method and apparatus, audio synthesis model training method and apparatus, electronic device, computer readable storage medium, and computer program product

Abstract

An audio synthesis method and apparatus, an audio synthesis model training method and apparatus, an electronic device, a computer readable storage medium, and a computer program product, relating to artificial intelligence technology. The audio synthesis method comprises: on the basis of language information of a target text and preset style information, calling an audio synthesis model to execute the following processing, wherein the audio synthesis model comprises a prior encoder and a waveform decoder: by means of the prior encoder and on the basis of the language information and the preset style information, generating audio features corresponding to the target text; by means of the prior encoder, performing normalizing flow processing on the audio features to obtain hidden variables of the target text; and by means of the waveform decoder, performing waveform decoding processing on the hidden variables of the target text to obtain a waveform that conforms to an audio style described by the preset style information and corresponds to the target text.

Description

Audio synthesis method, audio synthesis model training method, device, electronic device, computer-readable storage medium and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The embodiments of this application are based on the Chinese patent application with application number 202311307828.7 and application date October 10, 2023, and claim the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into the embodiments of this application as a reference. Technical Field

[0003] The present application relates to artificial intelligence technology, and in particular to an audio synthesis method, a training method for an audio synthesis model, a device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0004] Audio synthesis is one of the important applications in the field of artificial intelligence, which can generate artificial audio through mechanical and electronic methods.

[0005] The audio synthesis method in related technologies is relatively rough. Usually, the mel-spectrogram of the text data is first predicted through an acoustic model, and then an audio waveform is generated by combining the mel-spectrogram with the vocoder to synthesize the mel-spectrogram corresponding to the text data to obtain the audio data corresponding to the text data. This synthesis method cannot achieve accurate audio synthesis and is prone to artifacts in the synthesized audio, thereby affecting the user experience of normal audio synthesis.

[0006] Summary of the Invention

[0007] Embodiments of the present application provide an audio synthesis method, an audio synthesis model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the quality and expressiveness of audio synthesis.

[0008] The technical solution of the embodiment of the present application is implemented as follows:

[0009] The present invention provides an audio synthesis method for electronic devices, including:

[0010] Based on the language information and preset style information of the target text, the audio synthesis model is called to perform the following processing, wherein the audio synthesis model includes a priori encoder and a waveform decoder:

[0011] Generating, by the a priori encoder, audio features corresponding to the target text based on the language information and the preset style information;

[0012] Performing normalized stream processing on the audio features through the prior encoder to obtain hidden variables of the target text;

[0013] The waveform decoder performs waveform decoding processing on the hidden variables of the target text to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

[0014] An embodiment of the present application provides a training method for an audio synthesis model, which is applied to an electronic device. The audio synthesis model includes a priori encoder, a posteriori encoder, a waveform decoder, and a discriminator. The method includes:

[0015] Obtaining sample language information, sample style information, waveform annotation of the text sample, and a spectrum corresponding to the waveform annotation;

[0016] The posterior encoder is used to encode the spectrogram corresponding to the waveform annotation and the sample style information to obtain hidden variables of the spectrogram;

[0017] Performing waveform decoding processing on the hidden variables of the spectrogram by the waveform decoder to obtain a synthetic waveform that conforms to the audio style described by the sample style information and corresponds to the text sample;

[0018] Performing identification processing on the synthesized waveform by the discriminator to obtain an identification result;

[0019] Encoding the sample language information and the sample style information through the prior encoder to obtain hidden variables of the text sample;

[0020] The audio synthesis model is trained based on the hidden variables of the text sample, the hidden variables of the spectrogram, and the identification result to obtain a trained audio synthesis model.

[0021] The present invention provides an audio synthesis device, including:

[0022] The calling module is configured to call the audio synthesis model to perform the following processing based on the language information and preset style information of the target text, wherein the audio synthesis model includes a priori encoder and a waveform decoder:

[0023] A generating module configured to generate audio features corresponding to the target text based on the language information and the preset style information through the priori encoder;

[0024] a normalization module configured to perform normalized stream processing on the audio features through the prior encoder to obtain hidden variables of the target text;

[0025] The waveform decoding module is configured to perform waveform decoding processing on the hidden variables of the target text through the waveform decoder to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

[0026] The present embodiment provides a training device for an audio synthesis model, wherein the audio synthesis model includes a priori encoder, a posteriori encoder, a waveform decoder, and a discriminator; the device includes:

[0027] an acquisition module configured to acquire sample language information, sample style information, waveform annotations of the text sample, and a spectrum corresponding to the waveform annotations;

[0028] a posterior module configured to encode the spectrogram corresponding to the waveform annotation and the sample style information through the posterior encoder to obtain hidden variables of the spectrogram;

[0029] a decoding module configured to perform waveform decoding processing on the hidden variables of the spectrogram through the waveform decoder to obtain a synthetic waveform that conforms to the audio style described by the sample style information and corresponds to the text sample;

[0030] an identification module configured to perform identification processing on the synthesized waveform and the waveform annotation through the identifier to obtain an identification result;

[0031] A priori module, configured to encode the sample language information and the sample style information through the priori encoder to obtain hidden variables of the text sample;

[0032] The training module is configured to train the audio synthesis model based on the hidden variables of the text sample, the hidden variables of the spectrogram and the identification result to obtain a trained audio synthesis model.

[0033] An embodiment of the present application provides an electronic device, comprising:

[0034] Memory for storing computer programs or computer-executable instructions;

[0035] The processor is used to implement the audio synthesis method or audio synthesis model training method provided in the embodiment of the present application when executing the computer program or computer executable instructions stored in the memory.

[0036] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the audio synthesis method or the audio synthesis model training method provided in the embodiment of the present application is implemented.

[0037] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the audio synthesis method or the audio synthesis model training method provided in the embodiment of the present application is implemented.

[0038] The embodiments of the present application have the following beneficial effects:

[0039] The audio features corresponding to the target text are generated through prior encoding, and the audio features are normalized and stream processed to obtain the hidden variables of the target text. Then, the hidden variables of the target text are waveform decoded through a waveform decoder to directly generate a synthetic waveform. Compared with related technologies, there is no need to use a vocoder combined with a Mel-spectrogram as an intermediate product to generate a synthetic waveform, thus avoiding the generation of artifacts. Accurate audio generation can be achieved based on accurate hidden variables, thereby improving the quality and expressiveness of audio synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG1 is a schematic diagram of an application scenario of an audio synthesis system provided in an embodiment of the present application;

[0041] 2A-2B are schematic structural diagrams of electronic devices provided in embodiments of the present application;

[0042] 3A-3F are schematic flow diagrams of an audio synthesis method provided in an embodiment of the present application;

[0043] 4A-4B are flowcharts of a method for training an audio synthesis model according to an embodiment of the present application;

[0044] FIG5 is a schematic diagram of an audio synthesis model provided in an embodiment of the present application;

[0045] FIG6 is a schematic diagram of a speech encoder provided in an embodiment of the present application;

[0046] FIG7 is a schematic diagram of a frame-level encoder provided in an embodiment of the present application;

[0047] FIG8 is a schematic diagram of a posteriori encoder provided in an embodiment of the present application;

[0048] FIG9 is an example of pitch contours of different DurIAN models provided in an embodiment of the present application;

[0049] FIG10 is a schematic diagram of spectra of different TTS models provided in the embodiments of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] In the following description, the terms "first\second" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0052] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0054] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed can be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0058] 2) Convolutional Neural Networks (CNN): A type of feedforward neural network (FNN) with a deep structure that incorporates convolutional computations, CNNs are a representative algorithm for deep learning. CNNs possess representation learning capabilities and can perform shift-invariant classification of input images based on their hierarchical structure.

[0059] 3) Recurrent Neural Network (RNN): A type of recursive neural network that takes sequence data as input and performs recursion in the direction of the sequence's evolution, with all nodes (recurrent units) connected in a chain-like fashion. RNNs have memory, parameter sharing, and Turing completeness, making them particularly advantageous for learning nonlinear features of sequences.

[0060] 4) Phoneme: The smallest unit of audio. Phonemes are the basis for humans to distinguish one word from another. Phonemes form syllables, which in turn form different words and phrases.

[0061] 5) Hidden state: The decoder (e.g., hidden Markov model) outputs a sequence used to represent spectral data. By smoothing the hidden state, the corresponding spectral data can be obtained. Since the audio signal is a non-stationary signal in a long time period (e.g., more than one second), it can be approximated as a stationary signal in a short time period (e.g., 50 milliseconds). The characteristic of a stationary signal is that the spectral distribution of the signal is stable, and the spectral distribution in different time periods is similar. The hidden Markov model classifies the continuous signal corresponding to a small segment of similar spectrum into a hidden state. The hidden state is the state actually implied in the Markov model, and it is a sequence used to represent spectral data that cannot be obtained through direct observation. The training process of the hidden Markov model is to maximize the likelihood. The data generated by each hidden state is represented by a probability distribution. Only when similar continuous signals are classified as the same state can the likelihood be as large as possible.

[0062] 6) Timbre: Different audio waves always exhibit distinct characteristics in their waveforms, just as different objects vibrate with different characteristics. Different sound-producing bodies, due to their materials and structures, produce different timbres. For example, the sounds produced by a piano, violin, and human are all different, and the audio produced by each person is also unique. Timbre can be understood as the characteristic of an audio sound.

[0063] 7) Target object: a real object in the real world or a virtual object in a virtual scene, such as a real user, a virtual character, a virtual animal, an anime character, etc.

[0064] 8) Text-to-Speech (TTS) Technology: Leveraging the latest advances in linguistics and psychology, and powered by a built-in chip and neural network design, TTS intelligently converts text into natural speech. TTS technology converts text files in real time, in seconds. Its unique intelligent voice controller delivers smooth, natural audio, unlike the cold, awkward nature of machine-generated speech.

[0065] 9) Loss value: This value represents the difference between the model's output and the actual result (or labeled result). Model training primarily involves two processes: forward propagation and back propagation. For example, in a neural network model consisting of an input layer, hidden layers, and an output layer, forward propagation sequentially processes the input, hidden, and output layers to obtain the output result. Back propagation, on the other hand, propagates the calculated loss value to the output, hidden, and input layers, updating the weight parameters in each layer.

[0066] Embodiments of the present application provide an audio synthesis method, an audio synthesis model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of audio synthesis.

[0067] The audio synthesis method and the audio synthesis model training method provided in the embodiments of the present application can be implemented using artificial intelligence (AI) technology. The audio synthesis method provided in the embodiments of the present application can be implemented by the terminal alone; it can also be implemented collaboratively by the terminal and the server. For example, the terminal alone undertakes the audio synthesis method or the audio synthesis model training method described below, or the terminal sends a request for audio generation (including language information of the target text and preset style information) to the server, and the server executes the audio synthesis method according to the received synthesis request for audio, and generates a synthesized waveform in response to the generation request for audio based on the language information of the target text and the preset style information, thereby realizing intelligent and accurate generation of audio.

[0068] The following describes exemplary applications of electronic devices provided by embodiments of the present application. The electronic devices provided by embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, in-vehicle devices), smart phones, smart speakers, smart watches, smart TVs, in-vehicle terminals, etc. In some embodiments, the server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The cloud service can be an audio synthesis service for the terminal to call.

[0069] Next, an exemplary application when the electronic device is implemented as a server will be described.

[0070] Taking servers as an example, it can be a server cluster deployed in the cloud, opening artificial intelligence cloud services (AI as a Service, AIaaS) to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI theme mall. All users can access and use one or more artificial intelligence services provided by the AIaaS platform through the application programming interface.

[0071] For example, one AI cloud service may be an audio synthesis service, where a cloud server encapsulates an audio synthesis program provided by an embodiment of the present application. A user invokes the audio synthesis service in the cloud service through a terminal (running a client, such as an audio client or an in-car client), causing the cloud server to invoke the encapsulated audio synthesis program and generate a synthesized waveform based on the language information of the target text and preset style information, thereby achieving intelligent and accurate audio generation.

[0072] As an application example, for an in-car client, while a user is driving, it is not convenient to obtain information via text. However, they can obtain information by reading audio, thus avoiding missing important information. For example, while a user is driving, their leader sends a text message about an important meeting, which the user needs to read and process promptly. After receiving the text, the in-car client needs to convert it into audio and play it to the user. Based on the leader's style information and the target text, the in-car client performs synthesis processing to obtain a synthesized waveform that matches the leader's style and corresponds to the target text. The generated audio is then played to the user, allowing the user to read the leader's audio in a timely manner.

[0073] Refer to Figure 1, which is a schematic diagram of an application scenario of an audio synthesis system 10 provided in an embodiment of the present application. The terminal 200 is connected to the server 100 through a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0074] Terminal 200 (running a client, such as an audio client, a car client, etc.) can be used to obtain a synthesis request for audio. For example, the user inputs the language information and preset style information of the target text through terminal 200, and the terminal 200 automatically obtains the language information and preset style information of the target text, and automatically generates a generation request for audio.

[0075] In some embodiments, an audio synthesis plug-in may be embedded in a client running in a terminal to implement an audio synthesis method and an audio synthesis model training method locally on the client. For example, after the terminal 200 obtains a synthesis request for audio (including language information and preset style information of a target text), it calls a training plug-in of an audio synthesis model to implement the training method of the audio synthesis model. First, a posterior encoder is used to encode the spectrogram corresponding to the waveform annotation and the sample style information to obtain hidden variables of the spectrogram; a waveform decoder is used to perform waveform decoding on the hidden variables of the spectrogram to obtain a synthesized waveform that conforms to the audio style described by the sample style information and corresponds to the text sample; a discriminator is used to perform identification processing on the synthesized waveform to obtain an identification result; a priori encoder is used to encode the sample language information and the sample style information to obtain hidden variables of the text sample; based on the hidden variables of the text sample, the hidden variables of the spectrogram, and the identification result, the audio synthesis model is trained to obtain a trained audio synthesis model. Then, the terminal 200 calls the audio synthesis plug-in based on the synthesis request for the audio, and calls the trained audio synthesis model based on the language information of the target text and the preset style information, wherein the audio synthesis model includes a priori encoder and a waveform decoder; through the priori encoder, the audio features corresponding to the target text are generated based on the language information and the preset style information; through the priori encoder, the audio features are normalized stream processed to obtain the hidden variables of the target text; through the waveform decoder, the hidden variables of the target text are waveform decoded to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text, in response to the synthesis request for the audio, thereby realizing intelligent and accurate generation of audio.

[0076] In some embodiments, after the terminal 200 obtains a synthesis request for audio, it calls the audio synthesis interface of the server 100 (which can be provided in the form of a cloud service, i.e., an audio generation service), and the server 100 calls the training plug-in of the audio synthesis model to implement the training method of the audio synthesis model. First, the posterior encoder is used to encode the spectrogram corresponding to the waveform annotation and the sample style information to obtain the hidden variables of the spectrogram; the waveform decoder is used to perform waveform decoding on the hidden variables of the spectrogram to obtain a synthetic waveform that conforms to the audio style described by the sample style information and corresponds to the text sample; the discriminator is used to perform identification on the synthetic waveform to obtain an identification result; the a priori encoder is used to encode the sample language information and the sample style information to obtain the hidden variables of the text sample; based on the hidden variables of the text sample, the hidden variables of the spectrogram and the identification result, the audio synthesis model is trained to obtain a trained audio synthesis model. Then, the server 100 calls the audio synthesis plug-in to call the trained audio synthesis model based on the language information and preset style information of the target text, wherein the audio synthesis model includes a priori encoder and a waveform decoder; through the priori encoder, the audio features corresponding to the target text are generated based on the language information and preset style information; through the priori encoder, the audio features are normalized stream processed to obtain the hidden variables of the target text; through the waveform decoder, the hidden variables of the target text are waveform decoded to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text, and the synthetic waveform is sent to the terminal 200 to respond to the synthesis request for the audio, thereby realizing intelligent and accurate generation of audio.

[0077] In some embodiments, the terminal or server can implement the audio synthesis method or the training method of the audio synthesis model provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, the computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as an audio application or an instant messaging application; it can also be a small program embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0078] The structure of the electronic device provided in an embodiment of the present application is described below. Referring to FIG. 2A , FIG. 2A is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application. Taking the electronic device 500 as a terminal as an example, the electronic device 500 for audio synthesis shown in FIG. 2A includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It will be understood that the bus system 540 is used to enable connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the various buses are labeled as the bus system 540 in FIG. 2A.

[0079] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0080] The memory 550 includes a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550 may optionally include one or more storage devices physically remote from the processor 510.

[0081] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0082] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0083] A network communication module 552 for reaching other electronic devices via one or more (wired or wireless) network interfaces 520 . Exemplary network interfaces 520 include Bluetooth, WiFi, and USB.

[0084] In some embodiments, the audio synthesis device provided in the embodiments of the present application can be implemented in software. The audio synthesis device provided in the embodiments of the present application can be provided as various software embodiments, including various forms including applications, software, software modules, scripts or codes.

[0085] Figure 2A shows an audio synthesis device 555 stored in the memory 550, which can be software in the form of a program and plug-in, and includes a series of modules, including a calling module 5551, a generation module 5552, a normalization module 5553 and a waveform decoding module 5554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0086] In some embodiments, the training device of the audio synthesis model provided in the embodiments of the present application can be implemented in software. Figure 2B shows the training device 556 of the audio synthesis model stored in the memory 550, which can be software in the form of programs and plug-ins, including the following software modules: acquisition module 5561, posterior module 5562, decoding module 5563, identification module 5564, priori module 5565 and training module 5566. These modules are logical and can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below. It is worth noting that, except for the training device 556 of the audio synthesis model shown in Figure 2B, the rest of the structure can be the same as Figure 2A.

[0087] It should be noted that the audio synthesis device 555 and the audio synthesis model training device 556 can be integrated on one electronic device, that is, the electronic device can simultaneously realize model training and model application; the audio synthesis device 555 and the audio synthesis model training device 556 can be respectively integrated on two electronic devices, that is, the two electronic devices respectively realize model training and model application.

[0088] The following describes the audio synthesis method provided by an embodiment of the present application. As previously mentioned, the audio synthesis method provided by an embodiment of the present application can be implemented by various types of electronic devices, such as a terminal, a server, or a combination of the two. Therefore, the execution entities of each step will not be repeated below. Referring to Figure 3A, Figure 3A is a schematic flow chart of the audio synthesis method provided by an embodiment of the present application, which will be described in conjunction with the steps shown in Figure 3A.

[0089] In step 101, based on the language information and preset style information of the target text, an audio synthesis model is called to perform the following processing, wherein the audio synthesis model includes a priori encoder and a waveform decoder.

[0090] As an example of obtaining the language information and preset style information of the target text, the user inputs the language information and preset style information of the target text through the terminal, and the terminal automatically generates a generation request for audio, and sends the generation request for audio to the server. The server parses the generation request for audio, obtains the language information and preset style information of the target text, and based on the language information and preset style information of the target text, calls the audio synthesis model, and executes the audio synthesis method through the audio synthesis model to perform audio synthesis on the language information and preset style information of the target text.

[0091] The target text's linguistic information is used to express its content, or the content of the synthesized audio. It's important to note that the target text's content information represents the specific information and meaning it aims to convey. Content information is abstract and independent of linguistic form, and the target text serves as its carrier. In audio synthesis, this content information must be accurately understood and converted to ensure the meaning of the audio output is consistent with the target text. For example, the target text's content information, "The weather is great today," conveys a positive assessment of the weather conditions on a particular date. The target text's linguistic information includes linguistic features such as the target text's language, pronunciation, intonation, and rhythm. In audio synthesis, this content information is converted into audio using these linguistic features, including pronunciation, intonation, rhythm, and emotion. For example, the target text "The weather is great today" conveys different linguistic information in different languages. If output in Mandarin, the linguistic information includes the phonemes "jin tian tian qi hen hao."

[0092] The language information can be the language text of the target text, or it can be the language vector (also known as language embedding) obtained by mapping each word in the language text to each word vector. The language vector is the vector form of the language text, and the language vector and the language text express the same content.

[0093] Preset style information is preset style information used to describe the style of the audio to be synthesized. In the field of audio synthesis, preset style information refers to parameters or features used to define the audio characteristics of the synthesized audio during the audio synthesis process. Preset style information reflects non-semantic characteristics of the speaker outputting the synthesized audio, such as personality, emotion, speaking rate, and intonation, enabling the audio synthesis system to generate audio with specific emotional overtones, language styles, or speaker characteristics. Preset style information includes, but is not limited to, emotional style, speaker style, language style, speaking rate, intonation, and volume. Emotional style refers to the emotional overtones expressed in the audio, such as happiness, sadness, anger, and surprise; speaker style refers to the speaker's personality traits reflected in the audio, such as the thickness of the voice and the rhythm of speech; language style refers to the speaker's linguistic habits reflected in the audio, such as the use of dialects, jargon, and professional terminology; speaking rate refers to the speed of the speaker's expression, which affects the fluency and naturalness of the audio; intonation refers to the pattern of pitch changes in the audio, reflecting the structure and intention of the sentence; and volume refers to the loudness or volume of the audio, reflecting the speaker's confidence or emphasis. In practical applications, audio synthesis systems mimic specific stylistic information through parameter adjustments, allowing users to synthesize audio in different styles as needed. For example, synthesizing speech for a news broadcast might require a formal, objective, and calm tone, while synthesizing speech for a storybook reading might require a more expressive, lively, and emotional style. By precisely controlling the parameters or features included in the stylistic information, the system can more richly and flexibly meet the needs of various application scenarios.

[0094] The preset style information can be the style text to be synthesized, or it can be the style vector (also known as style embedding) obtained by mapping each word in the style text to its own word vector. The style vector is the vector form of the style text, and the style vector and the style text express the same content. The preset style information is used to describe the audio style of the synthesized audio after the target text is synthesized into audio. The audio style can be happy, sad, angry, excited, annoying, surprised, suspicious, cunning, solemn, enchanting, sarcastic, and so on.

[0095] In step 102, an a priori encoder is used to generate audio features corresponding to the target text based on language information and preset style information.

[0096] Among them, the audio features are used to characterize the features related to the audio of the generated audio frame. The process of generating audio features is based on the prior encoder to perform audio frame-level calculations on the language information and preset style information to obtain the audio features corresponding to the target text. In the field of audio synthesis, the prior encoder refers to a model component used to learn the prior knowledge of audio frames, which aims to capture the statistical characteristics of the audio frames or the implied semantic information, that is, the audio features. The embodiments of the present application are not limited to the structural form of the prior encoder. For example, when the prior encoder is a variational autoencoder (VAE), during audio synthesis, the variational autoencoder is used to learn the potential representation of the input audio features to capture the statistical distribution of the audio features, so that when generating new audio, the audio synthesis model can effectively use statistical information to generate more natural and higher quality audio.

[0097] In practical applications, a priori encoders help improve the naturalness, fluency, and expressiveness of audio synthesis. For example, in scenarios like developing voice assistants, automated dubbing, and virtual character speech, the prior knowledge of audio captured by a priori encoder is crucial for generating audio that sounds more natural and more in line with a specific style. A priori encoders help models understand and utilize the inherent patterns and prior knowledge of audio signals, thereby generating high-quality, natural, and smooth audio.

[0098] Refer to Figure 3B, which is a flowchart of the audio synthesis method provided in an embodiment of the present application. Figure 3B shows that step 102 in Figure 3A can be implemented through steps 1021 to 1023.

[0099] In step 1021, the a priori encoder performs the following processing: performing phoneme-level encoding processing on the language information to obtain text features of the phonemes in the language information.

[0100] Here, the language information of the target text consists of multiple phonemes. Phonemes are the smallest units of pronunciation defined by the natural properties of speech. They represent the basic audio units in audio signals and are analyzed based on the pronunciation movements within a syllable. Each movement constitutes a phoneme. For example, the word "ma" consists of two pronunciation movements, "m" and "a," meaning that "ma" comprises two phonemes. In the field of audio synthesis, phoneme-level encoding is used to encode language information according to phonemes to obtain textual features of the phonemes within the language information.

[0101] When the language information is a language text, each word in the language text can be mapped to a word vector to obtain a language vector. Then, the language vector is encoded at the phoneme level to obtain the text features of the phonemes in the language information (also known as phoneme sequence features). When the language information is a language vector, the language vector is encoded at the phoneme level to obtain the text features of the phonemes in the language information (also known as phoneme sequence features). The text features of the phonemes are used to represent the semantics of the phonemes.

[0102] In the embodiments of this application, phoneme-level encoding processing is crucial in audio synthesis. This allows the audio synthesis model to finely control each pronunciation unit of the audio, thereby generating more natural and accurate audio. Furthermore, phoneme-level processing can help the audio synthesis model better capture the specific manifestations of phonemes in different languages ​​and dialects, improving the diversity and adaptability of audio synthesis.

[0103] Refer to Figure 3C, which is a flow chart of the audio synthesis method provided in an embodiment of the present application. The priori encoder includes a language encoder. Figure 3C shows that step 1021 in Figure 3B can be implemented through steps 10211-10212.

[0104] In step 10211, the language information is acoustically encoded by a language encoder to obtain language features of the target text, wherein the language features include text features of multiple phonemes and text features of multiple prosody.

[0105] Here, the language text includes multiple phonemes and multiple prosody, with prosody being used to describe the pronunciation of the phonemes. Prosody includes at least one of pitch, volume, and rhythm. Pitch characterizes the degree and frequency of pitch changes in audio; for example, pitch includes rising, falling, and flat pitches; volume characterizes the intensity of audio; and rhythm characterizes the pause intervals in audio; for example, rhythm includes long pauses, short pauses, and no pauses. Since the language text includes multiple phonemes and multiple prosody, after obtaining a language vector based on the language text, the language encoder in the a priori encoder 501 shown in FIG5 can be used to acoustically encode the language vector, obtaining language features including text features of each phoneme and text features of each prosody. The text features of prosody are used to characterize the semantics of prosody. In the field of audio synthesis, acoustic encoding is the process of converting abstract phonetic units such as phonemes or prosody into acoustic features (i.e., text features of phonemes and text features of prosody). These acoustic features are the physical manifestations of the audio signal and determine the auditory properties of the audio, such as fundamental frequency (F0), intensity, timbre, and duration. The goal of acoustic coding is to ultimately convert these text features into a continuous audio waveform through a vocoder, thereby generating actual synthesized audio.

[0106] The embodiments of the present application are not limited to the structure, parameters, size, etc. of the language encoder. Exemplarily, the language encoder may be an autoencoder, a variational autoencoder, etc. For example, the language encoder may include at least one of a convolutional layer, a pooling layer, an attention layer, a self-attention layer, a multi-head self-attention layer, a batch normalization layer, a layer normalization layer, etc. For example, the language encoder includes an embedding layer and a convolutional layer. After the language text is input into the language encoder, the following acoustic encoding method is performed: the embedding layer of the language encoder maps each phoneme and each prosody in the language text to a corresponding vector to obtain a language vector, and the language vector is convolved by the convolutional layer to obtain the language features of the target text.

[0107] In some embodiments, the language encoder includes a language encoding block, which can be used to perform acoustic encoding processing on the language information to extract the language features of the target text. Therefore, step 10211 can be implemented as follows: using a language encoding block, acoustic encoding processing is performed on the language information to obtain the language features of the target text.

[0108] In some embodiments, as shown in FIG6 , the language encoder includes multiple cascaded language encoding blocks. The cascaded language encoding blocks can be used to acoustically encode the language information, thereby accurately extracting the language features of the target text through a progressive structure. Therefore, step 10211 can be implemented as follows: acoustically encode the language information using the first language encoding block in the multiple cascaded language encoding blocks; output the encoding result of the first language encoding block to the subsequent cascaded language encoding blocks, and continue acoustically encoding and outputting the encoding results through the subsequent cascaded language encoding blocks until the last language encoding block is output; and the encoding result output from the last language encoding block is used as the language feature of the target text.

[0109] The processing process of each language coding block is similar, with only the processing objects being different. The processing object of the first language coding block is language information, and the processing objects of other language coding blocks are the outputs of the previous language coding blocks of other language coding blocks. Among them, the other language coding blocks are the language coding blocks in the cascade except the first language coding block.

[0110] As an example, as shown in Figure 6, each language encoding block includes an attention layer, an activation network layer, a discarding and adding layer, and a layer normalization layer. The following description takes the first language encoding block as an example. See Figure 3D, which is a flow chart of the audio synthesis method provided in an embodiment of the present application. Figure 3D shows that "acoustic encoding processing of language information is performed through the first language encoding block in multiple cascaded language encoding blocks", which can be achieved by following the steps 102111-102114.

[0111] In step 102111, the attention layer included in the first language coding block is used to perform attention processing on the language information to obtain the first attention feature of the first language coding block.

[0112] In an embodiment of the present application, the speech encoder includes a network layer based on an attention mechanism. The network layer based on the attention mechanism can be an attention layer such as a self-attention layer or a multi-head self-attention layer. When the language information is a language vector, the language vector includes a vector for each phoneme and a vector for each prosody. The first attention feature includes an attention feature for the phoneme and an attention feature for the prosody.

[0113] It should be noted that, since the processing methods of phonemes and rhythms are similar, for the sake of ease of description, a phoneme or a rhythm is called an acoustic element, that is, the language vector includes vectors of multiple acoustic elements. For any acoustic element, the vector of the acoustic element can be subjected to attention processing based on the attention mechanism to obtain the attention feature of the acoustic element. Among them, the attention processing is to calculate the attention of the acoustic elements in the language information based on the attention mechanism to obtain the first attention feature. The embodiments of the present application are not limited to the form of attention calculation, such as self-attention calculation, multi-head self-attention calculation, etc.

[0114] In step 102112, the first attention feature of the first language coding block is mapped through the activation network layer included in the first language coding block to obtain the mapping feature of the first language coding block.

[0115] Here, the first attention feature includes attention features of multiple acoustic elements, and the acoustic elements are phonemes or melodies. The following mapping processing is performed through the activation network layer included in the first language coding block: the relevant features of any acoustic element and the attention features of any acoustic element are mapped through the activation function and the linear function to obtain the mapping features of any acoustic element, and the mapping features of all acoustic elements are used as the mapping features of the first language coding block. Among them, when any acoustic element is the first acoustic element (that is, the first acoustic element), the relevant features of any acoustic element are set features, and when any acoustic element is not the first acoustic element (that is, an acoustic element other than the first acoustic element), the relevant features of any acoustic element are obtained based on the set features and the attention features of each acoustic element located before any acoustic element.

[0116] Among them, if any acoustic element is the first acoustic element among multiple acoustic elements, the configured set features can be obtained and the set features can be used as the relevant features of the first acoustic element. For example, the configured set features are a zero matrix, and can also be features obtained by random sampling according to a statistical distribution function, wherein the statistical distribution function includes but is not limited to a normal distribution function, a binomial distribution function, a Bernoulli distribution function, etc. If any acoustic element is not the first acoustic element among multiple acoustic elements, the relevant features of any acoustic element can be determined based on the set features and the attention features of each acoustic element located before any acoustic element. For example, if any acoustic element is the third acoustic element among multiple acoustic elements, the relevant features of the third acoustic element can be determined based on the relevant features of the second acoustic element, the first feature of the first acoustic element, and the first feature of the second acoustic element.

[0117] In an embodiment of the present application, the language encoder includes an activation network layer, and the activation network layer can use an activation function and a linear function to map the relevant features of any acoustic element and the attention features of the acoustic element to obtain the mapping features of the acoustic element. Among them, activation can be performed according to at least one activation function. The embodiment of the present application does not limit the form of the activation function. For example, the activation function includes but is not limited to the S-type (Sigmoid) activation function, the Swish activation function (combining the Sigmoid function and the linear function), the rectified linear unit (ReLU, Rectified Linear Unit) activation function, etc. Of course, linear transformation is performed according to at least one linear function.

[0118] First, the activation network layer performs two linear transformations on the attention features of each acoustic element, as shown in formula (1).

[0119] Among them, X represents the attention feature of each acoustic element, and the attention feature of the t-th acoustic element is X t W1 represents the weight term for the first linear transformation. W2 represents the weight term for the second linear transformation. W1 and W2 are model parameters that need to be adjusted during training, i.e., learning parameters. Includes the characteristics of each acoustic element after the first linear transformation. Includes the features of each acoustic element after the second linear transformation. l represents the number of acoustic elements.

[0120] Then, the activation network layer performs the first activation based on the relevant features of any acoustic element and the features of any acoustic element after the first linear transformation, and determines the relevant features of the next acoustic element, as shown in the following formula (2). c[i]=Swish(c[i-1]-x1[i])+x1[i] (2)

[0121] Wherein, c[i] represents the feature obtained by first activating the feature x1[i] after the first linear transformation of the activation network layer based on the relevant feature c[i-1] of the i-th acoustic element and the i-th acoustic element. This feature can be used as the relevant feature of the i+1-th acoustic element. Swish() represents the element-by-element Swish activation function. In formula (2), the first activation is the Swish activation. It can be understood that in actual applications, the first activation can also be other types of activation, for example, the first activation is the Softmax activation. Swish(x) = sigmoid(α·x+β)·x, where α and β are the two parameters of the Swish activation function and x is the independent variable. Swish(c[i-1]-x1[i]) is equivalent to selecting the larger value between c[i-1] and x1[i] for pooling operation, so that when x1[i]>>c[i-1], c[i]=x1[i], and when x1[i]<<c[i-1], c[i]=c[i-1]. According to formula (2), the relevant features of each acoustic element can be determined

[0122] Next, the activation network layer performs a second activation based on the features of any acoustic element after the second linear transformation, and performs a linear transformation on the features obtained after the second activation and the features obtained after the first activation to obtain the mapping features of any acoustic element, as shown in formula (3). c )⊙σ(X2+b σ ))+b3 (3)

[0123] Among them, H represents the mapping feature of each acoustic element, and the mapping feature of the t-th acoustic element is ht. W3 represents the weight term during linear transformation. C includes the features obtained after the first activation of each acoustic element, which is equivalent to b c Represents the bias term corresponding to C. ⊙ represents element-by-element multiplication. σ() represents the sigmoid gate activation function. In formula (3), the second activation is the sigmoid activation. It can be understood that in practical applications, the second activation can also be other types of activation, for example, the second activation is the Softmax activation. X2 includes the features obtained by each acoustic element after the second activation, which is equivalent to bσ represents the bias term corresponding to X2. b3 represents the bias term during linear transformation.

[0124] For example, when the first activation in the activation network layer is a Swish activation and the second activation is a sigmoid activation, the activation network layer is also called a Swish Recurrent Neural Network (SwishRNN). SwishRNN includes a multiplication-gated recurrent unit that uses two matrix multiplication operations and a simple sequential pooling operation. Because the activation network layer has a simple structure, mainly including activation and linear transformation, feature mapping through the activation network layer can speed up encoding and improve the efficiency of audio feature generation.

[0125] In step 102113, the discarding and adding layer included in the first language coding block is used to perform discarding and adding processing on the mapping features of the first language coding block and the first attention features of the first language coding block to obtain the discarding and adding features of the first language coding block.

[0126] Here, through the dropout and add layer included in the first language encoding block, the mapping features and the attention features can be spliced ​​to obtain the spliced ​​features, and then the spliced ​​features are dropped and added (Dropout & Add) to obtain the dropout and add features of the first language encoding block.

[0127] The drop-and-add process is used to randomly drop and / or add the mapping features of the first language coding block and the first attention features of the first language coding block through a drop-and-add layer to obtain the drop-and-add features of the first language coding block. The drop-and-add process can also be used to randomly drop and / or add the concatenated features obtained by concatenating the mapping features of the first language coding block and the first attention features of the first language coding block through a drop-and-add layer to obtain the drop-and-add features of the first language coding block. Dropout is a regularization technique used to reduce overfitting when training a model. In the dropout process, features fed into the drop-and-add layer (usually with a certain probability) are randomly dropped (set to 0) during the forward propagation, thereby reducing the model's reliance on specific training samples and improving the model's generalization ability. Add is used to perform additional processing on the features fed into the drop-and-add layer by adding additional layers to the language coding block to increase the complexity of the model or introduce specific functionality. For example, additional fully connected layers, convolutional layers, or other types of layers can be added to the language coding block to process specific features or perform specific tasks.

[0128] This embodiment combines Dropout and Add. During model training, Dropout is first applied to reduce overfitting, and then the weights of certain layers are gradually increased during training to introduce more complexity. This approach allows the model to learn more complex features at specific training stages while maintaining the model's generalization ability, achieving better training results.

[0129] In step 102114, the discarded and added features of the first language coding block are normalized by the layer normalization layer included in the first language coding block to obtain the encoding result of the first language coding block.

[0130] Here, the discarded and added features of the first language coding block are normalized using the LayerNorm function through the layer normalization layer (LayerNorm) included in the first language coding block to obtain the encoding result of the first language coding block. Of course, the embodiments of the present application are not limited to the LayerNorm function, and other normalization functions can also be used, such as min-max normalization and max normalization. Min-max normalization scales the data to between 0 and 1, that is, new data = (original data - minimum value) / (maximum value - minimum value); max normalization scales the data to between 0 and a fixed value (usually 1). The normalization process is used to perform normalization calculations on the discarded and added features of the first language coding block, that is, to convert the discarded and added features of the first language coding block into a specific range to obtain a normalized result, which is determined as the encoding result of the first language coding block. The normalization process makes the encoding result more suitable for the next step of processing.

[0131] It should be noted that the processing process of each language coding block is similar, and only the processing objects are different. The processing object of the first language coding block is language information, and the processing objects of other language coding blocks are the outputs of the previous language coding blocks of other language coding blocks, wherein other language coding blocks are language coding blocks other than the first language coding block in the cascaded language coding blocks. Taking the subsequent cascaded language coding blocks of the first language coding block as an example, "continuing acoustic coding processing and outputting coding results through subsequent cascaded language coding blocks" can be achieved in the following ways: through the attention layer included in the k-th language coding block, attention processing is performed on the coding results output by the k-1-th language coding block to obtain the first attention feature of the k-th language coding block; through the activation network layer included in the k-th language coding block, mapping processing is performed on the first attention feature of the k-th language coding block to obtain the mapping feature of the k-th language coding block. The method comprises the steps of: performing a discarding and adding process on the mapping features of the k-th language coding block and the first attention features of the k-th language coding block through a discarding and adding layer included in the k-th language coding block, thereby obtaining the discarding and adding features of the k-th language coding block; performing a normalization process on the discarding and adding features of the k-th language coding block through a layer normalization layer included in the k-th language coding block, thereby obtaining the encoding result of the k-th language coding block, and outputting the encoding result of the k-th language coding block, wherein k is a positive integer that increases in sequence, 1<k<K, and K is the number of language coding blocks. The processing process of the k-th language coding block is similar to that of the first language coding block, except that the processing object is different.

[0132] In summary, the language encoder is a phoneme-level model whose input is a language vector derived from the conversion of speech text, which includes multiple phonemes and the rhythm between different phonemes. The language encoder includes a language encoding block, in which the activation network layer is the SwishRNN. Because SwishRNN is a recurrent architecture, it can model the temporal order of each phoneme and rhythm in a language sequence, improving the stability and representational power of language features. Furthermore, combining attention processing with the recurrent processing of SwishRNN can further improve the stability and representational power of language features.

[0133] In step 10212, text features of multiple phonemes are extracted from the language features.

[0134] Here, since language features include text features of multiple phonemes and text features of multiple prosody, the relative positional relationship between each phoneme and each prosody remains unchanged during the phoneme encoding process of language information. Therefore, when the text features of phonemes are needed, the text features of each phoneme can be extracted from the language features based on the position of each phoneme in the language text.

[0135] In some embodiments, step 10212 can extract the text features of the phonemes by hiding the text features of each rhythm included in the language features. For example, based on the position of each rhythm in the language text, the text features of each rhythm included in the language features are hidden. As shown in Figure 5, the priori encoder 501 also includes a feature hiding network (also known as a skip encoder (Skip-Encoder)), which hides the text features of each rhythm included in the language features through the feature hiding network, thereby discarding the text features of the rhythm and retaining the text features of the phonemes, and obtaining the phoneme sequence features including the text features of each phoneme. The embodiment of the present application does not limit the structure, parameters, size, etc. of the feature hiding network. For example, the feature hiding network includes a discarding layer, and the text features of the rhythm are hidden by the discarding layer.

[0136] Following step 1021 , in step 1022 , the acoustic features of the phonemes are determined based on the text features of the phonemes and the preset style information.

[0137] Here, the a priori encoder also includes a variation predictor. If the preset style information is a style text, the style text can be converted into a style vector. The text features and style vectors of multiple phonemes are input into the variation predictor, and the acoustic features of the multiple phonemes are determined by the variation predictor. The embodiments of the present application do not limit the structure, parameters, size, etc. of the variation predictor.

[0138] Among them, the acoustic characteristics of the phoneme are used to characterize the physical characteristics of the sound when the phoneme is read based on the audio style described by the preset style information. The physical characteristics can reflect the sound quality, timbre, etc. of the sound. The embodiment of the present application does not limit the acoustic characteristics of the phoneme. For example, the acoustic characteristics of the phoneme include at least one of the phoneme's vocalization duration, the phoneme's fundamental frequency, and the fundamental frequency range. The phoneme's vocalization duration can characterize the number of audio frames occupied by the phoneme when reading the phoneme. Based on this, the phoneme's vocalization duration can also be called the number of phoneme's vocalization frames. The fundamental frequency of the phoneme is used to characterize the basic frequency generated during the vocal cords emitting the phoneme. The basic frequency can be the lowest frequency. The fundamental frequency range is used to characterize the fluctuation amplitude of the fundamental frequency. By predicting the fundamental frequency range, the selection range of the fundamental frequency is expanded, the rhythm modeling ability is improved, and it is conducive to generating highly expressive audio.

[0139] Here, the input of the variable predictor may include other information in addition to the text features of multiple phonemes and the target style vector. For example, the input of the variable predictor may also include identification information of the target object. The identification information of the target object may be the identification text of the target object, or it may be the identification vector of the target object (also known as the identification embedding of the target object) obtained by mapping each word in the identification text of the target object to each word vector. The identification embedding of the target object is the vector form of the identification text of the target object, and expresses the same content as the identification text of the target object. Different target objects correspond to different identification information, that is, the identification information of the target object is unique.

[0140] Therefore, the acoustic features of each phoneme are determined through the identification vector of the target object, the target style vector and the text features of multiple phonemes, so that the acoustic features of the phoneme can characterize the physical characteristics of the sound when the target object reads the phoneme based on the audio style described by the preset style information, thereby achieving the distinction between different objects and different styles of sounds, which is conducive to generating personalized audio and improving the expressiveness of the audio.

[0141] Following the above step 1022, in step 1023, the acoustic features of the phonemes are encoded at the audio frame level to obtain audio features corresponding to the target text.

[0142] In an embodiment of the present application, the a priori encoder combines the text features of the phonemes and the preset style information to perform audio frame-level encoding processing on the acoustic features of the phonemes to obtain the audio features corresponding to the target text. Among them, the audio frame can be measured by duration, for example, one audio frame is 10 milliseconds or 25 milliseconds. Among them, in the field of audio synthesis, the audio frame-level encoding processing is used to encode the acoustic features with the audio frame as the basic unit to obtain the audio features corresponding to the target text. The audio features can be divided according to the duration corresponding to the audio frame to obtain audio frame data.

[0143] Among them, a phoneme occupies at least one audio frame (the basic unit of digital audio processing), and the audio features of the audio frame are used to characterize the audio style and content of the audio frame (that is, the audio style and content of a frame). The audio style of the audio frame is the audio style described by the preset style information, and the content of the audio frame is the phoneme.

[0144] See Figure 3E, which is a flow chart of the audio synthesis method provided in an embodiment of the present application. The prior encoder includes a frame-level encoder, and the acoustic features of the phoneme include the number of audio frames containing the phoneme (that is, the number of audio frames occupied by the phoneme). Figure 3E shows that step 1023 in Figure 3B can be implemented through steps 10231-10232.

[0145] In step 10231, initial features corresponding to the target text are determined based on the number of audio frames containing phonemes and the text features of each phoneme.

[0146] As an example, the acoustic features of a phoneme include the duration of the phoneme's pronunciation, and the duration of the phoneme's pronunciation is also the number of audio frames containing the phoneme. As shown in Figure 5, the priori encoder 501 may also include a feature expansion network. For any phoneme, the feature expansion network expands the text features of the phoneme based on the number of audio frames occupied by the phoneme to obtain the initial features corresponding to the target text, that is, the initial features of each audio frame occupied by the phoneme. For example, if a phoneme occupies 3 audio frames, the text features of the phoneme can be expanded into 3 parts through the feature expansion network, and the text features of one phoneme are the initial features of one audio frame occupied by the phoneme. In this case, the initial features of the audio frame are used to describe the semantics of the phoneme.

[0147] As an example, the acoustic features of a phoneme also include the phoneme's fundamental frequency and fundamental frequency range. The text features of any phoneme, the fundamental frequency of the phoneme, and the fundamental frequency range of the phoneme can be fused through a feature expansion network to obtain the fusion features of the phoneme, and based on the number of audio frames occupied by the phoneme, the fusion features of the phoneme are expanded to obtain the initial features of each audio frame occupied by the phoneme. For example, if a phoneme occupies 3 audio frames, the text features of the phoneme, the fundamental frequency of the phoneme, and the fundamental frequency range of the phoneme can be fused through a feature expansion network to obtain the fusion features of the phoneme, and the fusion features of the phoneme are expanded into 3 parts, and the fusion features of one phoneme are the initial features of one audio frame occupied by the phoneme. In this case, the initial features of the phoneme occupying the audio frame are used to describe the semantics of the phoneme and the physical characteristics of the sound when reading the phoneme.

[0148] The present application embodiment does not limit the structure, size, parameters, etc. of the feature expansion network. The feature expansion network can be based on the number of phonemes in the audio frame by replicating the phoneme sequence features. Expanded to the initial features of each audio frame when aligned to the audio frame Where h[i] represents the text feature or fusion feature of the i-th phoneme, e[i] represents the initial feature of the i-th audio frame, N′ represents the number of phonemes, and T represents the number of audio frames.

[0149] In step 10232, the preset style information and the initial features are fused through a frame-level encoder to obtain audio features corresponding to the target text.

[0150] Here, for any audio frame, the preset style information and the initial features of the audio frame are fused through a frame-level encoder (also known as a style encoder) to obtain the audio features of the audio frame. The audio features of the audio frame describe the semantics of the audio frame and the physical characteristics of the sound under the audio style described by the preset style information. Among them, when the preset style information is a style text, the style text can be first converted into a style vector through the frame-level encoder, and then the style vector and the initial features of the audio frame are fused. If the preset style information is a style vector, the style vector and the initial features of the audio frame can be directly fused through the frame-level encoder.

[0151] The fusion process is used to fuse the preset style information and the initial features of the audio frame to obtain the audio features of the audio frame, so as to obtain the audio features corresponding to the target text (i.e., the set of audio features of all audio frames). The embodiments of the present application are not limited to the fusion methods of the fusion process, such as weighted summation, splicing, mapping, etc.

[0152] Refer to Figure 3F, which is a flowchart of the audio synthesis method provided in an embodiment of the present application. Figure 3F shows that step 10232 in Figure 3E can be implemented through steps 102321 to 102324.

[0153] In step 102321, attention processing is performed on the preset style information and the initial feature to obtain a second attention feature.

[0154] In an embodiment of the present application, as shown in FIG7 , the frame-level encoder includes a network layer based on an attention mechanism, and the network layer based on the attention mechanism can be an attention layer such as a self-attention layer and a multi-head self-attention layer. For any audio frame, the preset style information and the initial features of the audio frame can be subjected to attention processing based on the attention mechanism to obtain a second attention feature of the audio frame. The second attention feature of the audio frame includes the attention feature of the audio frame in at least one channel. It should be noted that the audio signal corresponds to a frequency range, and multiple frequency segments are obtained by dividing the frequency range, and a frequency segment is a channel. In other words, any channel is a frequency segment corresponding to the audio signal, and the frequency segments corresponding to different channels may partially overlap or not overlap. The attention processing is to perform attention calculation on the preset style information and the initial features of the audio frame based on the attention mechanism to obtain the second attention feature. The embodiment of the present application is not limited to the form of attention calculation, such as self-attention calculation, multi-head self-attention calculation, etc.

[0155] Following step 102321, in step 102322, style-adaptive instance normalization is performed on the second attention feature to obtain a style normalized feature.

[0156] Among them, Adaptive Instance Normalization (AdaIN) is a normalization technique and a variant of Instance Normalization, which is used to improve the performance of style transfer in the field of audio synthesis. Instance normalization is a method that normalizes the input (i.e., the second attention feature) directly at the level of a single audio instance, calculates the mean and variance of each input separately, and then uses these statistics to normalize each channel of the input. Adaptive Instance Normalization adds an extra step to Instance Normalization, that is, it introduces another statistical information to adjust the normalization process. Among them, style-adaptive instance normalization processes the style of the second attention feature for adaptive instance normalization to obtain style-normalized features. Style-adaptive instance normalization provides a flexible method to combine content and style information to generate high-quality auditory output.

[0157] The second attention feature includes an attention feature of the audio frame on at least one channel.

[0158] In some embodiments, step 102322 can be implemented in the following manner: for any channel, based on the first feature of the audio frame on any channel, determine the first statistical feature on any channel; for any audio frame, based on the preset style information and the first statistical feature on any channel, perform a linear transformation on the attention feature of any audio frame on any channel to obtain a style normalized feature.

[0159] As an example, for any channel, the attention features of each audio frame in that channel can be counted to obtain the first statistical information of the channel. The first statistical information of the channel includes the mean and variance. The mean of the channel reflects the average state of the attention feature in the channel, and the variance of the channel reflects the fluctuation of the attention feature in the channel.

[0160] Then, the preset style information can be mapped to a first linear transformation parameter, and the mapping method is not limited here. The first linear transformation parameter includes at least one of a weight term and a bias term. The weight term is used to control the importance of the attention feature of any audio frame in any channel, and the bias term is used to control the style normalization feature of any audio frame in any channel to better reflect the audio style, making the style normalization feature of any audio frame in any channel more expressive.

[0161] Next, based on the mean and variance of any channel, the attention feature of any audio frame in that channel is adjusted to obtain the adjusted feature of the audio frame in that channel. Then, based on at least one of the weight term and the bias term, a linear transformation is performed on the adjusted feature of the audio frame in that channel to obtain the style normalized feature of the audio frame in that channel.

[0162] In summary, as shown in Figure 7, the frame-level encoder includes a Style Adaptive Instance Normalization (Style-AdaIN) layer (also known as a style normalization layer). The style normalization layer is used to perform style-adaptive instance normalization processing, and the style normalization layer normalizes along the sequence. For example, the style normalization layer determines the style normalization feature (referred to as the style normalization feature) of any audio frame in any channel according to formula (4).

[0163] Among them, Style-AdaIN(x,s) represents the style normalization feature, x represents the attention feature of the audio frame in the channel, S represents the style vector, G(s) represents the weight term obtained by mapping the style vector s, B(s) represents the bias term obtained by mapping the style vector s, μ(x) represents the average value of the attention feature in the channel, and σ(x) represents the variance of the attention feature in the channel.

[0164] In the embodiment of the present application, the frame-level encoder performs feature processing based on the first statistical information of the channel and the first linear transformation parameter to improve coding efficiency. Feature processing is performed through the style normalization layer to improve the expressiveness of the features and facilitate style control.

[0165] Following step 102322, in step 102323, the style normalization features are mapped to obtain mapping features.

[0166] Here, the embodiments of the present application are not limited to the form of mapping processing. For example, the style normalization features can be sequentially subjected to convolution, random discarding, random addition, etc. to achieve mapping processing of the style normalization features. Taking convolution as an example of mapping processing, the embodiments of the present application can perform convolution processing on the style normalization features and then discard and add the convolution results through a discarding and adding layer to obtain the mapping features.

[0167] Following step 102323, in step 102324, style-adaptive instance normalization is performed on the mapping features to obtain audio features corresponding to the target text.

[0168] Here, the style-adaptive instance normalization process in step 102324 is similar to the style-adaptive instance normalization process in step 102322, except that the processing objects are different.

[0169] In some embodiments, the frame-level encoder includes a style coding block, and steps 102321 to 102324 can be executed by the style coding block to fuse the preset style information and the initial features to obtain audio features corresponding to the target text.

[0170] In some embodiments, as shown in FIG7 , the frame-level encoder includes a plurality of cascaded style coding blocks, and the preset style information and the initial features of the audio frame can be fused through the cascaded style coding blocks, thereby accurately extracting the audio features corresponding to the target text through a progressive structure. Therefore, step 10232 can be implemented in the following manner: the preset style information and the initial features of the audio frame are fused through the first style coding block in the plurality of cascaded style coding blocks; the fusion result of the first style coding block is output to the subsequent cascaded style coding blocks, and the fusion processing and fusion result output are continued through the subsequent cascaded style coding blocks until the last style coding block is output; and the fusion result output by the last style coding block is used as the audio feature corresponding to the target text.

[0171] It should be noted that the processing process of each style coding block is similar, only the processing objects are different. The processing objects of the first style coding block are the preset style information and the initial features of the audio frame, and the processing objects of other style coding blocks are the output of the previous style coding block of other style coding blocks, among which other style coding blocks are the style coding blocks in the cascaded style coding blocks except the first style coding block.

[0172] Following step 102, in step 103, the audio features are normalized and stream processed by a priori encoder to obtain hidden variables of the target text.

[0173] Among them, the hidden variables include the prior distribution data that implicitly characterizes the waveform of the audio data to be synthesized (i.e., the synthesized waveform), such as the mean and variance of the audio features that conform to the prior distribution. The hidden variables are used to allow the waveform generator to synthesize a synthesized waveform that conforms to the audio style described by the preset style information and corresponds to the target text as much as possible, while also being able to generate data that does not exist in the synthesized waveform. Among them, the normalizing flow processing is used to convert the audio features into a continuous manifold representation (i.e., the hidden variables of the target text) through the prior encoder, and then sample from these manifold representations to generate a new audio waveform. Normalizing flow processing transforms the data distribution into a standard normal distribution (i.e., a Gaussian distribution with a mean of 0 and a variance of 1) through a series of reversible transformations (such as volume changes, rotations, and translations). In the field of audio synthesis, normalizing flow processing can effectively capture the complex distribution of audio data and generate realistic synthesized audio.

[0174] In some embodiments, step 103 can be implemented by: performing linear projection processing on the audio features to obtain statistical features of the prior distribution; and performing reversible transformation on the statistical features of the prior distribution to obtain hidden variables of the target text.

[0175] Here, the prior encoder also includes a linear projection layer, which linearly projects the audio features to obtain statistical features of the prior distribution, where the statistical features include the mean and variance used to construct the prior distribution. The prior encoder also includes a normalized flow model with an affine coupling layer, which performs a reversible transformation on the statistical features of the prior distribution to composite the hidden variables of the target text.

[0176] It should be noted that the normalizing flow model is dedicated to learning a bijection between the target space (e.g., the space of the prior distribution) and the latent space (e.g., the hidden variable z that obeys the normal distribution). The model structure of the normalizing flow is usually an invertible neural network (Invertible Neural Network) composed of multiple layers of reversible transformations. The embodiments of the present application are not limited to the structure of the normalizing flow model. The normalizing model can adopt a standard normalizing flow, or a normalizing flow such as a fully normalizing flow.

[0177] The embodiments of the present application use probability distribution to describe the characteristics and distribution of audio data. Normalizing Flow technology is a transformation-based method that gradually transforms a simple probability distribution (i.e., the statistical characteristics of the prior distribution) into a more complex probability distribution (i.e., hidden variables) by performing a series of reversible transformations on it. This complex probability distribution can then be used to better describe the characteristics and distribution of audio data, so as to synthesize an accurate waveform based on the hidden variables.

[0178] Following step 103 , in step 104 , waveform decoding processing is performed on the hidden variables of the target text by a waveform decoder to obtain a synthesized waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

[0179] Here, a waveform decoder is used to directly perform waveform decoding on the hidden variables of the target text, resulting in a synthesized waveform that matches the audio style described by the preset style information and corresponds to the target text. This eliminates the need to predict the mel-spectrogram of the target text using an acoustic model and then generate an audio waveform using a vocoder combined with the mel-spectrogram. This avoids artifacts in the synthesized audio caused by a distribution mismatch between the acoustic features predicted by the acoustic model and the acoustic features used to train the vocoder, thereby improving the quality and expressiveness of the audio synthesis. The synthesized waveform is a visual representation of the synthesized audio, meaning it uniquely represents the synthesized audio.

[0180] In some embodiments, the waveform decoder is a generator in a generative adversarial network, and step 104 can be implemented in the following manner: the following processing is performed by the generator in the trained generative adversarial network: convolution is performed on the hidden variables of the target text to obtain a convolution result; anti-aliasing multi-cycle synthesis processing is performed on the convolution result to obtain semantic features; nonlinear mapping processing is performed on the semantic features to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

[0181] Here, the waveform decoder can be a large visual generative adversarial network (BigVGAN) generator, that is, a generator in a large visual generative adversarial network, wherein the large visual generative adversarial network is a variant of the variational generative adversarial network (Variational Generative Adversarial Network). The large visual generative adversarial network combines the characteristics of the generative adversarial network (GAN) and the variational autoencoder (VAE), and generates high-quality audio through adversarial training and variational reasoning. The large visual generative adversarial network includes a generator and a discriminator. The BigVGAN generator introduces periodic nonlinearity and anti-aliasing representation, which brings the required inductive bias to waveform synthesis to improve audio quality. The BigVGAN generator can extract semantic features from audio data, and the semantic features represent the audio style described by the preset style information and are related to the target text. It should be noted that the embodiments of the present application are not limited to the structure of the waveform decoder. Among them, the anti-aliasing multi-cycle synthesis processing includes anti-aliasing multi-cycle synthesis operation.

[0182] It should be noted that the anti-aliasing multi-cycle synthesis process can be a one-time process, that is, the convolution result is subjected to a one-time anti-aliasing multi-cycle synthesis process to obtain semantic features; the anti-aliasing multi-cycle synthesis process can also be a cascade process, that is, the convolution result is subjected to a cascaded anti-aliasing multi-cycle synthesis process to obtain semantic features, and the semantic features are gradually extracted through the cascaded anti-aliasing multi-cycle synthesis process.

[0183] In some embodiments, anti-aliasing multi-cycle synthesis processing is performed on the convolution result to obtain semantic features, which can be achieved by: performing transposed convolution on the convolution result to obtain transposed convolution features; and performing anti-aliasing multi-cycle synthesis operation on the transposed convolution features to obtain semantic features.

[0184] Among them, in the field of audio synthesis, anti-aliasing multi-cycle synthesis operations are used to perform multi-cycle sampling and filtering on input data (such as transposed convolution features) to reduce or eliminate the aliasing effect caused by the digital audio sampling rate limit and improve the quality of the synthesized audio. For example, the anti-aliasing multi-cycle synthesis operation includes the following key steps: 1. Multi-cycle sampling, first sampling the input data multiple times within one cycle to obtain finer-grained data, so as to capture the details inside the waveform, thereby reducing distortion in the subsequent digital-to-analog conversion process; 2. Filtering, filtering the data obtained by multi-cycle sampling, for example, filtering out frequency components higher than half the sampling rate (i.e., the part above the folding frequency) to prevent these high-frequency components from aliasing during the digital-to-analog conversion; 3. Anti-aliasing, further smoothing the waveform edges of the filtered data through anti-aliasing technology to reduce the step-like edges introduced due to the sampling rate limit, that is, the sawtooth effect.

[0185] The embodiment of the present application can generate a smoother and more natural waveform through anti-aliasing multi-cycle synthesis operation, reduce the distortion and noise caused by sampling rate limitations, and thus improve the overall quality of the synthesized audio.

[0186] For example, a one-dimensional transposed convolution (TransposedConv1d) is first performed on the convolution result, and then an anti-aliased multi-cycle synthesis operation (AMP, Anti-aliased Multi-periodicity) is performed on the one-dimensional transposed convolution feature to obtain a semantic feature. It should be noted that when the anti-aliasing multi-cycle synthesis process is a cascade process, each anti-aliasing multi-cycle synthesis process is similar, and only the processing object is different. The processing object of the first anti-aliasing multi-cycle synthesis process is the convolution result, and the processing object of other anti-aliasing multi-cycle synthesis processes is the output of the previous anti-aliasing multi-cycle synthesis process of other anti-aliasing multi-cycle synthesis processes, where the other anti-aliasing multi-cycle synthesis processes are anti-aliasing multi-cycle synthesis processes other than the first anti-aliasing multi-cycle synthesis process in the cascaded anti-aliasing multi-cycle synthesis process.

[0187] The following describes the training method of the audio synthesis model. As mentioned above, the training method of the audio synthesis model provided in the embodiment of the present application can be implemented by various types of electronic devices, such as terminals, servers, or a combination of the two. Therefore, the execution entities of each step will not be repeated below. Referring to Figure 4A, Figure 4A is a flow chart of the training method of the audio synthesis model provided in the embodiment of the present application, which will be described in conjunction with the steps shown in Figure 4A, wherein the audio synthesis model includes a priori encoder, a posteriori encoder, a waveform decoder, and a discriminator.

[0188] In step 201 , sample language information, sample style information, waveform annotations of the text sample, and a spectrum corresponding to the waveform annotations are obtained.

[0189] The text sample is used as a training sample for the audio synthesis model. The sample language information of the text sample is used to express the content of the text sample, that is, the content of the synthesized audio. The sample language information can be the language text of the text sample, or it can be the language vector (also known as the sample language embedding) obtained by mapping each word in the language text to its own word vector. The language vector is the vector form of the language text and expresses the same content as the language text.

[0190] The sample style information is used to describe the synthesized audio style. This can be a synthesized style text or a style vector (also known as a sample style embedding) obtained by mapping each word in the style text to a word vector. The style vector is a vector representation of the style text and expresses the same content as the style text.

[0191] The waveform annotation of the text sample represents the actual waveform corresponding to the actual audio corresponding to the text sample, and the spectrum corresponding to the waveform annotation is the spectrum obtained by performing signal processing on the waveform annotation.

[0192] Following step 201 , in step 202 , the spectrogram corresponding to the waveform annotation and the sample style information are encoded by a posterior encoder to obtain hidden variables of the spectrogram.

[0193] The hidden variables of the spectrogram include posterior distribution data that implicitly represent the waveform of the audio data to be synthesized (ie, the synthesized waveform), such as the mean and variance of the spectrogram that conform to the posterior distribution.

[0194] Referring to FIG. 4B , FIG. 4A is a flow chart of the training method for the audio synthesis model provided in an embodiment of the present application. FIG. 4B shows that step 202 in FIG. 4A can be implemented through steps 2021 to 2023 .

[0195] In step 2021, the following processing is performed by the posterior encoder: causal residual processing is performed on the spectrum diagram corresponding to the waveform annotation and the sample style information to obtain a causal residual result.

[0196] Among them, causal residual processing is used to calculate causal residuals for the spectrogram corresponding to the waveform annotation and the sample style information. Here, the posterior encoder of the embodiment of the present application includes a WaveNet residual block (WaveNet Residual Block), and the sample style information is added to the WaveNet residual block as a global condition. Through the WaveNet residual block, the spectrogram corresponding to the waveform annotation and the sample style information are causally processed to obtain a causal residual result to supplement the missing information (such as phase, etc.) of the audio data reconstructed from the Mel spectrum. The audio data reconstructed using the WaveNet residual block is more natural than the sound generated using a traditional vocoder or algorithm.

[0197] In some embodiments, step 2021 can be implemented in the following manner: convolving the spectrum graph corresponding to the waveform annotation to obtain a convolution result; performing multiple residual processing based on the gated activation unit (Gated Activation Unit) on the convolution result and the sample style information to obtain multiple residual results, wherein one residual processing based on the gated activation unit corresponds to one residual result; performing skip-connection processing on the multiple residual results to obtain a causal residual result.

[0198] For example, causal convolution is first performed on the spectrum graph corresponding to the waveform annotation to obtain a convolution result. The embodiment of the present application is not limited to causal convolution, and can also be dilated convolution, etc.

[0199] Then, the convolution result and the sample style information are subjected to multiple residual processing based on the gated activation unit to obtain multiple residual results. The residual processing based on the gated activation unit can be implemented in the following ways: performing dilated convolution processing on the convolution result to obtain a dilated convolution result; adding the sample style information as a global condition to the gated activation unit, performing activation processing on the dilated convolution result through the gated activation unit to obtain an activation result; performing residual processing on the activation result and the convolution result to obtain a residual result. The activation processing is shown in formula (5).

[0200] Among them, W f,k 、W g,k 、 represents the learnable parameters of the kth residual processing based on the gated activation unit. tanh() represents the Tanh activation function, and σ() represents the sigmoid gated activation function. x represents the input of the gated activation unit, i.e., the dilated convolution result. z represents the output of the gated activation unit, i.e., the activation result. y represents the sample style information.

[0201] Next, skip connections are performed on the multiple residual results to obtain causal residual results. Of course, after skip connections are performed on the multiple residual results, the skip connection results can also be mapped to obtain causal residual results. For example, the skip connection results can be processed by the ReLU function, followed by one-dimensional ordinary convolution, and finally nonlinear mapping by the softmax function to obtain the causal residual result.

[0202] Following step 2021 , in step 2022 , style-adaptive instance normalization processing is performed on the causal residual result to obtain a style normalized feature.

[0203] Here, the style-adaptive instance normalization process in step 2022 is similar to the style-adaptive instance normalization process in step 102322, differing only in the processing objects. By performing style-adaptive instance normalization on the causal residual results, different acoustic features from multiple styles can be better distinguished and the effectiveness of audio synthesis can be further improved.

[0204] Following step 2022 , in step 2023 , linear projection processing is performed on the style normalized features to obtain statistical features of the posterior distribution.

[0205] Here, the linear projection process in step 2023 is similar to the linear projection process in step 103, with only the processing object being different. The posterior encoder further includes a linear projection layer, which performs linear projection processing on the style normalization features to obtain statistical features of the posterior distribution, wherein the statistical features include the mean and variance used to construct the posterior distribution.

[0206] Following step 2023 , in step 2024 , the statistical features of the posterior distribution are reparameterized to obtain the hidden variables of the spectrum graph.

[0207] Among them, the hidden variables are used to allow the waveform generator to synthesize a synthetic waveform that conforms to the audio style described by the sample style information and corresponds to the text sample as much as possible, while also generating data that does not exist in the synthetic waveform.

[0208] Here, step 2024 can be implemented by using a reparameterization function, combined with random noise ε, to reparameterize the statistical characteristics of the posterior distribution to obtain the hidden variables of the spectrogram. For example, if the statistical characteristics of the posterior distribution include variance σ and mean μ, then the reparameterization function is z = μ + σ⊙ε, where ε represents random noise and z represents the hidden variable. It should be noted that the embodiments of the present application are not limited to the form of the reparameterization function.

[0209] Following step 202 , in step 203 , waveform decoding processing is performed on the hidden variables of the spectrogram by a waveform decoder to obtain a synthetic waveform that conforms to the audio style described by the sample style information and corresponds to the text sample.

[0210] Here, the waveform decoding process in step 203 is similar to the waveform decoding process in step 104 , and only the processing object is different.

[0211] Following step 203 , in step 204 , the synthesized waveform and the waveform annotation are discriminated by a discriminator to obtain a discrimination result.

[0212] Here, the discriminator is a discriminator in a generative adversarial network, and the discriminator can be a BigVGAN discriminator (i.e., the discriminator in BigVGAN). The discriminator performs a discrimination process on the synthesized waveform and the waveform annotation to obtain a discrimination result, wherein the discrimination result includes a discrimination result of the synthesized waveform and a discrimination result of the waveform annotation, and the discrimination result is true or false. The discrimination process is used to discriminate the synthesized waveform and the waveform annotation to obtain a discrimination result of the synthesized waveform and a discrimination result of the waveform annotation.

[0213] Among them, the discriminator in the embodiment of the present application may include multiple multi-resolution discriminators (MRDs), that is, discriminator sub-modules, through which the synthetic waveform and waveform annotation are discriminated at different resolutions to obtain discrimination results at different resolutions.

[0214] Following step 204 , in step 205 , the sample language information and the sample style information are encoded by a priori encoder to obtain hidden variables of the text sample.

[0215] Here, step 205 can be implemented as follows: using an a priori encoder to generate audio features corresponding to the text sample based on the sample language information and the sample style information; and using the a priori encoder to perform normalized stream processing on the audio features corresponding to the text sample to obtain the hidden variables of the text sample. It should be noted that the generation process in step 205 is similar to that in step 102, with only the processing object being different. The normalized stream processing in step 205 is similar to that in step 103, with only the processing object being different.

[0216] Continuing from step 205, in step 206, the audio synthesis model is trained based on the hidden variables of the text sample, the hidden variables of the spectrum graph, and the identification results to obtain a trained audio synthesis model, wherein the trained audio synthesis model includes a priori encoder and a waveform decoder for executing the audio synthesis method.

[0217] Here, based on the hidden variables of the text sample, the hidden variables of the spectrum graph and the identification results, a loss function of the audio synthesis model is constructed, and the parameters of the audio synthesis model are updated based on the loss function to train the audio synthesis model and obtain a trained audio synthesis model.

[0218] Among them, the loss value is used to represent the difference between the output result of the model (i.e., the audio synthesis model in the embodiment of the present application) and the actual result (or the labeled result). The training of the model mainly involves two processes: forward propagation and back propagation. Taking the neural network model including the input layer, hidden layer and output layer as an example, the forward propagation process refers to processing through the input layer, hidden layer and output layer in sequence to finally obtain the output result; the back propagation process refers to propagating the calculated loss value to the output layer, hidden layer and input layer in sequence, thereby updating the weight parameters in each layer.

[0219] In some embodiments, the waveform decoder and the discriminator included in the audio synthesis model can be trained separately, and step 206 can be implemented by: training the waveform decoder and the discriminator included in the audio synthesis model based on the identification results; training the prior encoder included in the audio synthesis model based on the hidden variables of the text sample and the hidden variables of the spectrum graph.

[0220] For example, based on the waveform annotation discrimination results, the waveform decoder is trained to generate fake synthetic waveforms based on the hidden variables of real text samples. Based on the discrimination results of the fake synthetic waveforms and the real waveform annotations, the discriminator is trained to distinguish between real waveform annotations and fake synthetic waveforms. When training the waveform decoder, the discriminator parameters are fixed; when training the discriminator, the parameters of the generative model are fixed. During the discriminator optimization phase, the discriminator is optimized to distinguish between real waveforms and fake waveforms generated by the waveform decoder. During the waveform decoder optimization phase, the waveform decoder is optimized to generate waveforms that can confuse the discriminator. Through the interplay between the discriminator and the waveform decoder, the waveform decoder is able to generate waveforms that are as realistic as possible.

[0221] For example, based on the hidden variables of the text sample and the hidden variables of the spectrum graph, a loss function of the prior encoder is constructed, and the parameters of the prior encoder are updated based on the loss function of the prior encoder to train the prior encoder.

[0222] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0223] The audio synthesis method in related technologies is relatively rough. Usually, the mel-spectrogram of the text data is first predicted through an acoustic model, and then the audio waveform is generated by combining the mel-spectrogram with the vocoder to synthesize the mel-spectrogram corresponding to the text data to obtain the audio data corresponding to the text data. This synthesis method is easily affected by the distribution mismatch between the acoustic features predicted by the acoustic model and the acoustic features used to train the vocoder, which easily leads to artifacts in the synthesized audio and cannot achieve accurate audio synthesis, thereby affecting the user experience of normal audio synthesis.

[0224] To this end, an embodiment of the present application provides an audio synthesis model based on a temporal attention network neural network for achieving high expressiveness and high fidelity, and an audio synthesis method is implemented through the audio synthesis model. In the proposed audio synthesis model, the transformation block based on the multi-layer stacked SwishRNN is used as the above-mentioned language encoder, and the style-adaptive instance normalization (SAIN) layer is also developed into the frame-level encoder to improve the expressiveness of the modeling. At the same time, the variational autoencoders (VAEs) of the normalized flow and the BigVGAN generator (also known as the waveform decoder) with the adversarial training strategy are added to the audio synthesis model to further improve the quality and expressiveness of the speech synthesis.

[0225] The audio synthesis method provided in the embodiment of the present application is described below with reference to the audio synthesis model shown in FIG5 .

[0226] As shown in Figure 5, the audio synthesis model 50 provided in the embodiment of the present application can be regarded as a variational autoencoder VAE with an added normalization flow. The audio synthesis model includes a state-skip prior encoder (hereinafter referred to as the prior encoder), a SAIN-based posterior encoder 502 (SAIN-based Posterior Encoder), a BigVGAN generator 503 (BigVGAN Decoder, referred to as the generator) and a discriminator (Discriminators) located after the BigVGAN generator. Among them, the prior encoder is conditioned on language information, and the posterior encoder is conditioned on acoustic features (such as Mel-spectrogram). The prior encoder also includes variable predictors (Variance Predictors), which can predict the utterance duration, fundamental frequency, fundamental frequency range, etc. of each phoneme. Among them, the posterior encoder (SAIN-based Posterior Encoder) and the discriminators (Discriminators) are only used to train the audio synthesis model.

[0227] The following describes the prior encoder, posterior encoder, and BigVGAN generator in the audio synthesis model.

[0228] 1) About the prior encoder

[0229] The a priori encoder in the embodiment of the present application adopts a two-level hierarchical skip encoder (Hierarchical Skip-encoders) to significantly reduce pronunciation errors and improve voice quality and expressiveness. The two-level hierarchical skip encoder includes a phoneme-level language encoder (Linguistic Encoder) with a stack of transformation blocks based on SwishRNN (as shown in Figure 6), and a frame-level encoder (Frame-level Encoder) with a SAIN layer (as shown in Figure 7). The a priori encoder also includes a linear projection layer (Linear Projection Layer) after the skip encoder, and constructs the statistical features (including mean and variance) of the a priori distribution for the sequence c output by the SAIN-based frame-level encoder (Frame-level Encoder) through the linear projection layer. The a priori encoder also has a normalized flow f of a stack of affine coupling layers. θ , used to improve the flexibility of the prior distribution. Among them, the normalized flow f θ The normal prior distribution of is compounded with a set of reversible transformations to obtain the probability distribution p as shown in formula (6) θ (z|c):

[0230] Where c represents the output of the SAIN-based frame-level encoder. Z represents the hidden variable of the prior distribution. θ represents the normalized stream function, μ θ represents the mean of the prior distribution, σ θ represents the variance of the prior distribution.

[0231] 2) About SAIN-based Posterior Encoder

[0232] As shown in Figure 8, for the posterior encoder that estimates the posterior distribution of the hidden variable z given the acoustic feature condition x, a WaveNet residual block consisting of a gated activation unit and dilated convolutions with jump connections is used. The style embedding of the style label is added to the WaveNet residual block as a global condition. In order to better distinguish different acoustic features from multiple styles, a SAIN layer (Style-AdaIN) and a linear projection layer (Linear) are added after the WaveNet residual block to produce the mean and variance of the posterior distribution. The KL divergence is then calculated to optimize the audio synthesis model, where the KL divergence L KLThe calculation formula of L is shown in formula (7). KL =logq φ (z|x)-logp θ (z|c) (7)

[0233] Among them, z~q φ (z|x)=N(z;μ φ (x),σ φ (x)), x represents the high-resolution linear scale spectrogram (referred to as spectrogram) of the target speech. φ The z in (z|x) represents the hidden variable output by the SAIN-based posterior encoder, logp θ The z in (z|c) represents the normalized flow f θ Hidden variables for output.

[0234] 3) About BigVGAN Generator

[0235] The BigVGAN generator is used as a waveform decoder, conditioned on the potential hidden variable z in the VAE. The BigVGAN generator includes multiple transposed convolution blocks and an anti-aliasing multi-cycle synthesis (AMP) module. The transposed convolution block implements the transposed convolution operation, and the anti-aliasing multi-cycle synthesis module implements the anti-aliasing multi-cycle synthesis operation. The AMP module adds features from multiple residual blocks of different channel periods before the one-dimensional dilated convolution. The one-dimensional dilated convolution uses a Snake function to provide a periodic induction bias and a low-pass filter to achieve anti-aliasing.

[0236] Following the discriminator architecture of BigVGAN, a multi-resolution discriminator (MRD) is used in the time-frequency domain. The multi-resolution discriminator operates on multiple spectrograms with different STFT resolutions. The multi-resolution discriminator uses a multi-cycle discriminator. The one-dimensional signal in the multi-cycle discriminator is reshaped into a two-dimensional representation with varying height and width to capture multiple periodic structures separately through two-dimensional convolution. The loss function L of the BigVGAN generator (G) is G and the discriminator's loss function L D They are as follows, through the loss function L G and the discriminator's loss function L D Train BigVGAN generator and discriminator, loss function L G As shown in formula (8), the loss function L of the discriminator is D As shown in formula (9):

[0237] Among them, Dk L represents the kth MRD discriminator. adv Use least squares to generate adversarial loss, L fm represents the feature matching loss (intermediate features from the discriminator distance), L mel (G) Spectrum between the synthesized waveform’s mel-spectrogram and the corresponding waveform annotation Regression loss.

[0238] In order to better evaluate the performance of the audio synthesis model proposed in the embodiment of the present application (hereinafter referred to as this model) and other TTS models, the following experimental settings were adopted: a highly expressive Chinese corpus containing 11.8 hours of speech from 7 different speakers was used as a training data set. The TTS models include DurIAN, FastSpeech2, DiffSpeech, VITS, and DurIAN-E. Except for VITS and the audio synthesis model proposed in the embodiment of the present application, all models share an additional BigVGAN vocoder, which is trained separately under the conditions of real Mel-spectrograms to better compare the performance between different acoustic models.

[0239] Different TTS models were objectively evaluated using a targeted test. In this targeted test, all TTS models were maintained at the same duration as natural speech (i.e., real speech) for ease of comparison. Table 1 shows the root mean square error (RMSE) and correlation coefficient of the F0 values ​​on a linear scale between natural and synthesized speech for different TTS models. The models were divided into three groups based on whether they used a denoiser to generate waveforms or generated waveforms directly. Acoustic features for comparison were re-extracted from the synthesized waveforms.

[0240] Table 1 Distortion between acoustic features of natural speech and synthesized speech from different TTS models. V / UV represents frame-level voiced / unvoiced error, BAP and Corr. represent BAP prediction error and correlation coefficient, respectively. MCD represents Mel-frequency cepstral distortion, and BAP represents band-wise aperiodic distortion.

[0241] The results shown in Table 1 show that the speech synthesized by this model can obtain the most accurate F0 and BAP values.

[0242] Figure 9 shows examples of pitch contours from different DurIAN models. Compared to DurIAN-E and our model, DurIAN's F0 curve is smoother and less volatile. Among TTS models using a DDPM-based denoiser and direct waveform generation, our model also achieves the lowest MCD value. Of the three systems, the one corresponding to DurIAN and FastSpeech2 exhibits the least spectral distortion, primarily because the acoustic model in this group directly optimizes the MSE loss of spectral features. Although DurIAN and FastSpeech2 achieve the lowest MCD, the oversmoothing issue shown in Figure 10 severely degrades speech quality. As shown in Figure 10, our model outperforms DurIAN-E, producing clearer harmonic structure and more realistic spectral details.

[0243] For the subjective MOS test, 20 test speech sounds synthesized by different models were randomly selected from the test set and evaluated by 10 experienced listeners, and the MOS results were obtained as shown in Table 2.

[0244] Table 2 MOS values ​​of different models with 95% confidence intervals

[0245] As shown in Table 2, our model achieves the best MOS scores compared to other TTS models, demonstrating the sufficient capacity of the proposed model. The VITS model can directly generate waveforms in an end-to-end manner, surpassing the multi-stage model DiffSpeech, in which the predicted mel-spectrogram is first enhanced by a DDPM-based denoiser and then converted into a waveform by a vocoder. Although DurIAN-E is also a multi-stage model, it achieves better MOS results than VITS due to its two-stage hierarchical skip encoder architecture. Our model combines the advantages of these models, including the end-to-end waveform generation approach of VITS and other complex structures such as SwishRNN and SAIN layers as prior encoders and pre-encoders, resulting in more expressive and high-fidelity speech synthesis.

[0246] In summary, the audio synthesis model proposed in the embodiments of this application utilizes a two-level hierarchical skip encoder and a SAIN-based a posteriori encoder to achieve more natural prosody and better expressiveness. To further improve speech quality, a BigVGAN decoder based on variable hidden variables and style embedding is also adopted. Experimental results from objective and subjective tests demonstrate that the audio synthesis model proposed in the embodiments of this application can achieve better performance than other TTS models.

[0247] So far, the audio synthesis method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. The following will continue to describe the cooperation of various modules in the audio synthesis device 555 provided by the embodiment of the present application to implement the audio synthesis scheme.

[0248] The calling module 5551 is configured to call the audio synthesis model to perform the following processing based on the language information and preset style information of the target text, wherein the audio synthesis model includes a priori encoder and a waveform decoder; the generating module 5552 is configured to generate the audio features corresponding to the target text based on the language information and the preset style information through the priori encoder; the normalizing module 5553 is configured to perform normalized stream processing on the audio features through the priori encoder to obtain the hidden variables of the target text; the waveform decoding module 5554 is configured to perform waveform decoding processing on the hidden variables of the target text through the waveform decoder to obtain a synthesized waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

[0249] In some embodiments, the normalization module 5553 is further configured to perform linear projection processing on the audio features to obtain statistical features of the prior distribution; and perform reversible transformation on the statistical features of the prior distribution to obtain hidden variables of the target text.

[0250] In some embodiments, the waveform decoder is a generator in a generative adversarial network; the waveform decoding module 5554 is also configured to perform the following processing through the trained generator in the generative adversarial network: convolve the hidden variables of the target text to obtain a convolution result; perform anti-aliasing multi-cycle synthesis processing on the convolution result to obtain semantic features; perform nonlinear mapping processing on the semantic features to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

[0251] In some embodiments, the waveform decoding module 5554 is further configured to perform transposed convolution on the convolution result to obtain transposed convolution features; and perform anti-aliasing multi-cycle synthesis operation on the transposed convolution features to obtain the semantic features.

[0252] In some embodiments, the generation module 5552 is further configured to perform the following processing through the prior encoder: encoding the language information at the phoneme level to obtain the text features of the phonemes in the language information; determining the acoustic features of the phonemes based on the text features of the phonemes and the preset style information; encoding the acoustic features of the phonemes at the audio frame level to obtain the audio features corresponding to the target text.

[0253] In some embodiments, the a priori encoder includes a language encoder; the generation module 5552 is also configured to perform acoustic encoding processing on the language information through the language encoder to obtain the language features of the target text, wherein the language features include text features of multiple phonemes and text features of multiple rhythms; and extract the text features of the multiple phonemes from the language features.

[0254] In some embodiments, the language encoder includes multiple cascaded language coding blocks; the generation module 5552 is further configured to perform acoustic coding processing on the language information through the first language coding block among the multiple cascaded language coding blocks; output the coding result of the first language coding block to the subsequently cascaded language coding blocks, continue to perform acoustic coding processing and output coding results through the subsequently cascaded language coding blocks until the last language coding block is output; and use the coding result output by the last language coding block as the language feature of the target text.

[0255] In some embodiments, each of the language encoding blocks includes an attention layer, an activation network layer, a discarding and adding layer, and a layer normalization layer; the generating module 5552 is further configured to perform attention processing on the encoding result output by the k-1 language encoding block through the attention layer included in the k-th language encoding block to obtain the first attention feature of the k-th language encoding block; perform mapping processing on the first attention feature of the k-th language encoding block through the activation network layer included in the k-th language encoding block to obtain the mapping feature of the k-th language encoding block; The language coding block includes a discarding and adding layer, which performs a discarding and adding process on the mapping features of the k-th language coding block and the first attention features of the k-th language coding block to obtain the discarding and adding features of the k-th language coding block; the layer normalization layer included in the k-th language coding block normalizes the discarding and adding features of the k-th language coding block to obtain an encoding result of the k-th language coding block, and outputs the encoding result of the k-th language coding block, wherein k is a successively increasing positive integer, 1<k<K, and K is the number of language coding blocks.

[0256] In some embodiments, the a priori encoder includes a frame-level encoder, and the acoustic features of the phoneme include the number of audio frames containing the phoneme; the generation module 5552 is also configured to determine the initial features of the audio frame corresponding to the target text based on the number of audio frames containing the phoneme and the text features of each phoneme; through the frame-level encoder, the preset style information and the initial features of the audio frame are fused to obtain the audio features corresponding to the target text.

[0257] In some embodiments, the generation module 5552 is further configured to perform attention processing on the preset style information and the initial features of the audio frame to obtain a second attention feature; perform style-adaptive instance normalization processing on the second attention feature to obtain a style-normalized feature; perform mapping processing on the style-normalized feature to obtain a mapping feature; perform style-adaptive instance normalization processing on the mapping feature to obtain an audio feature corresponding to the target text.

[0258] So far, the training method of the audio synthesis model provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. The following will continue to describe the various modules in the training device 556 of the audio synthesis model provided by the embodiment of the present application to cooperate to implement the training scheme of the audio synthesis model.

[0259] The acquisition module 5561 is configured to acquire the sample language information, sample style information, waveform annotation of the text sample, and the spectrogram corresponding to the waveform annotation of the text sample; the posterior module 5562 is configured to encode the spectrogram corresponding to the waveform annotation and the sample style information through the posterior encoder to obtain the hidden variables of the spectrogram; the decoding module 5563 is configured to perform waveform decoding processing on the hidden variables of the spectrogram through the waveform decoder to obtain a synthesized waveform that conforms to the audio style described by the sample style information and corresponds to the text sample; the identification module 5564 is configured to perform identification processing on the synthesized waveform and the waveform annotation through the discriminator to obtain an identification result; the a priori module 5565 is configured to encode the sample language information and the sample style information through the a priori encoder to obtain the hidden variables of the text sample; the training module 5566 is configured to train the audio synthesis model based on the hidden variables of the text sample, the hidden variables of the spectrogram, and the identification result to obtain a trained audio synthesis model.

[0260] In some embodiments, the training module 5566 is further configured to train the waveform decoder and the discriminator included in the audio synthesis model based on the identification results; and to train the prior encoder included in the audio synthesis model based on the hidden variables of the text sample and the hidden variables of the spectrogram.

[0261] In some embodiments, the posterior module 5562 is further configured to perform the following processing through the posterior encoder: performing causal residual processing on the spectrum graph corresponding to the waveform annotation and the sample style information to obtain a causal residual result; performing style adaptive instance normalization processing on the causal residual result to obtain a style normalization feature; performing linear projection processing on the style normalization feature to obtain the statistical characteristics of the posterior distribution; and performing reparameterization processing on the statistical characteristics of the posterior distribution to obtain the hidden variables of the spectrum graph.

[0262] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the audio synthesis method or audio synthesis model training method described in the present invention.

[0263] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the audio synthesis method provided in an embodiment of the present application, or the training method of an audio synthesis model, for example, the audio synthesis method shown in Figure 3A.

[0264] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0265] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0266] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0267] As an example, computer executable instructions can be deployed to be executed on one electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed in multiple locations and interconnected by a communication network. The above is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An audio synthesis method, applied to an electronic device, the method comprising: Based on the language information and preset style information of the target text, the audio synthesis model is called to perform the following processing, wherein the audio synthesis model includes a priori encoder and a waveform decoder: Generate, by the a priori encoder, audio features corresponding to the target text based on the language information and the preset style information; By using the prior encoder, the audio features are subjected to normalized stream processing to obtain hidden variables of the target text; The waveform decoder performs waveform decoding processing on the hidden variables of the target text to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

2. The method according to claim 1, wherein: The step of performing normalized stream processing on the audio features to obtain hidden variables of the target text includes: Performing linear projection processing on the audio features to obtain statistical features of prior distribution; The statistical features of the prior distribution are reversibly transformed to obtain hidden variables of the target text.

3. The method according to claim 1 or 2, wherein: The waveform decoder is a generator in a generative adversarial network; The waveform decoding process is performed on the hidden variables of the target text by the waveform decoder to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text, including: The following processing is performed by the generator in the generative adversarial network: Convolving the hidden variables of the target text to obtain a convolution result; Performing anti-aliasing multi-cycle synthesis processing on the convolution result to obtain semantic features; Nonlinear mapping is performed on the semantic features to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

4. The method according to claim 3, wherein: The anti-aliasing multi-cycle synthesis process is performed on the convolution result to obtain the semantic feature, including: Performing transposed convolution on the convolution result to obtain a transposed convolution feature; An anti-aliasing multi-cycle synthesis operation is performed on the transposed convolution feature to obtain the semantic feature.

5. The method according to any one of claims 1 to 4, wherein: The generating, by the a priori encoder, the audio features corresponding to the target text based on the language information and the preset style information includes: The following processing is performed by the a priori encoder: Performing phoneme-level encoding processing on the language information to obtain text features of the phonemes in the language information; Determining an acoustic feature of the phoneme based on the text feature of the phoneme and the preset style information; The acoustic features of the phonemes are encoded at the audio frame level to obtain audio features corresponding to the target text.

6. The method according to claim 5, wherein: The a priori encoder includes a language encoder; The performing phoneme-level encoding processing on the language information to obtain text features of the phonemes in the language information includes: The language information is acoustically encoded by the language encoder to obtain language features of the target text, wherein the language features include text features of multiple phonemes and text features of multiple prosody; Text features of the plurality of phonemes are extracted from the language features.

7. The method according to claim 6, wherein: The language encoder comprises a plurality of cascaded language encoding blocks; The language information is acoustically encoded by the language encoder to obtain the target text Language features include: Performing acoustic coding processing on the language information by using a first language coding block among the plurality of cascaded language coding blocks; Outputting the encoding result of the first language encoding block to the subsequent cascaded language encoding blocks, and continuing to perform acoustic encoding processing and output encoding results through the subsequent cascaded language encoding blocks until outputting to the last language encoding block; The encoding result output by the last language encoding block is used as the language feature of the target text.

8. The method according to claim 7, wherein: Each of the language encoding blocks includes an attention layer, an activation network layer, a discard addition layer, and a layer normalization layer; The step of continuing the acoustic coding process and outputting the coding result through the subsequent cascaded language coding blocks includes: Performing attention processing on the encoding result output by the k-1th language encoding block through the attention layer included in the kth language encoding block to obtain a first attention feature of the kth language encoding block; Performing mapping processing on the first attention feature of the k-th language coding block through the activation network layer included in the k-th language coding block to obtain the mapping feature of the k-th language coding block; Performing a discarding and adding process on the mapping feature of the k-th language coding block and the first attention feature of the k-th language coding block through the discarding and adding layer included in the k-th language coding block to obtain the discarding and adding feature of the k-th language coding block; The discarded added features of the k-th language coding block are normalized by the layer normalization layer included in the k-th language coding block to obtain the encoding result of the k-th language coding block, and the encoding result of the k-th language coding block is output, wherein k is a positive integer that increases successively, 1<k<K, and K is the number of language coding blocks.

9. The method according to claim 5, wherein: The a priori encoder comprises a frame-level encoder, and the acoustic feature of the phoneme comprises the number of audio frames containing the phoneme; The step of performing audio frame-level encoding processing on the acoustic features of the phonemes to obtain audio features corresponding to the target text includes: Determining initial features corresponding to the target text based on the number of audio frames containing the phoneme and the text features of each phoneme; The preset style information and the initial features are fused through the frame-level encoder to obtain audio features corresponding to the target text.

10. The method according to claim 9, wherein: The step of fusing the preset style information and the initial features of the audio frame through the frame-level encoder to obtain the audio features corresponding to the target text includes: Performing attention processing on the preset style information and the initial feature to obtain a second attention feature; performing style adaptive instance normalization processing on the second attention feature to obtain a style normalized feature; Performing mapping processing on the style normalization feature to obtain a mapping feature; The mapping features are subjected to style-adaptive instance normalization processing to obtain audio features corresponding to the target text.

11. A training method for an audio synthesis model, applied to an electronic device, wherein the audio synthesis model comprises a priori encoder, a posteriori encoder, a waveform decoder, and a discriminator; The method comprises: Obtaining sample language information, sample style information, waveform annotation of the text sample, and a spectrum diagram corresponding to the waveform annotation; The spectrum diagram corresponding to the waveform annotation and the sample style information are encoded by the a posteriori encoder. Code processing to obtain hidden variables of the spectrum graph; By means of the waveform decoder, waveform decoding processing is performed on the hidden variables of the spectrogram to obtain a synthetic waveform that conforms to the audio style described by the sample style information and corresponds to the text sample; The synthesized waveform and the waveform annotation are subjected to identification processing by the identifier to obtain an identification result; By means of the a priori encoder, the sample language information and the sample style information are encoded to obtain hidden variables of the text sample; Based on the hidden variables of the text sample, the hidden variables of the spectrogram and the identification result, the audio synthesis model is trained to obtain a trained audio synthesis model.

12. The method according to claim 11, wherein: The step of training the audio synthesis model based on the hidden variables of the text sample, the hidden variables of the spectrogram, and the identification result comprises: Based on the identification result, training the waveform decoder and the discriminator included in the audio synthesis model; Based on the hidden variables of the text sample and the hidden variables of the spectrogram, the prior encoder included in the audio synthesis model is trained.

13. The method according to claim 11 or 12, wherein: The encoding process of the spectrum graph corresponding to the waveform annotation and the sample style information by the posterior encoder to obtain the hidden variables of the spectrum graph includes: The following processing is performed by the a posteriori encoder: Performing causal residual processing on the spectrum graph corresponding to the waveform annotation and the sample style information to obtain a causal residual result; Performing style adaptive instance normalization processing on the causal residual result to obtain a style normalized feature; Performing linear projection processing on the style normalization features to obtain statistical features of the posterior distribution; The statistical characteristics of the posterior distribution are reparameterized to obtain hidden variables of the spectrum graph.

14. An audio synthesis device, comprising: A calling module configured to call an audio synthesis model based on language information of a target text and preset style information, wherein the audio synthesis model includes a priori encoder and a waveform decoder; A generating module, configured to generate audio features corresponding to the target text based on the language information and the preset style information through the priori encoder; A normalization module configured to perform normalized stream processing on the audio features through the prior encoder to obtain hidden variables of the target text; The waveform decoding module is configured to perform waveform decoding processing on the hidden variables of the target text through the waveform decoder to obtain a synthetic waveform that conforms to the audio style described by the preset style information and corresponds to the target text.

15. An electronic device, comprising: Memory for storing computer programs or computer executable instructions; A processor, used to implement the audio synthesis method described in any one of claims 1 to 10, or the audio synthesis model training method described in any one of claims 11 to 13 when executing the computer program or computer executable instructions stored in the memory.

16. A computer-readable storage medium storing a computer program or a computer-executable instruction, wherein the computer program or the computer-executable instruction, when executed by a processor, implements the audio synthesis method described in any one of claims 1 to 10, or the audio synthesis model training method described in any one of claims 11 to 13.

17. A computer program product, comprising a computer program or computer executable instructions, which, when executed by a processor, implements the audio synthesis method described in any one of claims 1 to 10, or the audio synthesis model training method described in any one of claims 11 to 13.

Images