Artificial intelligence-based audio generation methods, devices, equipment, and storage media

Abstract

This application provides an audio generation method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence; it relates to artificial intelligence technology; the method includes: sampling multiple audio data of a target object to obtain reference audio data of the target object; performing audio encoding processing on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data; performing timbre-based attention processing on the reference embedding vector of the reference audio data to obtain a timbre embedding vector of the target object; performing text encoding processing on the target text to obtain a content embedding vector of the target text; and performing synthesis processing based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain audio data that conforms to the timbre of the target object and corresponds to the target text. This application can improve the stability of audio synthesis.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an artificial intelligence-based audio generation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Artificial Intelligence (AI) is a comprehensive technology within computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a multidisciplinary field encompassing a wide range of areas, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in more fields and play an increasingly important role.

[0003] The audio synthesis methods in related technologies are relatively crude. They usually extract features directly from the audio data of the target object and synthesize the audio data based on the extracted embedding vector of the target object. This synthesis method cannot achieve accurate audio synthesis, thus affecting the normal audio synthesis experience for users. Summary of the Invention

[0004] This application provides an artificial intelligence-based audio generation method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of audio synthesis.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an artificial intelligence-based audio generation method, including:

[0007] Multiple audio data of the target object are sampled and processed to obtain reference audio data of the target object;

[0008] The reference audio data of the target object is subjected to audio encoding processing to obtain the reference embedding vector of the reference audio data;

[0009] The reference embedding vector of the reference audio data is subjected to timbre-based attention processing to obtain the timbre embedding vector of the target object.

[0010] The target text is processed by text encoding to obtain the content embedding vector of the target text;

[0011] Based on the timbre embedding vector of the target object and the content embedding vector of the target text, a synthesis process is performed to obtain audio data that matches the timbre of the target object and corresponds to the target text.

[0012] This application provides an artificial intelligence-based audio generation device, comprising:

[0013] The sampling module is used to sample and process multiple audio data of the target object to obtain reference audio data of the target object;

[0014] The audio encoding module is used to perform audio encoding processing on the reference audio data of the target object to obtain the reference embedding vector of the reference audio data;

[0015] The attention module is used to perform timbre-based attention processing on the reference embedding vector of the reference audio data to obtain the timbre embedding vector of the target object;

[0016] The text encoding module is used to perform text encoding processing on the target text to obtain the content embedding vector of the target text.

[0017] The synthesis module is used to perform synthesis processing based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain audio data that conforms to the timbre of the target object and corresponds to the target text.

[0018] In the above technical solution, the attention module is further used to perform object timbre-based embedding space mapping processing on the reference embedding vector of the reference audio data to obtain the weights of multiple object timbres;

[0019] Based on the weights of multiple object timbres, the embedding vectors of multiple object timbres are weighted and summed to obtain the timbre embedding vector of the target object.

[0020] In the above technical solution, the audio encoding module is further used to perform convolution processing on the reference audio data of the target object to obtain the convolution embedding vector of the reference audio data;

[0021] The convolutional embedding vector of the reference audio data is forward encoded to obtain the reference embedding vector of the reference audio data.

[0022] In the above technical solution, the audio encoding is implemented through a reference encoder, which includes multiple cascaded convolutional layers; the audio encoding module is also used to perform convolution processing on the reference audio data of the target object through the first convolutional layer in the multiple cascaded convolutional layers.

[0023] The convolution result of the first convolutional layer is output to subsequent cascaded convolutional layers, so that convolution processing and convolution result output continue in the subsequent cascaded convolutional layers until the output is reached in the last convolutional layer.

[0024] The convolution result output from the last convolutional layer is used as the convolutional embedding vector of the reference audio data.

[0025] In the above technical solution, the audio encoding is implemented through a reference encoder, which includes a recurrent neural network; the audio encoding module is also used to perform vector update processing based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain the update information of the reference audio data;

[0026] Vector reset processing is performed based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain the reset information of the reference audio data;

[0027] Based on the update information and reset information of the reference audio data, the convolutional embedding vector of the reference audio data is subjected to context encoding to obtain the reference embedding vector of the reference audio data.

[0028] In the above technical solution, the audio encoding module is further used to concatenate the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain a concatenated vector;

[0029] The spliced ​​vector is subjected to an update gate-based mapping process to obtain the update information of the reference audio data.

[0030] In the above technical solution, the audio encoding module is further used to perform candidate vector-based mapping processing based on the reset information of the target object, the hidden vector of the recurrent neural network, and the convolutional embedding vector of the target object to obtain the candidate embedding vector of the reference audio data;

[0031] Based on the updated information of the target object, the hidden vector of the recurrent neural network, and the candidate embedding vector of the reference audio data, vector mapping processing is performed to obtain the reference embedding vector of the reference audio data.

[0032] In the above technical solution, the synthesis module is further used to align the timbre embedding vector of the target object and the content embedding vector of the target text to obtain the alignment vector of the target text;

[0033] Based on the alignment vector of the target text, the acoustic feature sequence of the target text is obtained by decoding.

[0034] The acoustic feature sequence is subjected to acoustic feature transformation to obtain audio data that matches the timbre of the target object and corresponds to the target text.

[0035] In the above technical solution, the synthesis module is further used to smooth the acoustic feature sequence to obtain the corresponding spectral data;

[0036] Perform a Fourier transform on the corresponding spectral data to obtain audio data that matches the timbre of the target object and corresponds to the target text.

[0037] In the above technical solution, the audio generation method is implemented by calling a neural network model; the device further includes:

[0038] The training module is used to perform timbre-based attention processing on the reference audio data of the object sample through the initialized neural network model to obtain the timbre embedding vector of the object sample.

[0039] Based on the timbre embedding vector of the object sample, object prediction processing is performed to obtain the predicted object of the reference audio data; based on the predicted object of the reference audio data and the object label of the object sample, the first loss function of the neural network model is constructed.

[0040] The parameters of the neural network model are updated based on the first loss function, and the updated parameters of the neural network model are used as the parameters of the trained neural network model.

[0041] In the above technical solution, the training module is further configured to perform at least one of the following processes through the initialized neural network model:

[0042] Text samples are encoded to obtain content embedding vectors of the text samples, and a second loss function of the neural network model is constructed based on the content embedding vectors of the text samples.

[0043] Based on the timbre embedding vector of the object sample and the content embedding vector of the text sample, a third loss function of the neural network model is constructed.

[0044] Wherein, at least one of the second loss function and the third loss function is used in combination with the first loss function to update the parameters of the neural network model.

[0045] In the above technical solution, the training module is further used to perform object prediction processing based on the content embedding vector of the text sample to obtain the predicted object of the text sample;

[0046] A fourth loss function is constructed based on the predicted object of the text sample and the object label of the object sample;

[0047] The fourth loss function is inverted to obtain the second loss function of the neural network model.

[0048] In the above technical solution, the training module is further used to perform synthesis processing based on the timbre embedding vector of the object sample and the content embedding vector of the text sample to obtain an audio data sample that conforms to the timbre of the object sample and corresponds to the text sample.

[0049] Alignment prediction processing is performed based on the timbre embedding vector of the object sample and the content embedding vector of the text sample to obtain the predicted alignment information of the text sample.

[0050] Forced alignment processing is performed on the timbre embedding vector of the object sample and the content embedding vector of the text sample to obtain the forced alignment information of the text sample;

[0051] Linear mapping is performed based on the number of characters in the text sample and the number of audio frames in the audio data sample to obtain the linear alignment information of the text sample.

[0052] Based on the predicted alignment information, the forced alignment information, and the linear alignment information of the text samples, a third loss function for the neural network model is constructed.

[0053] In the above technical solution, the training module is further used to construct first difference information based on the difference between the linear alignment information of the text sample and the predicted alignment information based on the text sample;

[0054] A second difference information is constructed based on the difference between the forced alignment information of the text sample and the predicted alignment information of the text sample.

[0055] The first difference information and the second difference information are weighted and summed to obtain the third loss function of the neural network model.

[0056] This application provides an electronic device for audio generation, the electronic device comprising:

[0057] Memory, used to store executable instructions;

[0058] The processor, when executing executable instructions stored in the memory, implements the AI-based audio generation method provided in the embodiments of this application.

[0059] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the artificial intelligence-based audio generation method provided in this application.

[0060] The embodiments of this application have the following beneficial effects:

[0061] By performing attention processing on the reference audio data of the sampled target object, the robustness of timbre embedding vector extraction is improved. The timbre embedding vector of the target object and the content embedding vector of the target text are combined for audio synthesis processing to improve the stability of audio synthesis. Finally, based on the accurate timbre embedding vector, precise personalized audio generation is achieved. Attached Figure Description

[0062] Figure 1 This is a schematic diagram illustrating an application scenario of the audio generation system provided in this application embodiment;

[0063] Figure 2 This is a schematic diagram of the structure of an electronic device for audio generation provided in an embodiment of this application;

[0064] Figures 3-5 This is a flowchart illustrating the artificial intelligence-based audio generation method provided in an embodiment of this application;

[0065] Figure 6 This is a schematic diagram of the structure of a reference encoder provided in an embodiment of this application;

[0066] Figure 7 This is a schematic diagram of the structure of the cascaded convolutional layer provided in the embodiments of this application;

[0067] Figure 8 This is a schematic diagram of the embedded space provided in an embodiment of this application;

[0068] Figure 9 This is a training flowchart provided in an embodiment of this application;

[0069] Figure 10 This is a schematic diagram of a fast and effective customized speech synthesis model structure provided in an embodiment of this application;

[0070] Figure 11 This is a structural block diagram of the reference encoder provided in the embodiments of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0074] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0075] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0076] 1) Convolutional Neural Networks (CNNs): A class of feedforward neural networks (FNNs) that incorporate convolutional computations and have a deep structure, CNNs are one of the representative algorithms of deep learning. CNNs possess representation learning capabilities, enabling them to perform shift-invariant classification of input images according to their hierarchical structure.

[0077] 2) Recurrent Neural Network (RNN): A type of recurrent neural network that takes sequence data as input, recurses along the direction of sequence evolution, and connects all nodes (recurrent units) in a chain-like manner. RNNs possess memory, parameter sharing, and Turing completeness, thus offering advantages in learning the nonlinear characteristics of sequences.

[0078] 3) Phoneme: The smallest basic unit of speech, phonemes are the foundation upon which humans can distinguish one word from another. Phonemes make up syllables, and syllables make up different words and phrases.

[0079] 4) Timbre: Different audio frequencies always exhibit unique waveform characteristics, just as different objects vibrate with different features. Different sound-producing bodies, due to their different materials and structures, produce audio frequencies with different timbres. For example, the sounds produced by a piano, violin, and human are different, and each person produces a different audio frequency. Therefore, timbre can be understood as a characteristic of audio frequencies.

[0080] 5) Target objects: Real objects in the real world or virtual objects in virtual scenes, such as real users, virtual characters, virtual animals, and anime characters.

[0081] In recent years, with the massive improvement in computing power, deep learning technology has been extensively researched and applied, further promoting the development of speech synthesis technology. End-to-end acoustic modeling methods for speech synthesis have begun to emerge. These methods directly predict the corresponding acoustic feature sequences from the input character or phoneme sequences, and have been widely used in both academia and industry. However, training such a commercial speech synthesis system typically requires tens of hours of data, a data requirement that is impractical in many application scenarios. Therefore, the need for customized speech synthesis technologies based on limited data is becoming increasingly urgent.

[0082] Customized acoustic modeling methods for speech synthesis can be divided into two main categories: The first method involves pre-training a multi-speaker model (also known as an averaging model) on corpora of multiple speakers (objects), and then adaptively training the model using a small amount of speaker data. The second method involves directly predicting the speaker embedding vector from the target speaker's audio, and then directly inputting the embedding vector into the averaging model without any fine-tuning of the model.

[0083] However, the relevant technologies have the following problems:

[0084] 1) One-hot encoding is used to represent speakers in speaker embedding space modeling. This representation information can only distinguish different speakers, but does not contain information related to speaker timbre. Another method is to extract the embedding space representation directly from the audio. Since it extracts speaker embedding information directly from the audio paired with the text, it inevitably introduces information related to audio content (such as prosody, style, etc.), which leads to unstable synthesis results when the content of the text to be synthesized and the reference audio are inconsistent.

[0085] 2) In the design of the averaging model, it cannot be guaranteed that the text encoder shares information with all speakers;

[0086] 3) In terms of training strategies for customized speech synthesis models, the limited amount of customized corpus and data sparsity often lead to inefficient training and unstable models.

[0087] To address the aforementioned issues, embodiments of this application provide an artificial intelligence-based audio generation method, apparatus, electronic device, and computer-readable storage medium, which can improve the stability of audio synthesis.

[0088] The AI-based audio generation method provided in this application can be implemented by a terminal / server alone; or it can be implemented collaboratively by a terminal and a server. For example, the terminal can independently implement the AI-based audio generation method described below, or the terminal can send an audio generation request (including a target object and target text) to the server. The server executes the AI-based audio generation method according to the received audio generation request. In response to the audio generation request, the server performs synthesis processing based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain audio data that matches the timbre of the target object and corresponds to the target text, thereby realizing intelligent and accurate audio generation.

[0089] The electronic device for audio generation provided in this application can be various types of terminal devices or servers. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0090] Taking servers as an example, such as server clusters deployed in the cloud, AI as a Service (AIaaS) is provided to users. The AIaaS platform breaks down several common AI services and provides them as independent or packaged services in the cloud. This service model is similar to an AI-themed marketplace. All users can access and use one or more artificial intelligence services provided by the AIaaS platform through application programming interfaces.

[0091] For example, one type of AI cloud service could be an audio generation service, where a cloud server encapsulates the audio generation program provided in this application embodiment. Users invoke the audio generation service in the cloud service through a terminal (running a client, such as an audio client, in-vehicle client, etc.), causing the cloud-deployed server to call the encapsulated audio generation program. This program samples and performs attention processing on the audio data of the target object, obtaining the timbre embedding vector of the target object. Based on the timbre embedding vector of the target object and the content embedding vector of the target text, it performs synthesis processing to obtain audio data that matches the timbre of the target object and corresponds to the target text, thereby achieving intelligent and accurate audio generation.

[0092] As an application example, for a speaker client, the target audience could be a broadcaster on a radio platform who needs to regularly broadcast notices, life tips, etc., to residents in a community. For instance, the broadcaster inputs a target text into the speaker client, which needs to be converted into audio for broadcast to residents. Based on the broadcaster's timbre embedding vector and the target text's content embedding vector, a synthesis process is performed to obtain audio data that matches the broadcaster's timbre and corresponds to the target text, which is then broadcast to residents.

[0093] As another application example, for in-vehicle clients, when users are driving, it's inconvenient to access information via text, but they can still learn about it through audio, avoiding missing important information. For instance, if a leader sends a text message about an important meeting while the user is driving, requiring the user to read and process it promptly, the in-vehicle client, upon receiving the text, needs to convert it into audio to play for the user. Based on the leader's timbre embedding vector and the text's content embedding vector, a synthesis process is performed to obtain audio data that matches the leader's timbre and corresponds to the text, allowing the user to promptly access the leader's audio.

[0094] See Figure 1 , Figure 1 This is a schematic diagram of an application scenario for the audio generation system 10 provided in this application embodiment. The terminal 200 connects to the server 100 through the network 300, which can be a wide area network or a local area network, or a combination of the two.

[0095] Terminal 200 (running a client, such as an audio client, vehicle client, etc.) can be used to obtain audio generation requests. For example, if a user inputs a target object and target text through terminal 200, terminal 200 will automatically obtain multiple audio data of the target object and the target text, and automatically generate an audio generation request.

[0096] In some embodiments, an audio generation plugin can be embedded in the client running on the terminal to implement an AI-based audio generation method locally on the client. For example, after the terminal 200 obtains a request for audio generation (including a target object and target text), it calls the audio generation plugin to implement an AI-based audio generation method. The plugin samples and performs attention processing on the audio data of the target object to obtain the timbre embedding vector of the target object. Then, it performs synthesis processing based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain audio data that matches the timbre of the target object and corresponds to the target text. This achieves intelligent and accurate audio generation. For example, in a recording application, where users cannot perform high-quality personalized sound customization in non-recording studio scenarios, they can input a text to be recorded in the recording client. This text needs to be converted into personalized audio. Synthesis processing is then performed based on the personalized timbre embedding vector and the content embedding vector of the text to generate accurate personalized audio based on the accurate timbre embedding vector, thus achieving personalized sound customization in non-recording studio scenarios.

[0097] In some embodiments, after receiving an audio generation request, the terminal 200 calls the audio generation interface of the server 100 (which can be provided as a cloud service, i.e., an audio generation service). The server 100 samples and performs attention processing on the audio data of the target object to obtain the timbre embedding vector of the target object. Based on the timbre embedding vector of the target object and the content embedding vector of the target text, the server 100 performs synthesis processing to obtain audio data that matches the timbre of the target object and corresponds to the target text. The server 100 then sends the audio data to the terminal 200. For example, in a recording application, if a user cannot perform high-quality personalized sound customization in a non-recording studio setting, the user can input a text to be recorded in the terminal 200, and the server 100 will automatically generate an audio generation request and send it to the server 100. The server 100 will perform synthesis processing based on the personalized timbre embedding vector and the content embedding vector of the text, thereby generating accurate personalized audio based on the accurate timbre embedding vector. The server 100 will then send the generated personalized audio to the terminal 200 in response to the audio generation request, thus realizing personalized sound customization in a non-recording studio setting.

[0098] The structure of the electronic device for audio generation provided in the embodiments of this application is described below. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 for audio generation provided in an embodiment of this application. The example given is of the electronic device 500 being a server. Figure 2The illustrated electronic device 500 for audio generation includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 540.

[0099] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0100] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0101] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0102] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0103] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0104] In some embodiments, the audio generation apparatus provided in this application can be implemented in software, for example, as an audio generation plugin in the terminal described above, or as an audio generation service in the server described above. Of course, this is not limited to this; the audio generation apparatus provided in this application can be provided in various software embodiments, including various forms such as applications, software, software modules, scripts, or code.

[0105] Figure 2 An audio generation apparatus 555 stored in memory 550 is shown. It can be software in the form of programs and plugins, such as an audio generation plugin, and includes a series of modules, including a sampling module 5551, an audio encoding module 5552, an attention module 5553, a text encoding module 5554, a synthesis module 5555, and a training module 5556. The sampling module 5551, the audio encoding module 5552, the attention module 5553, the text encoding module 5554, and the synthesis module 5555 are used to implement the audio generation function provided in the embodiments of this application. The training module 5556 is used to train a neural network model. The audio generation method is implemented by calling the neural network model.

[0106] As previously stated, the AI-based audio generation method provided in this application can be implemented by various types of electronic devices. See also Figure 3 , Figure 3 This is a flowchart illustrating the artificial intelligence-based audio generation method provided in this application embodiment, combined with... Figure 3 The steps shown are explained.

[0107] In step 101, multiple audio data of the target object are sampled to obtain reference audio data of the target object.

[0108] As an example of obtaining the target object, the user inputs the target object and target text through the terminal. The terminal automatically generates an audio generation request and sends the audio generation request to the server. The server parses the audio generation request to obtain the identifier of the target object, and retrieves multiple audio data of the target object from the database based on the identifier of the target object. The server randomly samples the multiple audio data and uses the sampled audio data as the reference audio data of the target object. Thus, random sampling ensures that the reference embedding vector obtained later is independent of the content and sentence of the audio data.

[0109] For example, for any target object (i.e., speaker m), an audio sample is randomly taken from the corpus corresponding to the target object as a reference audio y. r =y Random(N) , where Random(N) represents any positive integer in [1,N], and N represents the total number of audio texts of the target object.

[0110] In step 102, the reference audio data of the target object is subjected to audio encoding processing to obtain the reference embedding vector of the reference audio data.

[0111] For example, the reference audio data of the target object can be used as the input of the reference encoder. The reference encoder performs audio encoding processing on the randomly sampled reference audio data (variable-length audio sequence) to obtain the reference embedding vector of the reference audio data. This allows subsequent attention processing based on the reference embedding vector to construct a more robust and accurate object embedding space, thereby improving the naturalness of the generated audio and its similarity to the target object.

[0112] In some embodiments, audio encoding processing is performed on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data, including: performing convolution processing on the reference audio data of the target object to obtain a convolution embedding vector of the reference audio data; and performing forward encoding processing on the convolution embedding vector of the reference audio data to obtain a reference embedding vector of the reference audio data.

[0113] like Figure 6 As shown, the reference audio data of the target object is convolved by the convolutional layers cascaded in the reference encoder to obtain the convolutional embedding vector of the reference audio data. Then, the convolutional embedding vector of the reference audio data is forward encoded by the recurrent neural network in the reference encoder to obtain the reference embedding vector of the reference audio data.

[0114] In some embodiments, audio encoding is implemented using a reference encoder, which includes multiple cascaded convolutional layers. Convolution processing is performed on the reference audio data of the target object to obtain a convolutional embedding vector of the reference audio data. This includes: performing convolution processing on the reference audio data of the target object through the first convolutional layer in the multiple cascaded convolutional layers; outputting the convolution result of the first convolutional layer to subsequent cascaded convolutional layers to continue convolution processing and outputting convolution results in subsequent cascaded convolutional layers until the last convolutional layer is reached, and using the convolution result output by the last convolutional layer as the convolutional embedding vector of the reference audio data.

[0115] like Figure 7 As shown, the first convolutional layer performs convolutional encoding on the reference audio data and outputs the convolution result to the second convolutional layer. The second convolutional layer continues to perform convolutional encoding and output the convolution result until it outputs to the Jth convolutional layer. The Jth convolutional layer performs convolutional encoding on the convolution result output by the (J-1)th convolutional layer to obtain the convolutional embedding vector of the reference audio data, where J is the total number of multiple cascaded convolutional layers and J is a positive integer greater than 1.

[0116] In some embodiments, audio encoding is implemented using a reference encoder, which includes a recurrent neural network. Forward encoding is performed on the convolutional embedding vector of the reference audio data to obtain a reference embedding vector, including: performing vector update processing based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain update information for the reference audio data; performing vector reset processing based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain reset information for the reference audio data; and performing context encoding processing on the convolutional embedding vector of the reference audio data based on the update information and the reset information for the reference audio data to obtain a reference embedding vector for the reference audio data.

[0117] For example, recurrent neural networks (RNNs) can solve long-dependency problems. An RNN includes two gates: an update gate and a reset gate. The update gate controls the extent to which the hidden vector from the previous state is incorporated into the current state; a larger update gate value indicates that more hidden vectors from the previous state are incorporated. The reset gate controls how much information from the previous state is written into the current candidate set; a smaller reset gate value indicates that less information from the previous state is written. By using the update gate in the RNN, combined with the hidden vector of the RNN and the convolutional embedding vector of the reference audio data, vector update processing is performed to obtain the updated information z of the reference audio data. t By using the reset gate in the recurrent neural network, and combining the hidden vector of the recurrent neural network with the convolutional embedding vector of the reference audio data, vector reset processing is performed to obtain the reset information r of the reference audio data. t Based on the update and reset information of the reference audio data, context encoding is performed on the convolutional embedding vector of the reference audio data to obtain the reference embedding vector y. t .

[0118] In some embodiments, vector update processing is performed based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain the update information of the reference audio data, including: concatenating the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain a first concatenated vector; and performing a mapping process based on an update gate on the first concatenated vector to obtain the update information of the reference audio data.

[0119] For example, the vector update process is shown in the following formula: z t =σ(W z ·[h t-1 ,x t ], where σ represents the activation function, W z This indicates updating the learnable parameters of the gate, h. t-1 Let x represent the hidden vector of a recurrent neural network.t This represents the convolutional embedding vector of the reference audio data, [.] indicates concatenation processing, and z t This indicates updated information about the reference audio data.

[0120] In some embodiments, vector reset processing is performed based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain reset information of the reference audio data, including: concatenating the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain a second concatenated vector; and performing a mapping process based on a reset gate on the second concatenated vector to obtain reset information of the reference audio data.

[0121] For example, the vector reset process is shown in the following formula: r t =σ(W r ·[h t-1 ,x t ], where σ represents the activation function, W r h represents the learnable parameters for resetting the gate. t-1 Let x represent the hidden vector of a recurrent neural network. t This represents the convolutional embedding vector of the reference audio data, [.] indicates concatenation processing, and r t This indicates the reset information for the reference audio data.

[0122] In some embodiments, context encoding is performed on the convolutional embedding vector of the reference audio data based on the update information and reset information of the reference audio data to obtain the reference embedding vector of the reference audio data. This includes: performing candidate vector-based mapping based on the reset information of the target object, the hidden vector of the recurrent neural network, and the convolutional embedding vector of the target object to obtain the candidate embedding vector of the reference audio data; and performing vector mapping based on the update information of the target object, the hidden vector of the recurrent neural network, and the candidate embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data.

[0123] For example, the mapping process based on candidate vectors is shown in the following formula: Among them, W r h represents the learnable parameter. t-1 Let x represent the hidden vector of a recurrent neural network. t This represents the convolutional embedding vector of the reference audio data, [.] indicates concatenation, * indicates matrix multiplication, and z t This indicates the update information for the reference audio data, r t This indicates the reset information for the reference audio data. This represents a candidate embedding vector for the reference audio data.

[0124] For example, the process of vector mapping is shown in the following formula: Among them, W o Let y represent the learnable parameters, σ represent the activation function, and y represent the activation function. t The reference embedding vector represents the reference audio data.

[0125] In step 103, timbre-based attention processing is performed on the reference embedding vector of the reference audio data to obtain the timbre embedding vector of the target object.

[0126] For example, after obtaining the reference embedding vector of the reference audio data, attention processing is performed based on the reference embedding vector to obtain the timbre embedding vector of the target object. The attention processing can construct a more robust and accurate object embedding space, so that the timbre embedding vector is separated from the audio content and text. The timbre embedding vector can accurately identify the timbre of the target object, thereby improving the naturalness of the generated audio and its similarity to the target object.

[0127] See Figure 4 , Figure 4 This is a flowchart illustrating the artificial intelligence-based audio generation method provided in an embodiment of this application. Figure 4 Show Figure 3 Step 103 can be implemented through steps 1031-1032: In step 1031, the reference embedding vector of the reference audio data is mapped based on the embedding space of the object timbre to obtain the weights of multiple object timbres; In step 1032, based on the weights of multiple object timbres, the embedding vectors of multiple object timbres are weighted and summed to obtain the timbre embedding vector of the target object.

[0128] like Figure 8 As shown, when there are 4 objects (i.e., object timbres) in the embedding space based on object timbres, namely A, B, C, and D, the reference embedding vector of the reference audio data is mapped based on the embedding space of object timbres to obtain the weights of object timbres A (0.3), object timbres B (0.4), object timbres C (0.1), and object timbres D (0.2). Based on the weights of the 4 object timbres, the embedding vectors of the 4 object timbres are weighted and summed to obtain the timbre embedding vector of the target object (i.e., the speaker embedding vector).

[0129] In step 104, the target text is processed by text encoding to obtain the content embedding vector of the target text.

[0130] As an example of obtaining target text, the user inputs the target object and target text through the terminal. The terminal automatically generates an audio generation request and sends the audio generation request to the server. The server parses the audio generation request to obtain the target text, and performs text encoding on the target text through a text encoder to obtain the content embedding vector of the target text. This vector is then combined with the timbre embedding vector for audio synthesis to achieve personalized audio customization.

[0131] In step 105, the timbre embedding vector of the target object and the content embedding vector of the target text are synthesized to obtain audio data that matches the timbre of the target object and corresponds to the target text.

[0132] For example, since timbre embedding vectors can accurately represent the timbre of a target object, audio synthesis based on the timbre embedding vector of the target object and the content embedding vector of the target text can produce audio data that matches the timbre of the target object and corresponds to the target text. The synthesized audio data is similar to the real audio data of the target object, making the synthesized audio data more realistic.

[0133] In some embodiments, the synthesis process is performed based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain audio data that matches the timbre of the target object and corresponds to the target text. This includes: aligning the timbre embedding vector of the target object and the content embedding vector of the target text to obtain an alignment vector of the target text; decoding the alignment vector of the target text to obtain an acoustic feature sequence of the target text; and performing acoustic feature transformation on the acoustic feature sequence to obtain audio data that matches the timbre of the target object and corresponds to the target text.

[0134] For example, the alignment vector represents the probability (or degree of attention) of the current acoustic features for each character in the input target text, characterizing the alignment relationship between the input target text sequence and the output audio sequence. Alignment prediction is performed based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain the alignment vector of the target text. Decoding is then performed based on this alignment vector to obtain the acoustic feature sequence of the target text. A vocoder is used to perform acoustic feature transformation on the acoustic feature sequence to obtain audio data that matches the timbre of the target object and corresponds to the target text. In other words, the acoustic feature sequence is converted into synthesized speech data that matches the input target text sequence. Specifically, the acoustic feature sequence can be a Mel-Spectrogram sequence. The acoustic feature corresponding to each phoneme in the acoustic feature sequence is the acoustic feature of the target object. For example, if the length of the target text sequence is 100 (i.e., it includes 100 characters), then the acoustic feature corresponding to each character can be determined as the acoustic feature of the target object. Then, the acoustic features corresponding to the 100 characters can be combined into an acoustic feature sequence. When the acoustic feature is a 1*5 dimensional vector, then the acoustic feature sequence includes 100 1*5 dimensional vectors, which can be combined into a 100*5 dimensional vector.

[0135] Specifically, the vocoder can be a WaveGlow network (a network that synthesizes high-quality speech from Mel spectrograms by relying on streams), which can achieve parallel speech synthesis; or it can be a SqueezeWave network (a lightweight streaming model that can be used for speech synthesis on mobile devices), which can effectively improve the speed of speech synthesis; or it can use vocoders such as Griffin-Lim, WaveNet, or Parallel to synthesize speech from acoustic feature sequences. The appropriate vocoder can be selected according to actual needs, and this application does not limit this.

[0136] Following the example above, the acoustic feature conversion process is as follows: the acoustic feature sequence is smoothed to obtain the corresponding spectral data; the corresponding spectral data is subjected to Fourier transform to obtain audio data that matches the timbre of the target object and corresponds to the target text.

[0137] See Figure 5 , Figure 5 This is a flowchart illustrating an artificial intelligence-based audio generation method provided in an embodiment of this application. The audio generation method is implemented by calling a neural network model. Figure 5 Show Figure 3The training process of the neural network model is also included, namely, training steps 106-109: In step 106, the reference audio data of the object sample is subjected to timbre-based attention processing through the initialized neural network model to obtain the timbre embedding vector of the object sample; in step 107, object prediction processing is performed based on the timbre embedding vector of the object sample to obtain the predicted object of the reference audio data; in step 108, the first loss function of the neural network model is constructed based on the predicted object of the reference audio data and the object label of the object sample; in step 109, the parameters of the neural network model are updated based on the first loss function, and the updated parameters of the neural network model are used as the parameters of the trained neural network model.

[0138] For example, the neural network model includes a reference encoder and a first classifier. To obtain timbre embedding information independent of audio content and sentences, and to increase the discriminability of different objects, a classifier is further added on top of the timbre embedding vector. Multiple audio data of the object sample are sampled to obtain reference audio data. The reference encoder performs audio encoding on the reference audio data to obtain a reference embedding vector. Timbre-based attention processing is applied to the reference embedding vector to obtain the timbre embedding vector of the object sample. The first classifier performs object prediction processing on the timbre embedding vector to obtain the predicted object of the reference audio data (i.e., the probability of predicting the speaker). After determining the value of the first loss function of the neural network model based on the predicted object of the reference audio data and the object label of the object sample (i.e., the real speaker label), it can be determined whether the value of the first loss function exceeds a preset threshold. When the value of the first loss function exceeds the preset threshold, the error signal of the neural network model is determined based on the first loss function. The error information is backpropagated in the neural network model, and the model parameters of each layer are updated during the propagation process.

[0139] As an example, the formula for calculating the first loss function is: Where N represents the number of audio texts in object sample m. Represented as the object probability predicted based on the timbre embedding vector. Let L represent the object label, CE represent the cross-entropy loss, and L represent the cross-entropy loss. spk This represents the first loss function.

[0140] Here, we explain backpropagation. Training sample data is input into the input layer of the neural network model, passes through the hidden layers, and finally reaches the output layer to output the result. This is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated and backpropagated from the output layer to the hidden layers until it reaches the input layer. During the backpropagation process, the values ​​of the model parameters are adjusted according to the error. That is, a loss function is constructed based on the error between the output result and the actual value, and the partial derivatives of the loss function with respect to the model parameters are calculated layer by layer to generate the gradient of the loss function with respect to the model parameters of each layer. Since the direction of the gradient indicates the direction of error amplification, the gradient of the model parameters is inverted and summed with the original parameters of each layer. The summation result is used as the updated model parameters of each layer, thereby reducing the error caused by the model parameters. The above process is iterated until convergence.

[0141] In some embodiments, before updating the parameters of the neural network model, at least one of the following processes is performed on the initialized neural network model: text encoding is performed on the text samples to obtain the content embedding vector of the text samples, and a second loss function of the neural network model is constructed based on the content embedding vector of the text samples; a third loss function of the neural network model is constructed based on the timbre embedding vector of the object samples and the content embedding vector of the text samples; wherein at least one of the second loss function and the third loss function is used to combine with the first loss function (the combination method may be summation, or weighted summation based on attention mechanism, etc.) to update the parameters of the neural network model.

[0142] For example, a neural network model also includes a text encoder and a second classifier. To maximize the information that can be shared between different objects—that is, the text encoder is shared by all objects—an adversarial training mechanism is introduced at the text encoder end. This involves adding a classifier with a gradient reversal layer after the text encoder to prevent the text encoding from capturing object information. The text encoder processes the text samples to obtain the content embedding vectors. The second classifier then performs object prediction processing on these content embedding vectors to obtain the predicted object of the text sample. Based on the predicted object and the object label of the object sample, a fourth loss function is constructed. This fourth loss function is then inverted to obtain the second loss function of the neural network model.

[0143] As an example, the formula for calculating the fourth loss function is: Where μ represents the scaling factor, and N represents the number of audio texts in object sample m. Represented as the object probability predicted based on the text embedding vector. Let L represent the object label, CE represent the cross-entropy loss, and the formula for calculating the second loss function is L. txt =-L1.

[0144] For example, to help speech synthesis better learn the alignment relationship between text and audio with limited data, the neural network model is trained using existing relationships between text and audio, introducing two different pre-alignment information between text and audio. Based on the timbre embedding vector of the object sample and the content embedding vector of the text sample, synthesis processing is performed to obtain audio data samples that match the timbre of the object sample and correspond to the text sample. First, alignment prediction processing is performed based on the timbre embedding vector of the object sample and the content embedding vector of the text sample to obtain the predicted alignment information α of the text sample. t′t Then, forced alignment processing based on speech recognition is performed on the timbre embedding vector of the object sample and the content embedding vector of the text sample to obtain the forced alignment information of the text sample. Linear mapping is performed based on the number of characters in the text sample and the number of audio frames in the audio data sample to obtain the linear alignment information A′ of the text sample. t′t Based on the predicted alignment information, forced alignment information, and linear alignment information of text samples, a third loss function for the neural network model is constructed.

[0145] Among them, the linear alignment information A′ of the text sample t′t This assumes a near-linear, monotonic correspondence between text and audio (i.e., an alignment graph where the input text sequence and the output audio sequence are aligned diagonally). The formula for calculating linear alignment information is: Where T′ represents the maximum number of input texts, t′ represents the t′th input text, T represents the maximum number of output audios, t represents the t-th output audio frame, and g represents the scaling factor.

[0146] Following the example above, a third loss function for the neural network model is constructed based on the predicted alignment information, the forced alignment information, and the linear alignment information of the text samples. This includes: constructing a first difference information based on the difference between the linear alignment information and the predicted alignment information of the text samples; constructing a second difference information based on the difference between the forced alignment information and the predicted alignment information of the text samples; and performing a weighted summation of the first and second difference information to obtain the third loss function for the neural network model.

[0147] For example, the first difference information is A′ t′t -α t′t The second difference information is The formula for calculating the third loss function is as follows: Where, αt′t This indicates the alignment information between the predicted text and the audio. This represents the alignment information between text and audio obtained through Automatic Speech Recognition (ASR). If α t′t With A′ t′t , or α t′t and Significant discrepancies will result in severe penalties. This training strategy based on multi-alignment information can, on the one hand, obtain the matrix by introducing it into ASR. To avoid sensitivity to the scaling factor g, and on the other hand, it can be achieved through A′ t′t This is to avoid the impact of large alignment deviations in forced alignment.

[0148] The following will describe an exemplary application of the embodiments of this application in a real-world speech synthesis application scenario.

[0149] This application can be applied to various speech synthesis application scenarios (e.g., smart devices with speech synthesis capabilities such as smart speakers, smartwatches, smartphones, smart homes, smart maps, and smart cars; applications with speech synthesis capabilities such as news, audiobooks, online education, smart robots, AI customer service, and speech synthesis cloud services). For example, in in-vehicle applications, when users are driving, it is inconvenient to understand information through text, but they can understand information by reading speech to avoid missing important information. When the in-vehicle client receives the text, it needs to convert the text into speech to play the speech to the user so that the user can read the speech corresponding to the text in a timely manner.

[0150] To address the problems existing in customized acoustic modeling methods for speech synthesis in related technologies, this application proposes an artificial intelligence-based audio generation method. This method can separate speaker information (i.e., object information) and text content information during the modeling process. The speaker information is extracted using an audio encoder with a random sampling mechanism and a speaker classifier to ensure that the obtained speaker information does not contain prosody or style-related information. The method proposes an adversarial training strategy to deconstruct speaker-related information from the text encoder. To make customized training faster and more effective, the correlation between text and audio is fully utilized, and an attention loss based on multi-alignment information is further proposed to assist the model in learning.

[0151] To enable rapid, accurate, stable, and efficient customization of acoustic models for speech synthesis, this application provides an artificial intelligence-based audio generation method. For example... Figure 9 As shown, Figure 9This is a training flowchart provided in an embodiment of this application, which mainly includes three steps:

[0152] Step 11: Data preparation, including text preprocessing, acoustic feature extraction, and phoneme duration information extraction.

[0153] Step 12: Use the given data to train a multi-speaker speech synthesis acoustic model (i.e., a neural network model) as the initial model for training the custom acoustic model.

[0154] Step 13: Using the data of the given target speaker, train a customized speech synthesis model for the target speaker using the method provided in the embodiments of this application.

[0155] like Figure 10 As shown, Figure 10 This is a schematic diagram of a fast and efficient customized speech synthesis model structure provided in an embodiment of this application. The structure mainly includes a speaker encoder with a random sampling mechanism, a speaker attention module, and a speaker classifier; a text encoder with speaker adversarial training; and an end-to-end acoustic model based on sequence-to-sequence encoding and decoding guided by a multi-alignment mechanism.

[0156] like Figure 10 As shown, the first part is the speaker encoder. To enable the speaker encoder to focus more on characterizing the speaker's timbre while ignoring audio content-related information such as rhythm and style, this application provides a speaker encoder (i.e., an audio encoder) with a random sampling mechanism. For a given text and audio pair... Where N represents the number of audio texts of the speaker (i.e. the object). First, based on the random sampling mechanism, an audio sample is randomly sampled from the corpus corresponding to speaker m as the input of the reference encoder, as shown in formula (1):

[0157] y r =y Random(N) (1)

[0158] Where Random(N) represents any positive integer in [1,N], y r This represents an audio sample randomly sampled from the corpus corresponding to speaker m, i.e., a reference audio. It can be modeled based on characters or phonemes, with a maximum sequence length of 256.

[0159] Among them, the sampled audio y r The audio sequence y is directly fed into the reference encoder for encoding. The reference encoder encodes the variable-length audio sequence y. r Encode the data and output the corresponding reference embedding vector. For example, ... Figure 11 As shown, Figure 11 This is a block diagram of the specific structure of the reference encoder provided in the embodiments of this application. The input reference audio is encoded by multiple CNN layers (e.g., 6 convolutional layers) and then fed into a unidirectional GRU (e.g., a gated recurrent grid with 128 nodes). The representation of the unidirectional GRU at the last moment is then used as the reference embedding vector.

[0160] The aforementioned reference embedding vector can already be directly used as the speaker embedding vector to control the speaker's timbre. To construct a more robust and accurate speaker embedding space, embodiments of this application further introduce a speaker attention layer, such as... Figure 10 As shown, the speaker embedding space consists of M different types of speaker tokens (i.e., different objects). Each speaker m embedding representation can be represented by a linear combination of these M speaker tokens. This further diversification of the speaker embedding space representation can improve the robustness of the speaker embedding space representation, thereby improving the naturalness of the generated audio and the speaker similarity.

[0161] To obtain speaker embedding information independent of audio content and sentences, and to enhance the discriminability between different speakers, a speaker classifier was further added to the speaker embedding vectors. For example... Figure 5 As shown, for audio y randomly sampled from speaker m∈[1,S] r Where S represents the number of speakers, and the speaker loss function can be expressed as the speaker probability predicted based on the speaker embedding vector (i.e., timbre embedding vector). (i.e., the probability of predicting the target) and target speaker labels The cross-entropy (CE) loss (i.e., the first loss function) between them is shown in Equation (2):

[0162]

[0163] Where N represents the number of text segments in speaker m's audio, and L... spk This represents the speaker loss function, which helps to obtain a consistent speaker embedding vector from different audio recordings of the same speaker.

[0164] like Figure 10As shown, the second part is a text encoder based on speaker adversarial training. In order to maximize the information that can be shared between different speakers, that is, the text encoder is shared by all speakers, the embodiments of this application introduce an adversarial training mechanism at the text encoder end, that is, add a speaker classifier with a gradient reversal layer after the text encoder to prevent the text encoding from capturing speaker information, as shown in formula (3):

[0165]

[0166] Where μ represents the scaling factor, and N represents the number of text segments in speaker m's audio. This is expressed as the speaker probability predicted based on the text embedding vector. Let L represent the target speaker label, CE represent the cross-entropy loss, and L represent the target speaker label. txt The second loss function is represented by the adversarial training method, which speeds up training because it is not necessary to learn the text encoder parameters based on each speaker.

[0167] like Figure 10 As shown, the third part is a training strategy guided by multiple alignment information. In order to help the customized speech synthesis model learn the alignment relationship between text and audio better with a small amount of data, this application embodiment utilizes some relationships between text and audio to train the model, that is, it introduces two different pre-alignment information between text and audio, one of which comes from the forced alignment in speech recognition (i.e., the matrix in formula (5)). Another type of linear alignment information assumes a near-linear, monotonic correspondence between text and audio (i.e., an alignment diagram where the input text sequence and the output audio sequence are aligned diagonally). The linear alignment information is shown in formula (4):

[0168]

[0169] Where T′ represents the maximum number of input texts, t′ represents the t′th input text, T represents the maximum number of output acoustic feature frames, t represents the t-th output acoustic feature frame, and g represents the scaling factor (e.g., 0.2).

[0170] The attention loss function for multi-alignment information (i.e., the third loss function) is shown in Equation (5):

[0171]

[0172] Where, α t′t This represents the alignment information between text and audio learned by the acoustic model. This represents the alignment information between text and audio obtained through Automatic Speech Recognition (ASR). If α t′t With A′ t′t , or α t′t and Significant discrepancies will result in severe penalties. This training strategy based on multi-alignment information can, on the one hand, obtain the matrix by introducing it into ASR. To avoid the sensitivity of parameter g in formula (4), and on the other hand, it can be achieved through A′ t′t This is to avoid the impact of large alignment deviations in ASR forced alignment.

[0173] The method proposed in this application was tested on a recorded Chinese speech synthesis corpus, using approximately 120 hours of Mandarin Chinese speech data from 60 speakers for training. This application uses the Mean Opinion Score (MOS) of naturalness and similarity as the final evaluation metrics.

[0174] For speaker embedding space verification, three different speaker embedding space modeling methods were compared, including the one-hot representation of related techniques, the speaker encoding method of related techniques, and the speaker encoder (SE) proposed in the embodiments of this application. A target speaker was randomly selected for ablation studies. The experiments show that the results generated using the speaker encoding method of related techniques are unstable, and the synthesized text does not match the content of the reference audio.

[0175] An A / B preference test was conducted based on the one-hot representation of related technologies using 20 audio pairs (excluding those in the training set) and the speaker encoder (SE) proposed in this application. The results of the A / B preference test are shown in Table 1.

[0176] Table 1 AB Preference Test

[0177] Model SE One-hot AB preference test 56.1％ 20.5％

[0178] As shown in Table 1, the AI-based audio generation method proposed in this application can obtain more preferences because the extracted speaker embedding vector contains speaker feature-related information that helps decode the acoustic features of the corresponding speaker. Simultaneously, a speaker classifier is added to the speaker encoder (SE) to ensure that the speaker embedding vector has better discriminative power for different speakers, thereby better controlling the speaker features in the subsequent decoding process. Furthermore, the speaker embedding vector proposed in this application can be pre-calculated offline, thus incurring no additional computational cost during inference.

[0179] Based on the speaker encoder (SE), an adversarial training (SE+AT) is introduced to eliminate speaker information in the text encoder. The A / B preference test results are shown in Table 2.

[0180] Table 2 AB Preference Test

[0181] Model SE+AT SE AB preference test 48.2％ 20.3％

[0182] As shown in Table 2, increasing adversarial training can further improve the test results, because adversarial training can maximize the sharing of information between speakers.

[0183] The test results of this application embodiment on a very small corpus (20 sentences per speaker) are shown in Table 2. The baseline model adopts the current mainstream speech synthesis customization method. The speech synthesis customization model proposed in this application embodiment is as follows:

[0184] Table 3. MOS values ​​for speech synthesis acoustic modeling with very limited corpus data.

[0185]

[0186] As shown in Table 3, the AI-based audio generation method proposed in this application achieves better performance than the Baseline system.

[0187] Meanwhile, the embodiments of this application have also achieved significant performance improvements and faster convergence speeds under different speakers and different corpus sizes, as shown in Table 4:

[0188] Table 4. MOS values ​​for acoustic modeling of speech synthesis under different speakers and corpus sizes.

[0189]

[0190] As can be seen from the table above, the AI-based audio generation method proposed in this application has achieved higher naturalness and similarity than related technologies in the recorded speech synthesis corpus, under different speakers and different corpus sizes. The synthesized speech has better clarity and naturalness, and the spectral details of the synthesized speech are also clearer. In addition, it greatly shortens the training convergence time of the customized synthesis model.

[0191] In summary, the artificial intelligence-based audio generation method proposed in this application has the following beneficial effects:

[0192] 1) Randomly sample an audio file from all the audio data as the speaker encoder information to ensure that the obtained speaker embedding vector is independent of the audio content and sentence. In addition, a speaker classifier is introduced to ensure better distinguishability for different speakers.

[0193] 2) In order to better learn speaker-independent text, a speaker adversarial training mechanism is introduced, which makes the text encoder unable to distinguish which speaker the text belongs to, so as to maximize the information that can be shared between different speakers.

[0194] 3) By fully utilizing the correlation between training text and audio, a loss function with a multi-alignment mechanism is introduced, which effectively improves the convergence speed and stability of the model.

[0195] This concludes the description of the AI-based audio generation method provided in this application, using the exemplary application and implementation of the server provided in the embodiments of this application. This application also provides an audio generation apparatus. In practical applications, the functional modules in the audio generation apparatus can be collaboratively implemented using the hardware resources of electronic devices (such as terminal devices, servers, or server clusters), such as computing resources like processors, communication resources (such as those used to support various communication methods like fiber optic cables and cellular networks), and memory. Figure 2 An audio generation device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, such as software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules, application programming interfaces, plug-ins, cloud services, etc. in large software systems. Examples of different implementation methods are given below.

[0196] Example 1: Audio generation devices are mobile applications and modules.

[0197] The audio generation device 555 in this embodiment can provide a software module designed using programming languages ​​such as C / C++ and Java, which can be embedded into various mobile applications based on systems such as Android or iOS (stored as executable instructions in the storage medium of the mobile device and executed by the processor of the mobile device), thereby directly using the computing resources of the mobile device itself to complete the relevant audio synthesis tasks, and periodically or irregularly transmitting the processing results to a remote server through various network communication methods, or saving them locally on the mobile device.

[0198] Example 2: The audio generation device is a server application and platform.

[0199] The audio generation device 555 in this embodiment can be a dedicated software module in an application software or large software system designed using programming languages ​​such as C / C++ and Java. It runs on the server side (stored in the server-side storage medium in the form of executable instructions and executed by the server-side processor). The server uses its own computing resources to complete the relevant audio synthesis tasks.

[0200] This application embodiment can also provide an audio synthesis platform (for audio synthesis) for use by individuals, groups or organizations, which is formed by mounting a customized, easy-to-interact web interface or other user interfaces (UI) on a distributed, parallel computing platform composed of multiple servers.

[0201] Example 3: The audio generation device is a server-side application programming interface (API) and plugin.

[0202] The audio generation device 555 in this application embodiment can be provided as a server-side API or plugin for users to call, to execute the AI-based audio generation method of this application embodiment, and to be embedded in various applications.

[0203] Example 4: Audio generation devices are mobile device client APIs and plugins.

[0204] The audio generation device 555 in this application embodiment can be provided as an API or plugin on a mobile device for users to call in order to execute the artificial intelligence-based audio generation method of this application embodiment.

[0205] Example 5: The audio generation device is an open service in the cloud.

[0206] The audio generation device 555 in this embodiment can provide cloud services for information recommendation developed for users, allowing individuals, groups, or organizations to obtain audio.

[0207] The audio generation device 555 includes a series of modules, including a sampling module 5551, an audio encoding module 5552, an attention module 5553, a text encoding module 5554, a synthesis module 5555, and a training module 5556. The following description further illustrates how the various modules in the audio generation device 555 provided in this embodiment cooperate to implement the audio generation scheme.

[0208] The sampling module 5551 is used to sample multiple audio data of the target object to obtain reference audio data of the target object; the audio encoding module 5552 is used to perform audio encoding processing on the reference audio data of the target object to obtain a reference embedding vector of the reference audio data; the attention module 5553 is used to perform timbre-based attention processing on the reference embedding vector of the reference audio data to obtain a timbre embedding vector of the target object; the text encoding module 5554 is used to perform text encoding processing on the target text to obtain a content embedding vector of the target text; and the synthesis module 5555 is used to perform synthesis processing based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain audio data that conforms to the timbre of the target object and corresponds to the target text.

[0209] In some embodiments, the attention module 5553 is further configured to perform object timbre-based embedding space mapping processing on the reference embedding vector of the reference audio data to obtain weights of multiple object timbres; and to perform weighted summation processing on the embedding vectors of multiple object timbres based on the weights of the multiple object timbres to obtain the timbre embedding vector of the target object.

[0210] In some embodiments, the audio encoding module 5552 is further configured to perform convolution processing on the reference audio data of the target object to obtain the convolution embedding vector of the reference audio data; and to perform forward encoding processing on the convolution embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data.

[0211] In some embodiments, the audio encoding is implemented using a reference encoder, which includes multiple cascaded convolutional layers. The audio encoding module 5552 is further configured to perform convolution processing on the reference audio data of the target object through the first convolutional layer in the multiple cascaded convolutional layers; output the convolution result of the first convolutional layer to subsequent cascaded convolutional layers to continue convolution processing and output convolution results in the subsequent cascaded convolutional layers until the last convolutional layer is output, and use the convolution result output by the last convolutional layer as the convolutional embedding vector of the reference audio data.

[0212] In some embodiments, the audio encoding is implemented using a reference encoder, which includes a recurrent neural network. The audio encoding module 5552 is further configured to perform vector update processing based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain update information of the reference audio data; perform vector reset processing based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain reset information of the reference audio data; and perform context encoding processing on the convolutional embedding vector of the reference audio data based on the update information and the reset information of the reference audio data to obtain a reference embedding vector of the reference audio data.

[0213] In some embodiments, the audio encoding module 5552 is further configured to concatenate the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain a concatenated vector; and to perform a mapping process based on an update gate on the concatenated vector to obtain the update information of the reference audio data.

[0214] In some embodiments, the audio encoding module 5552 is further configured to perform candidate vector-based mapping processing based on the reset information of the target object, the hidden vector of the recurrent neural network, and the convolutional embedding vector of the target object to obtain a candidate embedding vector of the reference audio data; and to perform vector mapping processing based on the update information of the target object, the hidden vector of the recurrent neural network, and the candidate embedding vector of the reference audio data to obtain a reference embedding vector of the reference audio data.

[0215] In some embodiments, the synthesis module 5555 is further configured to align the timbre embedding vector of the target object and the content embedding vector of the target text to obtain the alignment vector of the target text; perform decoding processing based on the alignment vector of the target text to obtain the acoustic feature sequence of the target text; and perform acoustic feature transformation on the acoustic feature sequence to obtain audio data that conforms to the timbre of the target object and corresponds to the target text.

[0216] In some embodiments, the synthesis module 5555 is further configured to smooth the acoustic feature sequence to obtain corresponding spectral data; and to perform Fourier transform on the corresponding spectral data to obtain audio data that conforms to the timbre of the target object and corresponds to the target text.

[0217] In some embodiments, the audio generation method is implemented by calling a neural network model; the apparatus further includes: a training module 5556, configured to perform timbre-based attention processing on reference audio data of an object sample using the initialized neural network model to obtain a timbre embedding vector of the object sample; perform object prediction processing based on the timbre embedding vector of the object sample to obtain a predicted object of the reference audio data; construct a first loss function of the neural network model based on the predicted object of the reference audio data and the object label of the object sample; update the parameters of the neural network model based on the first loss function, and use the updated parameters of the neural network model as the parameters of the trained neural network model.

[0218] In some embodiments, the training module 5556 is further configured to perform at least one of the following processes on the initialized neural network model: perform text encoding processing on the text sample to obtain the content embedding vector of the text sample, and construct a second loss function of the neural network model based on the content embedding vector of the text sample; construct a third loss function of the neural network model based on the timbre embedding vector of the object sample and the content embedding vector of the text sample; wherein at least one of the second loss function and the third loss function is used to combine with the first loss function to update the parameters of the neural network model.

[0219] In some embodiments, the training module 5556 is further configured to perform object prediction processing based on the content embedding vector of the text sample to obtain the predicted object of the text sample; construct a fourth loss function based on the predicted object of the text sample and the object label of the object sample; and invert the fourth loss function to obtain the second loss function of the neural network model.

[0220] In some embodiments, the training module 5556 is further configured to perform synthesis processing based on the timbre embedding vector of the object sample and the content embedding vector of the text sample to obtain an audio data sample that conforms to the timbre of the object sample and corresponds to the text sample; perform alignment prediction processing based on the timbre embedding vector of the object sample and the content embedding vector of the text sample to obtain the predicted alignment information of the text sample; perform forced alignment processing on the timbre embedding vector of the object sample and the content embedding vector of the text sample to obtain the forced alignment information of the text sample; perform linear mapping processing based on the number of characters of the text sample and the number of audio frames of the audio data sample to obtain the linear alignment information of the text sample; and construct a third loss function of the neural network model based on the predicted alignment information, the forced alignment information, and the linear alignment information of the text sample.

[0221] In some embodiments, the training module 5556 is further configured to construct a first difference information based on the difference between the linear alignment information of the text sample and the predicted alignment information based on the text sample; construct a second difference information based on the difference between the forced alignment information of the text sample and the predicted alignment information based on the text sample; and perform weighted summation processing on the first difference information and the second difference information to obtain a third loss function of the neural network model.

[0222] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the artificial intelligence-based audio generation method described above in this application.

[0223] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the artificial intelligence-based audio generation method provided in this application. For example... Figures 3-5 The example shown is an AI-based audio generation method.

[0224] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0225] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0226] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0227] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0228] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An audio generation method based on artificial intelligence, characterized in that, The method includes: A variable-length audio sequence is randomly sampled from multiple audio data of the target object and used as reference audio data of the target object; The reference audio data of the target object is processed by a reference encoder to obtain a reference embedding vector of the reference audio data, which is used as an embedding vector to control the timbre of the target object. The audio encoding process includes encoding the reference audio data in a multi-layer convolutional network and then inputting it into a one-way gated recurrent network. The representation of the one-way gated recurrent network at the last moment is used as the reference embedding vector. The reference encoder is trained by a first classifier, which is used to predict the timbre embedding vector obtained by the reference encoder from the object sample encoding, and to determine an error signal based on the predicted object and the object label of the object sample. The error signal is used to update the parameters of the reference encoder. The reference embedding vector of the reference audio data is subjected to timbre-based attention processing to obtain the timbre embedding vector of the target object in the timbre embedding space; wherein, the attention processing is performed in the speaker embedding space, which is composed of M different types of speaker tokens, and the embedding of the target object is represented as a linear combination of the M different types of speaker tokens; The target text is encoded by a text encoder to obtain the content embedding vector of the target text. The text encoder is obtained by adversarial training of a second classifier. The second classifier is equipped with a gradient inversion layer to prevent the text encoder from obtaining the object information of the target object during the adversarial training. The text encoder is shared by all objects. Based on the timbre embedding vector of the target object and the content embedding vector of the target text, a synthesis process is performed to obtain audio data that matches the timbre of the target object and corresponds to the target text. The synthesis process is performed by a neural network model trained as follows: based on the timbre embedding vector of the object sample and the content embedding vector of the text sample, predicted alignment information, forced alignment information, and linear alignment information are obtained; a first difference information is constructed based on the difference between the linear alignment information and the predicted alignment information; a second difference information is constructed based on the difference between the forced alignment information and the predicted alignment information; a third loss function is obtained by weighted summation of the first difference information and the second difference information; and the neural network model is trained based on the third loss function.

2. The method according to claim 1, characterized in that, The step of performing timbre-based attention processing on the reference embedding vector of the reference audio data to obtain the timbre embedding vector corresponding to the target object in the timbre embedding space includes: The reference embedding vector of the reference audio data is mapped based on the embedding space of the object timbre to obtain the weights of multiple object timbres; Based on the weights of multiple object timbres, the embedding vectors of multiple object timbres are weighted and summed to obtain the timbre embedding vector of the target object.

3. The method according to claim 1, characterized in that, The step of performing audio encoding processing on the reference audio data of the target object to obtain the reference embedding vector of the reference audio data includes: The reference audio data of the target object is convolved to obtain the convolutional embedding vector of the reference audio data. The convolutional embedding vector of the reference audio data is forward encoded to obtain the reference embedding vector of the reference audio data.

4. The method according to claim 3, characterized in that, The audio encoding is implemented through the reference encoder, which includes multiple cascaded convolutional layers; The step of performing convolution processing on the reference audio data of the target object to obtain the convolution embedding vector of the reference audio data includes: The reference audio data of the target object is convolved through the first convolutional layer in the plurality of cascaded convolutional layers. The convolution result of the first convolutional layer is output to subsequent cascaded convolutional layers, so that convolution processing and convolution result output continue in the subsequent cascaded convolutional layers until the output is reached in the last convolutional layer. The convolution result output from the last convolutional layer is used as the convolutional embedding vector of the reference audio data.

5. The method according to claim 3, characterized in that, The audio encoding is implemented through the reference encoder, which includes a recurrent neural network; The step of performing forward encoding on the convolutional embedding vector of the reference audio data to obtain the reference embedding vector of the reference audio data includes: Vector update processing is performed based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain the update information of the reference audio data; Vector reset processing is performed based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain the reset information of the reference audio data; Based on the update information and reset information of the reference audio data, the convolutional embedding vector of the reference audio data is subjected to context encoding to obtain the reference embedding vector of the reference audio data.

6. The method according to claim 5, characterized in that, The vector update process based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data is performed to obtain the update information of the reference audio data, including: The hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data are concatenated to obtain a concatenated vector; The spliced ​​vector is subjected to an update gate-based mapping process to obtain the update information of the reference audio data.

7. The method according to claim 5, characterized in that, The step of performing context encoding on the convolutional embedding vector of the reference audio data based on the update information and reset information of the reference audio data to obtain the reference embedding vector of the reference audio data includes: Based on the reset information of the target object, the hidden vector of the recurrent neural network, and the convolutional embedding vector of the target object, a candidate vector-based mapping process is performed to obtain the candidate embedding vector of the reference audio data. Based on the updated information of the target object, the hidden vector of the recurrent neural network, and the candidate embedding vector of the reference audio data, vector mapping processing is performed to obtain the reference embedding vector of the reference audio data.

8. The method according to claim 1, characterized in that, The synthesis process based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain audio data that matches the timbre of the target object and corresponds to the target text includes: The neural network model trained based on the third loss function is used to align the timbre embedding vector of the target object and the content embedding vector of the target text to obtain the alignment vector of the target text. Based on the alignment vector of the target text, the acoustic feature sequence of the target text is obtained by decoding. The acoustic feature sequence is subjected to acoustic feature transformation to obtain audio data that matches the timbre of the target object and corresponds to the target text.

9. The method according to claim 8, characterized in that, The step of performing acoustic feature transformation on the acoustic feature sequence to obtain audio data that matches the timbre of the target object and corresponds to the target text includes: The acoustic feature sequence is smoothed to obtain the corresponding spectral data; Perform a Fourier transform on the corresponding spectral data to obtain audio data that matches the timbre of the target object and corresponds to the target text.

10. The method according to any one of claims 1-9, characterized in that, Training the neural network model based on the third loss function includes: The reference audio data of the object sample is subjected to timbre-based attention processing by the initialized neural network model to obtain the timbre embedding vector of the object sample. Based on the timbre embedding vector of the object sample, object prediction processing is performed to obtain the predicted object of the reference audio data; based on the predicted object of the reference audio data and the object label of the object sample, the first loss function of the neural network model is constructed. The parameters of the neural network model are updated based on the first loss function, and the updated parameters of the neural network model are used as the parameters of the trained neural network model. Perform at least one of the following processes on the initialized neural network model: The text sample is subjected to text encoding processing to obtain the content embedding vector of the text sample, and The second loss function of the neural network model is constructed based on the content embedding vector of the text sample; Based on the timbre embedding vector of the object sample and the content embedding vector of the text sample, a third loss function of the neural network model is constructed. The second loss function and the third loss function are used in combination with the first loss function to update the parameters of the neural network model.

11. The method according to claim 10, characterized in that, The second loss function for constructing the neural network model based on the content embedding vector of the text sample includes: Based on the content embedding vector of the text sample, object prediction processing is performed to obtain the predicted object of the text sample; A fourth loss function is constructed based on the predicted object of the text sample and the object label of the object sample; The fourth loss function is inverted to obtain the second loss function of the neural network model.

12. An audio generation device based on artificial intelligence, characterized in that, The device includes: The sampling module is used to randomly sample multiple audio data of the target object to obtain a variable-length audio sequence, which is then used as reference audio data for the target object. An audio encoding module is used to perform audio encoding processing on reference audio data of the target object through a reference encoder to obtain a reference embedding vector of the reference audio data, which is used as an embedding vector to control the timbre of the target object. The audio encoding processing includes encoding the reference audio data through a multi-layer convolutional network, then inputting it into a one-way gated recurrent network, and using the representation of the one-way gated recurrent network at the last moment as the reference embedding vector. The reference encoder is trained by a first classifier, which is used to predict the timbre embedding vector obtained by the reference encoder from encoding the object sample. Based on the predicted object and the object label of the object sample, an error signal is determined, and the error signal is used to update the parameters of the reference encoder. An attention module is used to perform timbre-based attention processing on the reference embedding vector of the reference audio data to obtain the timbre embedding vector corresponding to the target object in the timbre embedding space; wherein, the attention processing is performed in the speaker embedding space, which consists of M different types of speaker tokens, and the embedding of the target object is represented as a linear combination of the M different types of speaker tokens; A text encoding module is used to perform text encoding processing on target text through a text encoder to obtain the content embedding vector of the target text. The text encoder is obtained by adversarial training through a second classifier. The second classifier is equipped with a gradient inversion layer to prevent the text encoder from obtaining object information of the target object during the adversarial training. The text encoder is shared by all objects. The synthesis module is used to perform synthesis processing based on the timbre embedding vector of the target object and the content embedding vector of the target text to obtain audio data that conforms to the timbre of the target object and corresponds to the target text. The training module obtains predicted alignment information, forced alignment information, and linear alignment information based on the timbre embedding vector of the object sample and the content embedding vector of the text sample. It constructs a first difference information based on the difference between the linear alignment information and the predicted alignment information, constructs a second difference information based on the difference between the forced alignment information and the predicted alignment information, and obtains a third loss function by weighted summation of the first difference information and the second difference information. The neural network model is trained based on the third loss function.

13. The apparatus according to claim 12, characterized in that, The attention module is also used for: The reference embedding vector of the reference audio data is mapped based on the embedding space of the object timbre to obtain the weights of multiple object timbres; Based on the weights of multiple object timbres, the embedding vectors of multiple object timbres are weighted and summed to obtain the timbre embedding vector of the target object.

14. The apparatus according to claim 12, characterized in that, The audio encoding module is also used for: The reference audio data of the target object is convolved to obtain the convolutional embedding vector of the reference audio data. The convolutional embedding vector of the reference audio data is forward encoded to obtain the reference embedding vector of the reference audio data.

15. The apparatus according to claim 14, characterized in that, The audio encoding is implemented through a reference encoder, which includes multiple cascaded convolutional layers; The audio encoding module is also used for: The reference audio data of the target object is convolved through the first convolutional layer in the plurality of cascaded convolutional layers. The convolution result of the first convolutional layer is output to subsequent cascaded convolutional layers, so that convolution processing and convolution result output continue in the subsequent cascaded convolutional layers until the output is reached in the last convolutional layer. The convolution result output from the last convolutional layer is used as the convolutional embedding vector of the reference audio data.

16. The apparatus according to claim 14, characterized in that, The audio encoding is implemented using a reference encoder, which includes a recurrent neural network; The audio encoding module is also used to perform vector update processing based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain the update information of the reference audio data. Vector reset processing is performed based on the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain the reset information of the reference audio data; Based on the update information and reset information of the reference audio data, the convolutional embedding vector of the reference audio data is subjected to context encoding to obtain the reference embedding vector of the reference audio data.

17. The apparatus according to claim 16, characterized in that, The audio encoding module is also used to concatenate the hidden vector of the recurrent neural network and the convolutional embedding vector of the reference audio data to obtain a concatenated vector. The spliced ​​vector is subjected to an update gate-based mapping process to obtain the update information of the reference audio data.

18. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the artificial intelligence-based audio generation method according to any one of claims 1 to 11.

19. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the AI-based audio generation method according to any one of claims 1 to 11 when executed by a processor.

20. A computer program product, characterized in that, The computer program product includes computer instructions for implementing the artificial intelligence-based audio generation method according to any one of claims 1 to 11 when executed by a processor.

Images