Target audio generation method and apparatus, electronic device, and medium

Abstract

The present disclosure provides a target audio generation method and device, electronic equipment, computer readable storage medium and computer program product, relating to the field of computer, and particularly to the field of artificial intelligence, deep learning and natural language processing technology. The implementation scheme is: obtaining reference audio, reference text and target text; obtaining reference audio acoustic mark sequence and reference audio semantic mark sequence based on feature discretization of the reference audio; performing feature discretization on the target text and the reference text respectively to obtain their corresponding text semantic mark sequences; obtaining target audio semantic mark sequence based on the text semantic mark sequences, the reference audio semantic mark sequence and the first generative language model; obtaining target audio acoustic mark sequence based on the reference audio semantic mark sequence, the target audio semantic mark sequence and the reference audio acoustic mark sequence and the second generative language model; and obtaining target audio based on the target audio acoustic mark sequence.

Description

Technical Field

[0001] This disclosure relates to the field of computer science, and more particularly to the fields of artificial intelligence, deep learning, and natural language processing, specifically to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating target audio. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] With the rapid development of artificial intelligence technology, speech synthesis, as one of the core technologies for realizing human-computer voice interaction systems, is an important direction in speech processing technology. Zero-shot text-to-speech is a speech synthesis technology that allows users to input target text and a reference audio clip, and then uses certain rules or model algorithms to convert the target text into an audio output that is similar to the speaker's voice in the reference audio clip. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating target audio.

[0005] According to one aspect of this disclosure, a method for generating target audio is provided, comprising: acquiring reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; obtaining a reference audio acoustic tag sequence and a reference audio semantic tag sequence by discretizing the features of the reference audio; discretizing the features of the target text and the reference text respectively to obtain their respective corresponding text semantic tag sequences; obtaining a target audio semantic tag sequence through a first generative language model based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence; obtaining a target audio acoustic tag sequence through a second generative language model based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence; and decoding the target audio based on the target audio acoustic tag sequence to obtain the target audio.

[0006] According to another aspect of this disclosure, a model training method is provided, comprising: acquiring a first marker sequence and a second marker sequence, and a label sequence; inputting the first marker sequence and the second marker sequence into a fifth generative language model to predict a first marker and its corresponding probability; inputting the first marker sequence, the second marker sequence, and the first marker in the label sequence into the fifth generative language model to predict a next marker and its corresponding probability; in response to determining that the next marker is not an end marker, inputting a marker sequence formed by the first marker sequence, the second marker sequence, and the markers corresponding to all currently predicted markers in the label sequence into the fifth generative language model to predict a new next marker and its corresponding probability; in response to determining that the next marker is an end marker, determining a third loss value based on the label sequence, all predicted markers, and their corresponding probabilities using a third loss function; and adjusting the parameters of the fifth generative language model based on the third loss value.

[0007] According to another aspect of this disclosure, a target audio generation agent is provided, comprising: an input module configured to: acquire reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; a processing module configured to perform the following operations: based on feature discretization of the reference audio, obtain a reference audio acoustic tag sequence and a reference audio semantic tag sequence; respectively discretize the target text and the reference text to obtain their respective corresponding text semantic tag sequences; based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence, obtain a target audio semantic tag sequence through a first generative language model; based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence, obtain a target audio acoustic tag sequence through a second generative language model; and based on the target audio acoustic tag sequence, decode to obtain the target audio; and an output module configured to output the target audio.

[0008] According to another aspect of this disclosure, a target audio generation apparatus is provided, comprising: a first acquisition unit configured to acquire reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; a first feature discretization unit configured to obtain a reference audio acoustic tag sequence and a reference audio semantic tag sequence based on feature discretization of the reference audio; a second feature discretization unit configured to perform feature discretization on the target text and the reference text respectively to obtain their respective corresponding text semantic tag sequences; a first processing unit configured to obtain a target audio semantic tag sequence based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence, through a first generative language model; a second processing unit configured to obtain a target audio acoustic tag sequence based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence, through a second generative language model; and a decoding unit configured to decode the target audio based on the target audio acoustic tag sequence to obtain the target audio.

[0009] According to another aspect of this disclosure, a model training apparatus is provided, comprising: a second acquisition unit configured to acquire a first label sequence, a second label sequence, and a tag sequence; a first prediction unit configured to input the first label sequence and the second label sequence into a fifth generative language model to predict a first label and its corresponding probability; a second prediction unit configured to input the first label sequence, the second label sequence, and the first label from the tag sequence into the fifth generative language model to predict a next label and its corresponding probability; a first determination unit configured to, in response to determining that the next label is not an end symbol, input a label sequence formed by the first label sequence, the second label sequence, and the labels corresponding to all currently predicted labels in the tag sequence into the fifth generative language model to predict a new next label and its corresponding probability; a second determination unit configured to, in response to determining that the next label is an end symbol, determine a third loss value based on the tag sequence, all predicted labels, and their corresponding probabilities using a third loss function; and an adjustment unit configured to adjust the parameters of the fifth generative language model based on the third loss value.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods described in this disclosure.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in this disclosure.

[0013] According to one or more embodiments of this disclosure, audio semantic tags are used as an intermediate bridge to connect text semantic tags and audio acoustic tags in order to better capture the semantic information of the context, thereby generating more natural and coherent speech output; and, by adopting a generative language model, its application in sound synthesis can produce a more natural and human-like speech effect.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0017] Figure 2 A flowchart of a target audio generation method according to an embodiment of the present disclosure is shown;

[0018] Figure 3 A schematic diagram is shown illustrating the feature discretization of audio to obtain a corresponding tag sequence according to an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram is shown illustrating the process of obtaining corresponding audio based on tag sequence decoding according to an embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram illustrating the acquisition of a target audio semantic tag sequence according to an embodiment of the present disclosure is shown;

[0021] Figure 6-7 A schematic diagram illustrating the acquisition of an N-layer target audio acoustic tag sequence according to an embodiment of the present disclosure is shown;

[0022] Figure 8 A flowchart of a model training method according to an embodiment of the present disclosure is shown;

[0023] Figure 9 A structural block diagram of a target audio generation agent according to an embodiment of the present disclosure is shown;

[0024] Figure 10 A structural block diagram of a target audio generation apparatus according to an embodiment of the present disclosure is shown;

[0025] Figure 11 A structural block diagram of a model training apparatus according to an embodiment of the present disclosure is shown; and

[0026] Figure 12 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0029] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0030] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0031] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0032] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of target audio generation methods and model training methods.

[0033] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105, and / or 106 under a Software as a Service (SaaS) model.

[0034] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0035] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to input reference audio, text, and acquire target audio, etc. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0036] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0037] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0038] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0039] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0040] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0041] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0042] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and text files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0043] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0044] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0045] Zero-shot text-to-speech is a speech synthesis technique that does not rely on specialized training data for the target speaker. Instead, it extracts acoustic features from a reference audio file using existing acoustic models and speech features, thereby achieving high-quality text-to-speech cloning.

[0046] Voice cloning schemes, exemplified by VALL-E, while performing well in speech synthesis tasks, still have some shortcomings in terms of semantic naturalness and coherence. For example, the synthesized audio may not correspond to the target text, some incoherent speech may appear in the synthesized audio, and the synthesized audio may sound unnatural or abrupt, not conforming to human language expression habits. This is because the data distribution of text semantic tags and audio acoustic tags differs significantly, making it challenging to directly utilize the model's ability to continue the sentence.

[0047] Therefore, an embodiment of the present disclosure provides a method for generating target audio. Figure 2 A flowchart of a target audio generation method according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, method 200 includes: acquiring reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated (step 210); obtaining a reference audio acoustic tag sequence and a reference audio semantic tag sequence based on feature discretization of the reference audio (step 220); discretizing the target text and the reference text respectively to obtain their respective text semantic tag sequences (step 230); obtaining a target audio semantic tag sequence through a first generative language model based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence (step 240); obtaining a target audio acoustic tag sequence through a second generative language model based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence (step 250); and decoding the target audio based on the target audio acoustic tag sequence (step 260).

[0048] According to embodiments of this disclosure, audio semantic tags are used as an intermediate bridge to connect text semantic tags and audio acoustic tags in order to better capture the semantic information of the context, thereby generating a more natural and coherent speech output; and, by adopting a generative language model, its application in voice synthesis can produce a more natural and human-like speech effect.

[0049] In this disclosure, a token can be understood as the smallest unit of information received and processed by a language model. For example, a token is typically used to represent a unit in text data. In different contexts, a token may represent a character, a word, or a sentence.

[0050] In this disclosure, the target text and the reference text can be discretized using any suitable method to obtain their respective text semantic tag sequences. Text semantic tags typically represent high-level semantic information of the text content.

[0051] According to some embodiments, discretizing the reference audio features to obtain a corresponding reference audio semantic tag sequence includes: dividing the reference audio into frames to obtain multiple frames of reference audio; for each frame of reference audio, determining the reference audio vector corresponding to that frame; for each frame of reference audio, determining, in a preset audio semantic vector table, the audio semantic vector closest to the reference audio vector corresponding to that frame and the index corresponding to the closest audio semantic vector, so as to use the index as its corresponding reference audio semantic tag; and determining the reference audio semantic tag sequence based on the indices corresponding to each of the multiple frames of reference audio. The preset audio semantic vector table includes multiple audio semantic vectors, each of the multiple audio semantic vectors corresponding to a corresponding pronunciation unit, and each audio semantic vector has a corresponding index.

[0052] In some embodiments, the reference audio vector corresponding to each frame of reference audio can be obtained based on the w2v-BERT model. That is, audio_vec = w2v-BERT(wav), where wav represents the reference audio (audio frame), and audio_vec represents the corresponding audio frame vector. Then, feature discretization is performed. That is, semantic_tokens = semantic_quantizer.encoder(audio_vec), where semantic_quantizer represents the audio semantic quantizer (i.e., a pre-defined audio semantic vector table), whose function is to transform the audio vector into a discretized representation of audio features. In other words, the reference audio semantic token sequence includes multiple reference audio semantic tokens.

[0053] In the example above, the encoding process of the audio semantic quantizer (i.e., the pre-defined audio semantic vector table) can be represented as: semantic_codebook:video_semantic_vec->video_semantic_token. That is, the ID (i.e., index) corresponding to the audio semantic vector that is closest to the corresponding audio vector in the audio semantic vector table (e.g., Euclidean distance) is used as its audio semantic token.

[0054] In some examples, each audio semantic vector in the audio semantic vector corresponds to a specific phonetic unit, and each audio semantic vector has a corresponding index. For example, the phonetic unit can refer to a phoneme, syllable, or other speech unit. The original audio waveform is converted into a high-dimensional continuous vector using a deep learning model (such as Wav2Vec 2.0), with each vector implicitly containing acoustic features (timbre, intonation, emotion, etc.) and semantic information.

[0055] In some examples, multiple cluster centers can be obtained by clustering the semantic vectors corresponding to multiple phonetic units, and the vector representation of the cluster centers can be used as the vector representation in the audio semantic vector table. For example, the number of vectors in the audio semantic vector table can be 1024. For example, during the prediction process, the index (token) corresponding to the nearest vector in the audio semantic vector table is selected as the discrete representation of the current audio frame.

[0056] In some examples, this audio semantic vector table can be trained.

[0057] According to some embodiments, the preset audio semantic vector table is trained through the following operations: obtaining the audio to be processed, multiple audio frames corresponding to the audio to be processed, and text tags corresponding to the audio to be processed; determining the audio vector corresponding to each of the multiple audio frames; for each audio frame, determining the audio semantic vector closest to the audio vector corresponding to that audio frame in the audio semantic vector table, so as to form a vector pair between the audio vector corresponding to that audio frame and the closest audio semantic vector; determining a first loss value based on the vector pairs corresponding to each of the multiple audio frames using a first loss function; predicting the target text corresponding to the audio to be processed based on the closest audio semantic vector corresponding to each of the multiple audio frames; determining a second loss value based on the target text and the text tags using a second loss function; and adjusting the corresponding audio semantic vectors in the audio semantic vector table based on the first loss value and the second loss value.

[0058] According to some embodiments, the first loss function includes a mean squared error loss function, and the second loss function includes a maximum likelihood loss function.

[0059] The first loss function (loss_quantizer) is used to calculate the difference between the input feature x (the audio vector corresponding to the audio frame) and its corresponding quantized vector quantize (i.e., the nearest audio semantic vector). For example, this first loss function can be the mean squared error loss function, as shown below:

[0060]

[0061] Where N represents the number of vectors in the audio semantic vector table.

[0062] In some examples, training can be based on Audio-to-Speech (ASR) technology. For instance, after classifying the audio semantic vectors corresponding to the nearest neighbors for each of multiple audio frames, a softmax layer is applied to obtain the predicted text corresponding to the audio. The second loss function (loss_ctc) is the loss function corresponding to the audio-to-text task (ASR). For example, the second loss function can be the maximum likelihood loss function, which aims to maximize the probability of generating the correct output sequence based on a given input sequence.

[0063] Therefore, the loss function `loss = loss_quantizer + loss_ctc` means that the corresponding audio semantic vectors in the audio semantic vector table are adjusted based on the first loss value and the second loss value. By training this audio semantic vector table, the accuracy of the labeled sequences obtained through this audio semantic vector table in representing audio features can be improved.

[0064] According to some embodiments, discretizing the reference audio features to obtain a corresponding reference audio acoustic tag sequence includes: dividing the reference audio into frames to obtain multiple frames of reference audio; for each frame of reference audio, determining the reference audio vector corresponding to that frame; for each frame of reference audio, determining, in a preset audio acoustic vector table, the audio acoustic vector closest to the reference audio vector corresponding to that frame and the index corresponding to the closest audio acoustic vector, so as to use the index as its corresponding reference audio acoustic tag; and determining the reference audio acoustic tag sequence based on the indices corresponding to each of the multiple frames of reference audio. The preset audio acoustic vector table includes multiple audio acoustic vectors, each of the multiple audio acoustic vectors corresponding to a corresponding audio feature, and each audio acoustic vector having a corresponding index.

[0065] In this embodiment, the preset audio acoustic vector table is similar to the audio semantic vector table described above, except that the audio acoustic vectors are directly related to the audio signal and typically contain specific acoustic features such as pitch, volume, and duration. For example, the audio acoustic vectors in the audio acoustic vector table correspond to corresponding audio features, which can be used to represent features such as basic speech features, background noise or ambient sound features, and instrument sound features. Further details will not be elaborated upon here.

[0066] According to some embodiments, decoding the target audio based on the target audio acoustic tag sequence includes: determining the audio acoustic vector corresponding to each target audio acoustic tag in the target audio acoustic tag sequence in the preset audio acoustic vector table based on the preset audio acoustic vector table; and decoding the target audio based on all the audio acoustic vectors corresponding to the target audio acoustic tag sequence.

[0067] Figure 3 A schematic diagram is shown illustrating the feature discretization of audio according to an embodiment of the present disclosure to obtain a corresponding tag sequence. For example... Figure 3 As shown, the audio is transformed into a set of vectors by the encoder, and this set of vectors is transformed into a set of labeled sequences by the audio semantic quantizer (i.e., RVQ) as a discretized representation of the audio features.

[0068] Accordingly, the decoding process of the audio semantic quantizer can be represented as: semantic_codebook:video_semantic_token->video_semantic_vec. That is, by looking up the audio semantic vector table, the audio semantic token is converted into the corresponding audio semantic vector.

[0069] Figure 4 A schematic diagram illustrating the process of obtaining corresponding audio based on tag sequence decoding according to an embodiment of the present disclosure is shown. Figure 4 As shown, a set of labeled sequences is transformed into a set of vectors by an audio semantic quantizer (RVQ), and this set of vectors is then converted into the corresponding audio by a decoder.

[0070] According to the above embodiment, in step 240, based on the text semantic tag sequence corresponding to the target text and the reference text respectively, and the reference audio semantic tag sequence, the target audio semantic tag sequence is obtained through the first generative language model.

[0071] Figure 5 A schematic diagram illustrating the acquisition of a target audio semantic tag sequence according to an embodiment of the present disclosure is shown. Figure 5As shown, the text semantic tag sequences corresponding to the reference text and target text, as well as the reference audio semantic tag sequence, are input into the first generative language model (i.e., Text-to-Semantic GPT) to progressively predict the target audio semantic tag sequence. Through the generative language model, subsequent target audio semantic tags are generated sequentially based on the given context, without needing to predict the length of the target audio. Generative language models are better at generating coherent audio, simulating the tone, speaking habits, and style of the reference audio; therefore, they can produce a more natural and human-like speaking effect in voice synthesis.

[0072] According to the above embodiment, in step 250, the target audio acoustic tag sequence is obtained through a second generative language model based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence.

[0073] According to some embodiments, the reference audio acoustic tag sequence includes N layers of reference audio acoustic tag sequences, which are obtained by discretizing the reference audio based on N preset audio acoustic vector tables, where N ≥ 2 and N is a positive integer. The target audio acoustic tag sequence includes N layers of target audio acoustic tag sequences.

[0074] In the above embodiments, each layer of reference audio acoustic tag sequence includes multiple reference audio acoustic tags, which correspond to multiple audio frames of the reference audio.

[0075] According to some embodiments, the N preset audio acoustic vector tables correspond to different types of audio features.

[0076] In some examples, each audio acoustic vector table (i.e., quantizer) captures different levels or types of information. For instance, the first audio acoustic vector table might focus on the basic speech features of the audio, the second might capture background noise or ambient sounds, and others might handle instrument sounds, rhythms, etc. This hierarchical approach allows the model to understand the audio content at a higher level while also capturing details at a lower level.

[0077] According to some embodiments, the second generative language model includes a third generative language model and a fourth generative language model. Therefore, obtaining the target audio acoustic tag sequence through the second generative language model includes: obtaining a first-layer target audio acoustic tag sequence through the third generative language model based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and a first-layer reference audio acoustic tag sequence in the reference audio acoustic tag sequence; predicting the next-layer target audio acoustic tag sequence through the fourth generative language model based on the N-layer reference audio acoustic tag sequence and the first-layer target audio acoustic tag sequence; in response to determining that the next-layer target audio acoustic tag sequence is not the Nth-layer target audio acoustic tag sequence, predicting a new next-layer target audio acoustic tag sequence through the fourth generative language model based on the N-layer reference audio acoustic tag sequence and the currently obtained multi-layer target audio acoustic tag sequence; and in response to determining that the next-layer target audio acoustic tag sequence is the Nth-layer target audio acoustic tag sequence, using the currently obtained multi-layer target audio acoustic tag sequence as the Nth-layer target audio acoustic tag sequence.

[0078] Figure 6-7 A schematic diagram illustrating the acquisition of an N-layer target audio acoustic tag sequence according to an embodiment of the present disclosure is shown. Figure 6 As shown, the first-layer reference audio acoustic tag sequence in the reference audio acoustic tag sequence is used as the coarse-grained reference audio acoustic tag sequence. Based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the coarse-grained reference audio acoustic tag sequence, a coarse-grained target audio acoustic tag sequence is obtained through a third generative language model (i.e., Semantic-to-Acoustic GPT). Then, as... Figure 7 As shown, a multi-layered input and multi-layered output module is designed in the fourth generative language model. For example, this fourth generative language model can be implemented based on the Transformer model, that is, by adding multi-layered input and multi-layered output units to the Transformer model. This involves combining N layers of reference audio acoustic tag sequences and reference... Figure 6The obtained coarse-grained target audio acoustic tag sequence (i.e., Layer = 1, corresponding to the first quantizer codebook1; other layers are empty) is input into the fourth generative language model (i.e., Corase-to-Fine Acoustic Transformer) to obtain a predicted new next-layer target audio acoustic tag sequence (i.e., the second-layer target audio acoustic tag sequence, Layer = 2, corresponding to the second quantizer codebook2). Then, the N-layer reference audio acoustic tag sequence and the currently obtained target audio acoustic tag sequence (i.e., Layer = 1, 2; other layers are empty) are input into the fourth generative language model to obtain a predicted new next-layer target audio acoustic tag sequence... until the N-layer target audio acoustic tag sequence is predicted layer by layer (in... Figure 7 (N=8).

[0079] Therefore, in some embodiments, decoding the target audio based on the target audio acoustic tag sequence may include: decoding the target audio based on the N-layer target audio acoustic tag sequence.

[0080] The embodiments disclosed herein can significantly improve the naturalness and coherence of speech synthesis, reduce incoherent speech and abrupt pauses during the speech synthesis process; at the same time, it integrates audio semantic information and acoustic information, and does not require display alignment training between text and speech, nor does it require audio length prediction.

[0081] According to embodiments of this disclosure, such as Figure 8 As shown, a model training method 800 is also provided, including: acquiring a first label sequence and a second label sequence, and a tag sequence (step 810); inputting the first label sequence and the second label sequence into a fifth generative language model to predict a first label and its corresponding probability (step 820); inputting the first label sequence, the second label sequence, and the first label from the tag sequence into the fifth generative language model to predict the next label and its corresponding probability (step 830); in response to determining that the next label is not an end symbol, inputting the label sequence formed by the first label sequence, the second label sequence, and the labels corresponding to all currently predicted labels in the tag sequence into the fifth generative language model to predict a new next label and its corresponding probability (step 840); in response to determining that the next label is an end symbol, determining a third loss value based on the tag sequence, all predicted labels, and their corresponding probabilities using a third loss function (step 850); and adjusting the parameters of the fifth generative language model based on the third loss value (step 860).

[0082] According to some embodiments, the first tag sequence includes: a text semantic tag sequence obtained by discretizing the target text and the reference text; the second tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the reference audio; the tag sequence includes: a target audio semantic tag obtained by discretizing the target audio, wherein the reference text is the text corresponding to the reference audio, the target text is the text corresponding to the target audio, and the target audio and the reference audio have the same audio features.

[0083] Specifically, the text semantic tag sequence corresponding to the target text and the reference text, and the reference audio semantic tag sequence are input into the fifth generative language model to predict the first target audio semantic tag and its corresponding probability; the text semantic tag sequence corresponding to the target text and the reference text, the reference audio semantic tag sequence, and the first target audio semantic tag in the tag sequence are input into the fifth generative language model to predict the next target audio semantic tag and its corresponding probability; in response to determining that the next target audio semantic tag is not an end-of-line character, the text semantic tag sequence corresponding to the target text and the reference text, the reference audio semantic tag sequence, and the target audio semantic tag in the tag sequence corresponding to all currently predicted tags are input into the fifth generative language model to predict the new next target audio semantic tag and its corresponding probability; in response to determining that the next target audio semantic tag is an end-of-line character, a third loss value is determined based on the tag sequence, all predicted tags, and their corresponding probabilities through a third loss function; and the parameters of the fifth generative language model are adjusted based on the third loss value.

[0084] In some embodiments, the third loss function may include a cross-entropy loss function. For example, the cross-entropy loss function L for each time step t. t It can be represented as:

[0085]

[0086] Where V represents the size of the corresponding audio semantic vector table; p t,i y represents the probability that the target audio semantic tag predicted by the model at time step t is the i-th tag (i.e., index) in the audio semantic vector table; t,i y represents the one-hot encoding of the true label in the label sequence corresponding to time step t (if the true label is the i-th predicted label, then y t,i =1, otherwise 0). The loss function corresponding to the entire labeled sequence can be obtained by summing the losses over all time steps, that is:

[0087]

[0088] Where T represents the number of tags in the tag sequence.

[0089] According to some embodiments, the first tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the features of the reference audio, and a target audio semantic tag obtained by discretizing the features of the target audio; the second tag sequence includes: a reference audio acoustic tag sequence obtained by discretizing the features of the reference audio; the tag sequence includes: a target audio acoustic tag obtained by discretizing the features of the target audio, wherein the target audio and the reference audio have the same audio features.

[0090] Specifically, the reference audio semantic tag sequence, the target audio semantic tag, and the reference audio acoustic tag sequence are input into the fifth generative language model to predict the first target audio acoustic tag and its corresponding probability; the reference audio semantic tag sequence, the target audio semantic tag, the reference audio acoustic tag sequence, and the first target audio acoustic tag in the tag sequence are input into the fifth generative language model to predict the next target audio acoustic tag and its corresponding probability; in response to determining that the next target audio acoustic tag is not an end symbol, the reference audio semantic tag sequence, the target audio semantic tag, the reference audio acoustic tag sequence, and the target audio acoustic tag in the tag sequence corresponding to all currently predicted tags are input into the fifth generative language model to predict the new next target audio acoustic tag and its corresponding probability; in response to determining that the next target audio acoustic tag is an end symbol, a third loss value is determined based on the tag sequence, all predicted tags, and their corresponding probabilities using a third loss function; and the parameters of the fifth generative language model are adjusted based on the third loss value.

[0091] According to some embodiments, the trained language model is used to implement the method described in any of the embodiments above.

[0092] According to embodiments of this disclosure, such as Figure 9As shown, a target audio generation agent 900 is also provided, including: an input module 910 configured to: acquire reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; a processing module 920 configured to perform the following operations: based on feature discretization of the reference audio, obtain a reference audio acoustic tag sequence and a reference audio semantic tag sequence; respectively discretize the target text and the reference text to obtain their respective corresponding text semantic tag sequences; based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence, obtain a target audio semantic tag sequence through a first generative language model; based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence, obtain a target audio acoustic tag sequence through a second generative language model; and based on the target audio acoustic tag sequence, decode to obtain the target audio; and an output module 930 configured to output the target audio.

[0093] According to some embodiments, the processing module is also configured to perform the method described in any of the embodiments described above.

[0094] According to embodiments of this disclosure, such as Figure 10 As shown, a target audio generation device 1000 is also provided, comprising: a first acquisition unit 1010 configured to acquire reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; a first feature discretization unit 1020 configured to obtain a reference audio acoustic tag sequence and a reference audio semantic tag sequence based on feature discretization of the reference audio; and a second feature discretization unit 1030 configured to perform feature discretization on the target text and the reference text respectively to obtain their respective corresponding texts. The system comprises: a semantic tagging sequence; a first processing unit 1040 configured to obtain a target audio semantic tagging sequence based on the text semantic tagging sequences corresponding to the target text and the reference text, and the reference audio semantic tagging sequence, using a first generative language model; a second processing unit 1050 configured to obtain a target audio acoustic tagging sequence based on the reference audio semantic tagging sequence, the target audio semantic tagging sequence, and the reference audio acoustic tagging sequence, using a second generative language model; and a decoding unit 1060 configured to decode the target audio based on the target audio acoustic tagging sequence.

[0095] Here, the operation of each of the above-mentioned units 1010 to 1060 of the target audio generation device 1000 is similar to the operation of steps 210 to 260 described above, and will not be repeated here.

[0096] According to embodiments of this disclosure, such as Figure 11 As shown, a model training apparatus 1100 is also provided, comprising: a second acquisition unit 1110 configured to acquire a first label sequence, a second label sequence, and a tag sequence; a first prediction unit 1120 configured to input the first label sequence and the second label sequence into a fifth generative language model to predict a first label and its corresponding probability; a second prediction unit 1130 configured to input the first label sequence, the second label sequence, and the first label from the tag sequence into the fifth generative language model to predict the next label and its corresponding probability; a first determination unit 1140 configured to, in response to determining that the next label is not an end symbol, input a label sequence formed by the first label sequence, the second label sequence, and the labels corresponding to all currently predicted labels in the tag sequence into the fifth generative language model to predict a new next label and its corresponding probability; a second determination unit 1150 configured to, in response to determining that the next label is an end symbol, determine a third loss value based on the tag sequence, all predicted labels, and their corresponding probabilities using a third loss function; and an adjustment unit 1160 configured to adjust the parameters of the fifth generative language model based on the third loss value.

[0097] Here, the operation of each of the above units 1110 to 1160 of the model training device 1100 is similar to the operation of steps 810 to 860 described above, and will not be repeated here.

[0098] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0099] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0100] refer to Figure 12The present invention describes a structural block diagram of an electronic device 1200 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0101] like Figure 12 As shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of the electronic device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0102] Multiple components in electronic device 1200 are connected to I / O interface 1205, including: input unit 1206, output unit 1207, storage unit 1208, and communication unit 1209. Input unit 1206 can be any type of device capable of inputting information to electronic device 1200. Input unit 1206 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 1207 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1208 may include, but is not limited to, a hard disk and an optical disk. The communication unit 1209 allows the electronic device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0103] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as method 200 or 800. For example, in some embodiments, method 200 or 800 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of method 200 or 800 described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute method 200 or 800 by any other suitable means (e.g., by means of firmware).

[0104] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0109] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0110] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0111] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for generating target audio, comprising: Obtain reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; Based on the feature discretization of the reference audio, a reference audio acoustic tag sequence and a reference audio semantic tag sequence are obtained. The reference audio acoustic tag sequence includes N layers of reference audio acoustic tag sequences. The N layers of reference audio acoustic tag sequences are obtained by discretizing the reference audio based on N preset audio acoustic vector tables, where N≥2 and N is a positive integer. The target text and the reference text are respectively discretized to obtain their respective text semantic tag sequences; Based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence, the target audio semantic tag sequence is obtained through a first generative language model; Based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence, a target audio acoustic tag sequence is obtained through a second generative language model. The second generative language model includes a third generative language model and a fourth generative language model. The target audio acoustic tag sequence comprises N layers of target audio acoustic tag sequences. Obtaining the target audio acoustic tag sequence through the second generative language model includes: Based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the first layer of the reference audio acoustic tag sequence, the first layer of the target audio acoustic tag sequence is obtained through a third generative language model; Based on the N-layer reference audio acoustic tag sequence and the first-layer target audio acoustic tag sequence, the next-layer target audio acoustic tag sequence is predicted by the fourth generative language model. In response to determining that the next-layer target audio acoustic tag sequence is not the Nth-layer target audio acoustic tag sequence, a new next-layer target audio acoustic tag sequence is predicted by the fourth generative language model based on the Nth-layer reference audio acoustic tag sequence and the currently obtained multi-layer target audio acoustic tag sequences; and In response to determining that the next-layer target audio acoustic tag sequence is the Nth-layer target audio acoustic tag sequence, the currently obtained multi-layer target audio acoustic tag sequences are used as the Nth-layer target audio acoustic tag sequence; and Based on the target audio acoustic tag sequence, the target audio is decoded to obtain the target audio, wherein decoding the target audio based on the target audio acoustic tag sequence includes: decoding the target audio based on the N-layer target audio acoustic tag sequence.

2. The method as described in claim 1, wherein, The N preset audio acoustic vector tables correspond to different types of audio features.

3. The method as described in claim 1, wherein, Discretizing the reference audio features to obtain the corresponding reference audio semantic tag sequence includes: The reference audio is divided into frames to obtain multiple frames of reference audio. For each frame of reference audio in the multi-frame reference audio, determine the reference audio vector corresponding to that frame of reference audio; For each frame of reference audio in the multi-frame reference audio, in the preset audio semantic vector table, the audio semantic vector that is closest to the reference audio vector corresponding to that frame of reference audio and the index corresponding to the closest audio semantic vector are determined, so that the index is used as its corresponding reference audio semantic tag. Based on the indices corresponding to the multiple frames of reference audio, the semantic tag sequence of the reference audio is determined. The preset audio semantic vector table includes multiple audio semantic vectors, each of which corresponds to a specific pronunciation unit, and each audio semantic vector has a corresponding index.

4. The method according to any one of claims 1-2, wherein, The reference audio is discretized to obtain the corresponding reference audio acoustic tag sequence, including: The reference audio is divided into frames to obtain multiple frames of reference audio; For each frame of reference audio in the multi-frame reference audio, determine the reference audio vector corresponding to that frame of reference audio; For each frame of reference audio in the multi-frame reference audio, in a preset audio acoustic vector table, the audio acoustic vector that is closest to the reference audio vector corresponding to that frame of reference audio and the index corresponding to the closest audio acoustic vector are determined, so that the index is used as its corresponding reference audio acoustic tag. Based on the indices corresponding to the multiple frames of reference audio, the acoustic tag sequence of the reference audio is determined. The preset audio acoustic vector table includes multiple audio acoustic vectors, each of which corresponds to a specific audio feature and has a corresponding index.

5. The method of claim 3, wherein, The preset audio semantic vector table is trained through the following operations: Obtain the audio to be processed, multiple audio frames corresponding to the audio to be processed, and text tags corresponding to the audio to be processed; Determine the audio vector corresponding to each of the plurality of audio frames; For each audio frame, the audio semantic vector that is closest to the audio vector corresponding to that audio frame is determined in the audio semantic vector table, so that the audio vector corresponding to that audio frame and the closest audio semantic vector are used as a vector pair; Based on the vector pairs corresponding to each of the plurality of audio frames, a first loss value is determined by a first loss function; Based on the nearest audio semantic vector corresponding to each of the multiple audio frames, the target text corresponding to the audio to be processed is predicted. Based on the target text and the text label, a second loss value is determined using a second loss function; as well as The corresponding audio semantic vector in the audio semantic vector table is adjusted based on the first loss value and the second loss value.

6. The method of claim 5, wherein, The first loss function includes the mean squared error loss function, and the second loss function includes the maximum likelihood loss function.

7. The method of claim 4, wherein, Based on the target audio acoustic tag sequence, the decoded target audio includes: Based on the preset audio acoustic vector table, determine the audio acoustic vector corresponding to each target audio acoustic tag in the target audio acoustic tag sequence in the preset audio acoustic vector table; and The target audio is obtained by decoding based on all the audio acoustic vectors corresponding to the target audio acoustic tag sequence.

8. A model training method, comprising: Obtain the first marker sequence, the second marker sequence, and the label sequence; The first and second tag sequences are input into the fifth generative language model to predict the first tag and its corresponding probability. The first tag sequence, the second tag sequence, and the first tag in the tag sequence are input into the fifth generative language model to predict the next tag and its corresponding probability. In response to determining that the next tag is not an end symbol, the tag sequence formed by the first tag sequence, the second tag sequence, and the tags corresponding to all currently predicted tags in the tag sequence is input into the fifth generative language model to predict the new next tag and its corresponding probability. In response to determining the next label as the end symbol, a third loss value is determined based on the label sequence, all predicted labels and their corresponding probabilities, using a third loss function. as well as The parameters of the fifth generative language model are adjusted based on the third loss value, wherein, The first tag sequence includes: a text semantic tag sequence obtained by discretizing the target text and the reference text; the second tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the reference audio; the tag sequence includes: a target audio semantic tag obtained by discretizing the target audio, wherein the reference text is the text corresponding to the reference audio, the target text is the text corresponding to the target audio, and the target audio and the reference audio have the same audio features; or... The first tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the features of the reference audio, and a target audio semantic tag obtained by discretizing the features of the target audio; the second tag sequence includes: a reference audio acoustic tag sequence obtained by discretizing the features of the reference audio; the tag sequence includes: a target audio acoustic tag obtained by discretizing the features of the target audio, wherein the target audio and the reference audio have the same audio features.

9. The method of claim 8, wherein, The trained language model is used to implement the method of any one of claims 1-7.

10. A target audio generation agent, comprising: The input module is configured to: acquire reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; The processing module is configured to perform the following operations: Based on the feature discretization of the reference audio, a reference audio acoustic tag sequence and a reference audio semantic tag sequence are obtained. The reference audio acoustic tag sequence includes N layers of reference audio acoustic tag sequences. The N layers of reference audio acoustic tag sequences are obtained by discretizing the reference audio based on N preset audio acoustic vector tables, where N≥2 and N is a positive integer. The target text and the reference text are respectively discretized to obtain their respective text semantic tag sequences; Based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence, the target audio semantic tag sequence is obtained through a first generative language model; Based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence, a target audio acoustic tag sequence is obtained through a second generative language model. The second generative language model includes a third generative language model and a fourth generative language model. The target audio acoustic tag sequence comprises N layers of target audio acoustic tag sequences. Obtaining the target audio acoustic tag sequence through the second generative language model includes: Based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the first layer of the reference audio acoustic tag sequence, the first layer of the target audio acoustic tag sequence is obtained through a third generative language model; Based on the N-layer reference audio acoustic tag sequence and the first-layer target audio acoustic tag sequence, the next-layer target audio acoustic tag sequence is predicted by the fourth generative language model. In response to determining that the next-layer target audio acoustic tag sequence is not the Nth-layer target audio acoustic tag sequence, a new next-layer target audio acoustic tag sequence is predicted by the fourth generative language model based on the Nth-layer reference audio acoustic tag sequence and the currently obtained multi-layer target audio acoustic tag sequences; and In response to determining that the next-layer target audio acoustic tag sequence is the Nth-layer target audio acoustic tag sequence, the currently obtained multi-layer target audio acoustic tag sequences are used as the Nth-layer target audio acoustic tag sequence; and Based on the target audio acoustic tag sequence, the target audio is decoded to obtain the target audio, wherein decoding the target audio based on the target audio acoustic tag sequence includes: decoding the target audio based on the N-layer target audio acoustic tag sequence; The output module is configured to output the target audio.

11. The intelligent agent as claimed in claim 10, wherein, The processing module is also configured to perform the method of any one of claims 2-7.

12. A target audio generation apparatus, comprising: The first acquisition unit is configured to acquire reference audio, reference text, and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; The first feature discretization unit is configured to obtain a reference audio acoustic tag sequence and a reference audio semantic tag sequence by discretizing the reference audio features. The reference audio acoustic tag sequence includes N layers of reference audio acoustic tag sequences, which are obtained by discretizing the reference audio features based on N preset audio acoustic vector tables, where N ≥ 2 and N is a positive integer. The second feature discretization unit is configured to discretize the target text and the reference text respectively to obtain their respective text semantic tag sequences; The first processing unit is configured to obtain the target audio semantic tag sequence based on the text semantic tag sequence corresponding to the target text and the reference text, and the reference audio semantic tag sequence, through a first generative language model; The second processing unit is configured to obtain a target audio acoustic tag sequence based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the reference audio acoustic tag sequence using a second generative language model. The second generative language model includes a third generative language model and a fourth generative language model. The target audio acoustic tag sequence includes N layers of target audio acoustic tag sequences. Obtaining the target audio acoustic tag sequence using the second generative language model includes: Based on the reference audio semantic tag sequence, the target audio semantic tag sequence, and the first layer of the reference audio acoustic tag sequence, the first layer of the target audio acoustic tag sequence is obtained through a third generative language model; Based on the N-layer reference audio acoustic tag sequence and the first-layer target audio acoustic tag sequence, the next-layer target audio acoustic tag sequence is predicted by the fourth generative language model. In response to determining that the next-layer target audio acoustic tag sequence is not the Nth-layer target audio acoustic tag sequence, a new next-layer target audio acoustic tag sequence is predicted by the fourth generative language model based on the Nth-layer reference audio acoustic tag sequence and the currently obtained multi-layer target audio acoustic tag sequences; and In response to determining that the next-layer target audio acoustic tag sequence is the Nth-layer target audio acoustic tag sequence, the currently obtained multi-layer target audio acoustic tag sequences are used as the Nth-layer target audio acoustic tag sequence; and The decoding unit is configured to decode the target audio based on the target audio acoustic tag sequence, wherein decoding the target audio based on the target audio acoustic tag sequence includes: decoding the target audio based on the N-layer target audio acoustic tag sequence.

13. A model training device, comprising: The second acquisition unit is configured to acquire the first marker sequence, the second marker sequence, and the tag sequence; The first prediction unit is configured to input the first label sequence and the second label sequence into the fifth generative language model to predict the first label and its corresponding probability. The second prediction unit is configured to input the first tag sequence, the second tag sequence, and the first tag in the label sequence into the fifth generative language model to predict the next tag and its corresponding probability. The first determining unit is configured to, in response to determining that the next tag is not an end symbol, input the first tag sequence, the second tag sequence, and the tag sequence formed by the currently predicted tags corresponding to all tags in the tag sequence into the fifth generative language model, so as to predict the new next tag and its corresponding probability; The second determining unit is configured to, in response to determining the next tag as the end symbol, determine a third loss value based on the tag sequence, all predicted tags and their corresponding probabilities, and through a third loss function. as well as The adjustment unit is configured to adjust the parameters of the fifth generative language model based on the third loss value, wherein, The first tag sequence includes: a text semantic tag sequence obtained by discretizing the target text and the reference text; the second tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the reference audio; the tag sequence includes: a target audio semantic tag obtained by discretizing the target audio, wherein the reference text is the text corresponding to the reference audio, the target text is the text corresponding to the target audio, and the target audio and the reference audio have the same audio features; or... The first tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the features of the reference audio, and a target audio semantic tag obtained by discretizing the features of the target audio; the second tag sequence includes: a reference audio acoustic tag sequence obtained by discretizing the features of the reference audio; the tag sequence includes: a target audio acoustic tag obtained by discretizing the features of the target audio, wherein the target audio and the reference audio have the same audio features.

14. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-9.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

16. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-9.

Images