Target audio generation method and device, electronic equipment and medium

Abstract

The invention provides a target audio generation method and device, electronic equipment, a computer readable storage medium and a computer program product, and relates to the field of computers, in particular to the technical field of artificial intelligence, deep learning and natural language processing. According to the implementation scheme, reference audio, a reference text and a target text are obtained; the method comprises the following steps: performing feature discretization on a reference audio to obtain a reference audio acoustic marking sequence and a reference audio semantic marking sequence; performing feature discretization on the target text and the reference text to obtain respective corresponding text semantic tag sequences; based on the text semantic tag sequence and the reference audio semantic tag sequence, obtaining a target audio semantic tag sequence through a first generative language model; based on the reference audio semantic marking sequence, the target audio semantic marking sequence and the reference audio acoustic marking sequence, obtaining a target audio acoustic marking sequence through a second generative language model; and decoding based on the target audio acoustic mark sequence to obtain a target audio.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular, to the fields of artificial intelligence, deep learning, and natural language processing. Specifically, it relates to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating a target audio. Background Art

[0002] Artificial intelligence is a discipline that studies how to make a computer simulate certain thinking processes and intelligent behaviors of humans (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level technologies and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0003] With the rapid development of artificial intelligence technology, speech synthesis, as one of the core technologies for realizing a human-computer speech interaction system, is an important direction in speech processing technology. Zero-shot text-to-speech is a speech synthesis technology that allows users to input a target text and a reference audio, and convert the target text into an audio output similar to the voice of the speaker in the reference audio through certain rules or model algorithms. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating a target audio.

[0005] According to one aspect of the present disclosure, there is provided a method for generating a target audio, including: obtaining a reference audio, a reference text, and a target text, where the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; discretizing the features of the reference audio to obtain a reference audio acoustic marker sequence and a reference audio semantic marker sequence; discretizing the features of the target text and the reference text respectively to obtain their respective corresponding text semantic marker sequences; obtaining a target audio semantic marker sequence through a first generative language model based on the text semantic marker sequences corresponding to the target text and the reference text respectively, and the reference audio semantic marker sequence; obtaining a target audio acoustic marker sequence through a second generative language model based on the reference audio semantic marker sequence, the target audio semantic marker sequence, and the reference audio acoustic marker sequence; and decoding to obtain the target audio based on the target audio acoustic marker sequence.

[0006] According to another aspect of the present disclosure, there is provided a model training method, including: obtaining a first token sequence, a second token sequence, and a label sequence; inputting the first token sequence and the second token sequence into a fifth generative language model to predict a first token and its corresponding probability; inputting the first token sequence, the second token sequence, and the first token in the label sequence into the fifth generative language model to predict a next token and its corresponding probability; in response to determining that the next token is not an end token, inputting the first token sequence, the second token sequence, and a token sequence formed by the tokens corresponding to all currently predicted tokens in the label sequence into the fifth generative language model to predict a new next token and its corresponding probability; in response to determining that the next token is an end token, determining a third loss value through a third loss function based on the label sequence, all predicted tokens, and their corresponding probabilities; and adjusting parameters of the fifth generative language model based on the third loss value.

[0007] According to another aspect of the present disclosure, there is provided a target audio generation agent, including: an input module configured to obtain a reference audio, a reference text, and a target text, where the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; a processing module configured to perform the following operations: based on feature discretization of the reference audio, obtaining a reference audio acoustic token sequence and a reference audio semantic token sequence; performing feature discretization on the target text and the reference text respectively to obtain their respective corresponding text semantic token sequences; obtaining a target audio semantic token sequence through a first generative language model based on the text semantic token sequences corresponding to the target text and the reference text respectively, and the reference audio semantic token sequence; obtaining a target audio acoustic token sequence through a second generative language model based on the reference audio semantic token sequence, the target audio semantic token sequence, and the reference audio acoustic token sequence; and decoding to obtain a target audio based on the target audio acoustic token sequence; and an output module configured to output the target audio.

[0008] According to another aspect of the present disclosure, there is provided a target audio generation device, including: a first acquisition unit configured to acquire a reference audio, a reference text, and a target text, where the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; a first feature discretization unit configured to obtain a reference audio acoustic marker sequence and a reference audio semantic marker sequence based on feature discretization of the reference audio; a second feature discretization unit configured to perform feature discretization on the target text and the reference text respectively to obtain respective corresponding text semantic marker sequences; a first processing unit configured to obtain a target audio semantic marker sequence through a first generative language model based on the text semantic marker sequences corresponding to the target text and the reference text respectively, and the reference audio semantic marker sequence; a second processing unit configured to obtain a target audio acoustic marker sequence through a second generative language model based on the reference audio semantic marker sequence, the target audio semantic marker sequence, and the reference audio acoustic marker sequence; and a decoding unit configured to decode to obtain a target audio based on the target audio acoustic marker sequence.

[0009] According to another aspect of the present disclosure, there is provided a model training device, including: a second acquisition unit configured to acquire a first marker sequence, a second marker sequence, and a label sequence; a first prediction unit configured to input the first marker sequence and the second marker sequence into a fifth generative language model to predict a first marker and its corresponding probability; a second prediction unit configured to input the first marker sequence, the second marker sequence, and the first marker in the label sequence into the fifth generative language model to predict a next marker and its corresponding probability; a first determination unit configured to, in response to determining that the next marker is not an end symbol, input the first marker sequence, the second marker sequence, and a marker sequence formed by the markers corresponding to all currently predicted markers in the label sequence into the fifth generative language model to predict a new next marker and its corresponding probability; a second determination unit configured to, in response to determining that the next marker is an end symbol, determine a third loss value through a third loss function based on the label sequence, all predicted markers, and their corresponding probabilities; and an adjustment unit configured to adjust parameters of the fifth generative language model based on the third loss value.

[0010] According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method described in the present disclosure.

[0011] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method described in the present disclosure.

[0012] According to another aspect of the present disclosure, there is provided a computer program product including a computer program which, when executed by a processor, implements the method described in the present disclosure.

[0013] According to one or more embodiments of the present disclosure, audio semantic tags are used as an intermediate bridge to connect text semantic tags and audio acoustic tags to better capture the semantic information of the context, thereby generating a more natural and coherent speech output; and, a generative language model is adopted, and its application in voice synthesis can produce a more natural and human-like speaking effect.

[0014] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings exemplarily illustrate embodiments and form a part of the specification, and are used together with the written description of the specification to explain the exemplary embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. In all the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein can be implemented according to an embodiment of the present disclosure;

[0017] Figure 2 A flowchart of a target audio generation method according to an embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of discretizing the features of an audio to obtain a corresponding tag sequence according to an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of decoding a corresponding audio based on a tag sequence according to an embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of obtaining a target audio semantic tag sequence according to an embodiment of the present disclosure;

[0021] Figures 6-7 A schematic diagram of obtaining an N-layer target audio acoustic tag sequence according to an embodiment of the present disclosure;

[0022] Figure 8 The flowchart of a model training method according to an embodiment of the present disclosure is shown;

[0023] Figure 9 The structural block diagram of a target audio generation agent according to an embodiment of the present disclosure is shown;

[0024] Figure 10 The structural block diagram of a target audio generation device according to an embodiment of the present disclosure is shown;

[0025] Figure 11 The structural block diagram of a model training device according to an embodiment of the present disclosure is shown; and

[0026] Figure 12 The structural block diagram of an exemplary electronic device capable of implementing the embodiments of the present disclosure is shown. Detailed Embodiments

[0027] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following.

[0028] In the present disclosure, unless otherwise specified, the terms "first", "second", etc. are used to describe various elements and are not intended to limit the positional relationship, timing relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, and in certain cases, based on the context description, they may also refer to different instances.

[0029] In the descriptions of various examples in the present disclosure, the terms used are only for the purpose of describing specific examples and are not intended to be restrictive. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in the present disclosure covers any one of the listed items and all possible combinations.

[0030] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0031] Figure 1 The schematic diagram of an exemplary system 100 in which various methods and devices described herein can be implemented according to an embodiment of the present disclosure is shown. Refer to Figure 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 that couple the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 may be configured to execute one or more applications.

[0032] In embodiments of the present disclosure, the server 120 may run one or more services or software applications that enable the execution of the target audio generation method, the model training method.

[0033] In certain embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. In certain embodiments, these services may be provided as web-based services or cloud services, for example, provided to users of the client devices 101, 102, 103, 104, 105 and / or 106 under a software as a service (SaaS) model.

[0034] In Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that may be executed by one or more processors. Users operating the client devices 101, 102, 103, 104, 105 and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible, which may be different from the system 100. Therefore, Figure 1 is an example of a system for implementing the various methods described herein and is not intended to be limiting.

[0035] Users may use the client devices 101, 102, 103, 104, 105 and / or 106 to input reference audio, text, and obtain target audio, etc. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via this interface. Although Figure 1 only six client devices are depicted, those skilled in the art will be able to understand that the present disclosure may support any number of client devices.

[0036] Client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computing devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices, etc. These computing devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT WindowsMobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0037] Network 110 can be any type of network known to those skilled in the art, which can support data communication using any one of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, token ring, wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (such as Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0038] Server 120 can include one or more general-purpose computers, dedicated server computers (such as PC (personal computer) servers, UNIX servers, midrange servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 can include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the server). In various embodiments, server 120 can run one or more services or software applications that provide the functions described below.

[0039] The computing unit in server 120 can run one or more operating systems including any of the above operating systems and any commercially available server operating system. Server 120 can also run any one of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0040] In some embodiments, server 120 can include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 can also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0041] In some embodiments, server 120 can be a server of a distributed system, or a server incorporating a blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system to address the defects of high management difficulty and weak business scalability existing in traditional physical hosts and virtual private server (VPS) services.

[0042] System 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of databases 130 can be used to store information such as audio files and text files. Databases 130 can reside in various locations. For example, the databases used by server 120 can be local to server 120, or can be remote from server 120 and can communicate with server 120 via a network-based or dedicated connection. Databases 130 can be of different types. In certain embodiments, the databases used by server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0043] In certain embodiments, one or more of databases 130 can also be used by applications to store application data. The databases used by applications can be different types of databases, such as key-value repositories, object repositories, or conventional repositories supported by a file system.

[0044] Figure 1System 100 can be configured and operated in various ways to enable the application of the various methods and apparatuses described in this disclosure.

[0045] Zero-shot text-to-speech is a voice synthesis technology that does not rely on dedicated training data of the target speaker, but extracts the acoustic features from the reference audio through existing acoustic models and voice features, thereby achieving high-quality voice cloning.

[0046] Voice cloning solutions represented by VALL-E, although performing well in voice synthesis tasks, still have some drawbacks in terms of semantic naturalness and coherence. For example: the synthesized audio does not correspond to the target text, there are some nonsense phenomena in the synthesized audio, the synthesized audio does not sound in line with the expression habits of human language, and there are rigid or unnatural pauses. This is because the data distributions of text semantic markers and audio acoustic markers are quite different, and there are certain challenges in directly using the model's ability to continue writing.

[0047] Therefore, according to an embodiment of the present disclosure, a method for generating a target audio is provided. Figure 2 The flowchart of the method for generating a target audio according to an embodiment of the present disclosure is shown, as Figure 2 shown, method 200 includes: obtaining a reference audio, a reference text, and a target text, where the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated (step 210); based on discretizing the features of the reference audio, obtaining a reference audio acoustic marker sequence and a reference audio semantic marker sequence (step 220); discretizing the features of the target text and the reference text respectively to obtain their respective corresponding text semantic marker sequences (step 230); based on the text semantic marker sequences corresponding to the target text and the reference text respectively, and the reference audio semantic marker sequence, obtaining a target audio semantic marker sequence through a first generative language model (step 240); based on the reference audio semantic marker sequence, the target audio semantic marker sequence, and the reference audio acoustic marker sequence, obtaining a target audio acoustic marker sequence through a second generative language model (step 250); and decoding the target audio acoustic marker sequence to obtain the target audio (step 260).

[0048] According to an embodiment of the present disclosure, audio semantic markers are used as an intermediate bridge to connect text semantic markers and audio acoustic markers to better capture the semantic information of the context, thereby generating a more natural and coherent voice output; and, the application of the generative language model in voice synthesis can produce a more natural and human-like speaking effect.

[0049] In the present disclosure, a token can be understood as the smallest information unit received and processed by a language model. For example, tokens are typically used to represent a unit in text data. In different contexts, a token may represent a character, a word, or a sentence.

[0050] In the present disclosure, the target text and the reference text can be respectively subjected to feature discretization by any suitable method to obtain their respective corresponding text semantic token sequences. The text semantic tokens generally represent high-level semantic information of the text content.

[0051] According to some embodiments, obtaining the corresponding reference audio semantic token sequence by performing feature discretization on the reference audio includes: dividing the reference audio into frames to obtain multiple frames of reference audio; for each frame of reference audio in the multiple frames of reference audio, determining the reference audio vector corresponding to this frame of reference audio; for each frame of reference audio in the multiple frames of reference audio, in a preset audio semantic vector table, determining the audio semantic vector with the closest distance to the reference audio vector corresponding to this frame of reference audio and the index corresponding to the audio semantic vector with the closest distance, so as to use the index as its corresponding reference audio semantic token; based on the indices respectively corresponding to the multiple frames of reference audio, determining the reference audio semantic token sequence. The preset audio semantic vector table includes multiple audio semantic vectors, each of the multiple audio semantic vectors respectively corresponds to a corresponding pronunciation unit, and each of the audio semantic vectors has a corresponding index.

[0052] In some embodiments, the reference audio vector corresponding to each frame of reference audio can be obtained based on the w2v-BERT model. That is, audio_vec = w2v-BERT(wav), where wav represents the reference audio (audio frame), and audio_vec represents the corresponding audio frame vector. Then, feature discretization is performed. That is, semantic_tokens = semantic_quantizer.encoder(audio_vec), where semantic_quantizer represents an audio semantic quantizer (i.e., the preset audio semantic vector table), and its function is to convert the audio vector into a discretized representation of audio features. That is, the reference audio semantic token sequence includes multiple reference audio semantic tokens.

[0053] In the above example, the encoding process of the audio semantic quantizer (i.e., the preset audio semantic vector table) can be expressed as: semantic_codebook:video_semantic_vec->video_semantic_token. That is, the ID (i.e., index) corresponding to the audio semantic vector with the closest distance (e.g., Euclidean distance) to the corresponding audio vector in the audio semantic vector table is used as its audio semantic token.

[0054] In some examples, each audio semantic vector in the audio semantic vector corresponds to a corresponding pronunciation unit respectively, and each audio semantic vector has a corresponding index. For example, the pronunciation unit can refer to a phoneme, a syllable, or other speech units. The original audio waveform is converted into a high-dimensional continuous vector through a deep learning model (such as Wav2Vec 2.0), and each vector implicitly contains acoustic features (timbre, intonation, emotion, etc.) and semantic information.

[0055] In some examples, multiple semantic vectors corresponding to pronunciation units can be clustered to obtain multiple cluster centers, and the vector representation of the cluster centers is used as the vector representation in the audio semantic vector table. For example, the number of vectors in the audio semantic vector table can be 1024. For example, during the prediction process, the index (token) corresponding to the vector with the closest distance in the audio semantic vector table is selected as the discrete representation of the current audio frame.

[0056] In some examples, the audio semantic vector table can be trained.

[0057] According to some embodiments, the preset audio semantic vector table is trained through the following operations: obtaining the audio to be processed, multiple audio frames corresponding to the audio to be processed, and text labels corresponding to the audio to be processed; determining the audio vector corresponding to each audio frame in the multiple audio frames; for each audio frame, determining the audio semantic vector with the closest distance to the audio vector corresponding to the audio frame in the audio semantic vector table, so as to use the audio vector corresponding to the audio frame and the audio semantic vector with the closest distance as a vector pair; based on the vector pairs corresponding to the multiple audio frames respectively, determining a first loss value through a first loss function; based on the audio semantic vectors with the closest distance corresponding to the multiple audio frames respectively, predicting the target text corresponding to the audio to be processed; based on the target text and the text labels, determining a second loss value through a second loss function; and adjusting the corresponding audio semantic vectors in the audio semantic vector table based on the first loss value and the second loss value.

[0058] According to some embodiments, the first loss function includes a mean square error loss function, and the second loss function includes a maximum likelihood loss function.

[0059] Among them, the first loss function (loss_quantizer) is used to calculate the difference between the input feature x (the audio vector corresponding to the audio frame) and its corresponding quantization vector quantize (i.e., the audio semantic vector with the closest distance). For example, the first loss function can adopt the mean squared error loss function, as shown below:

[0060]

[0061] Among them, N represents the number of vectors in the audio semantic vector table.

[0062] In some examples, it can be trained based on speech recognition technology (ASR). For example, after obtaining the audio semantic vectors with the closest distances corresponding to multiple audio frames respectively, a softmax layer is connected for classification to obtain the predicted text corresponding to the audio. The second loss function (loss_ctc) is the loss function corresponding to the audio-to-text task ASR. For example, the second loss function can adopt the maximum likelihood loss function, and the goal is to maximize the probability of the correct output sequence generated based on the given input sequence.

[0063] Therefore, the loss function loss = loss_quantizer + loss_ctc, that is, the corresponding audio semantic vectors in the audio semantic vector table are adjusted based on the first loss value and the second loss value. By training the audio semantic vector table, the accuracy of the labeled sequence obtained through the audio semantic vector table in representing audio features can be improved.

[0064] According to some embodiments, discretizing the features of the reference audio to obtain the corresponding reference audio acoustic label sequence includes: dividing the reference audio into frames to obtain multiple frames of reference audio; for each frame of reference audio in the multiple frames of reference audio, determining the reference audio vector corresponding to the frame of reference audio; for each frame of reference audio in the multiple frames of reference audio, in the preset audio acoustic vector table, determining the audio acoustic vector with the closest distance to the reference audio vector corresponding to the frame of reference audio and the index corresponding to the audio acoustic vector with the closest distance, and using the index as its corresponding reference audio acoustic label; based on the indexes corresponding to the multiple frames of reference audio respectively, determining the reference audio acoustic label sequence. The preset audio acoustic vector table includes multiple audio acoustic vectors, each audio acoustic vector in the multiple audio acoustic vectors corresponds to a corresponding audio feature respectively, and each audio acoustic vector has a corresponding index.

[0065] In this embodiment, the preset audio acoustic vector table is similar to the above-mentioned audio semantic vector table, except that the audio acoustic vectors are directly related to the audio signals and usually contain specific acoustic features such as pitch, volume, duration, etc. For example, the audio acoustic vectors in the audio acoustic vector table correspond to the corresponding audio features, which can be used to represent basic speech features, background noise or environmental sound features, musical instrument sound features, etc. Details are not described herein again.

[0066] According to some embodiments, decoding the target audio based on the target audio acoustic tag sequence includes: determining, based on the preset audio acoustic vector table, the audio acoustic vectors corresponding to each target audio acoustic tag in the target audio acoustic tag sequence; and decoding the target audio based on all the audio acoustic vectors corresponding to the target audio acoustic tag sequence.

[0067] Figure 3 FIG. shows a schematic diagram of discretizing the features of an audio to obtain a corresponding tag sequence according to an embodiment of the present disclosure. As Figure 3 shown, the audio is converted into a set of vectors by an encoder, and the set of vectors is converted into a set of tag sequences by an audio semantic quantizer (i.e., RVQ) as a discretized representation of the audio features.

[0068] Correspondingly, the decoding process of the audio semantic quantizer can be expressed as: semantic_codebook:video_semantic_token->video_semantic_vec. That is, by looking up the audio semantic vector table, the audio semantic tokens are converted into corresponding audio semantic vectors.

[0069] Figure 4 FIG. shows a schematic diagram of decoding a corresponding audio based on a tag sequence according to an embodiment of the present disclosure. As Figure 4 shown, a set of tag sequences is converted into a set of vectors by an audio semantic quantizer (i.e., RVQ), and the set of vectors is converted into the corresponding audio by a decoder.

[0070] According to the above embodiment, in step 240, based on the text semantic tag sequences corresponding to the target text and the reference text respectively, and the reference audio semantic tag sequence, a target audio semantic tag sequence is obtained through a first generative language model.

[0071] Figure 5 FIG. shows a schematic diagram of obtaining a target audio semantic tag sequence according to an embodiment of the present disclosure. As Figure 5As shown, the text semantic marker sequences corresponding to the reference text and the target text, as well as the reference audio semantic marker sequence, are input into the first generative language model (i.e., Text-to-Semantic GPT) to gradually predict the target audio semantic marker sequence. Through the generative language model, subsequent target audio semantic markers are generated sequentially based on the given context, and there is no need to predict the length of the target audio. The generative language model is better at generating coherent audio and simulating the tone, speaking habits, style, etc. of the reference audio. Therefore, in voice synthesis, it can produce a more natural and more human-like speaking effect.

[0072] According to the above embodiment, in step 250, based on the reference audio semantic marker sequence, the target audio semantic marker sequence, and the reference audio acoustic marker sequence, the target audio acoustic marker sequence is obtained through the second generative language model.

[0073] According to some embodiments, the reference audio acoustic marker sequence includes N layers of reference audio acoustic marker sequences, and the N layers of reference audio acoustic marker sequences are obtained by discretizing the features of the reference audio based on N preset audio acoustic vector tables respectively, where N≥2 and N is a positive integer. The target audio acoustic marker sequence includes N layers of target audio acoustic marker sequences.

[0074] In the above embodiment, each layer of the reference audio acoustic marker sequence includes a plurality of reference audio acoustic markers, and the plurality of reference audio acoustic markers correspond to multiple audio frames of the reference audio.

[0075] According to some embodiments, the N preset audio acoustic vector tables respectively correspond to different types of audio features.

[0076] In some examples, each audio acoustic vector table (i.e., quantizer) captures different levels or types of information. For example, the first audio acoustic vector table may focus on the basic speech features of the audio, the second audio acoustic vector table may capture background noise or ambient sound, and other audio acoustic vector tables may process instrument sounds, rhythms, etc. This hierarchical processing method allows the model to understand the audio content at a higher level while also capturing details at a lower level.

[0077] According to some embodiments, the second generative language model includes a third generative language model and a fourth generative language model. Accordingly, obtaining the N-layer target audio acoustic marker sequence through the second generative language model includes: obtaining a first-layer target audio acoustic marker sequence through the third generative language model based on the first-layer reference audio acoustic marker sequence among the reference audio semantic marker sequence, the target audio semantic marker sequence, and the reference audio acoustic marker sequence; predicting and obtaining a next-layer target audio acoustic marker sequence through the fourth generative language model based on the N-layer reference audio acoustic marker sequence and the first-layer target audio acoustic marker sequence; in response to determining that the next-layer target audio acoustic marker sequence is not the N-layer target audio acoustic marker sequence, predicting and obtaining a new next-layer target audio acoustic marker sequence through the fourth generative language model based on the N-layer reference audio acoustic marker sequence and the currently obtained multi-layer target audio acoustic marker sequences; and in response to determining that the next-layer target audio acoustic marker sequence is the N-layer target audio acoustic marker sequence, using the currently obtained multi-layer target audio acoustic marker sequences as the N-layer target audio acoustic marker sequence.

[0078] Figures 6-7 FIG. shows a schematic diagram of obtaining an N-layer target audio acoustic marker sequence according to an embodiment of the present disclosure. As Figure 6 shown, the first-layer reference audio acoustic marker sequence in the reference audio acoustic marker sequence is used as a coarse-grained reference audio acoustic marker sequence, and a coarse-grained target audio acoustic marker sequence is obtained through a third generative language model (i.e., Semantic-to-Acoustic GPT) based on the reference audio semantic marker sequence, the target audio semantic marker sequence, and the coarse-grained reference audio acoustic marker sequence. Then, as Figure 7 shown, a multi-layer input and multi-layer output module is designed in the fourth generative language model. For example, the fourth generative language model can be implemented based on a Transformer model, that is, multi-layer input and multi-layer output units are added to the Transformer model. The N-layer reference audio acoustic marker sequence and the reference Figure 6The obtained coarse-grained target audio acoustic label sequence (i.e., Layer = 1, corresponding to the first quantizer codebook1; other layers are empty) is input into the fourth generative language model (i.e., Corase-to-Fine Acoustic Transformer) to obtain a predicted new next-layer target audio acoustic label sequence (i.e., the second-layer target audio acoustic label sequence, i.e., Layer = 2, corresponding to the second quantizer codebook2). Then, the N-layer reference audio acoustic label sequence and the currently obtained target audio acoustic label sequence (i.e., Layer = 1, 2; other layers are empty) are input into the fourth generative language model to obtain a predicted new next-layer target audio acoustic label sequence... until the N-layer target audio acoustic label sequence is predicted layer by layer (in Figure 7 where N = 8).

[0079] Therefore, in some embodiments, decoding the target audio based on the target audio acoustic label sequence may include: decoding the target audio based on the N-layer target audio acoustic label sequence.

[0080] Through the embodiments of the present disclosure, the naturalness and coherence of speech synthesis can be significantly improved, the phenomenon of nonsense and the phenomenon of rigid pauses in the speech synthesis process can be reduced; at the same time, audio semantic information and acoustic information are fused, and there is no need for explicit alignment training between text and speech, and there is no need for audio length prediction.

[0081] According to an embodiment of the present disclosure, as Figure 8 shown, there is also provided a model training method 800, including: obtaining a first label sequence, a second label sequence, and a label sequence (step 810); inputting the first label sequence and the second label sequence into a fifth generative language model to predict a first label and its corresponding probability (step 820); inputting the first label sequence, the second label sequence, and the first label in the label sequence into the fifth generative language model to predict a next label and its corresponding probability (step 830); in response to determining that the next label is not an end symbol, inputting the first label sequence, the second label sequence, and the label sequence formed by the labels corresponding to all the currently predicted labels in the label sequence into the fifth generative language model to predict a new next label and its corresponding probability (step 840); in response to determining that the next label is an end symbol, determining a third loss value through a third loss function based on the label sequence, all the predicted labels and their corresponding probabilities (step 850); and adjusting the parameters of the fifth generative language model based on the third loss value (step 860).

[0082] According to some embodiments, the first tag sequence includes: a text semantic tag sequence obtained by discretizing the features of the target text and the reference text; the second tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the features of the reference audio; the tag sequence includes: a target audio semantic tag obtained by discretizing the features of the target audio, wherein the reference text is the text corresponding to the reference audio, the target text is the text corresponding to the target audio, and the target audio and the reference audio have the same audio features.

[0083] Specifically, input the text semantic tag sequences corresponding to the target text and the reference text and the reference audio semantic tag sequence into the fifth generative language model to predict the first target audio semantic tag and its corresponding probability; input the text semantic tag sequences corresponding to the target text and the reference text and the reference audio semantic tag sequence, and the first target audio semantic tag in the tag sequence into the fifth generative language model to predict the next target audio semantic tag and its corresponding probability; in response to determining that the next target audio semantic tag is not an end symbol, input the text semantic tag sequences corresponding to the target text and the reference text and the reference audio semantic tag sequence, and the target audio semantic tags corresponding to all the currently predicted tags in the tag sequence into the fifth generative language model to predict the new next target audio semantic tag and its corresponding probability; in response to determining that the next target audio semantic tag is an end symbol, based on the tag sequence, all the predicted tags and their corresponding probabilities, determine the third loss value through the third loss function; and adjust the parameters of the fifth generative language model based on the third loss value.

[0084] In some embodiments, the third loss function may include a cross-entropy loss function. For example, the cross-entropy loss function L for each time step t t can be expressed as:

[0085]

[0086] where V represents the size of the corresponding audio semantic vector table; p t,i represents the probability that the target audio semantic tag predicted by the model at time step t is the i-th tag (i.e., index) in the audio semantic vector table; y t,i represents the one-hot encoding of the true tag in the tag sequence corresponding to time step t (if the true tag is the i-th tag predicted, then y t,i = 1, otherwise 0). The loss function corresponding to the entire tag sequence can be obtained by summing the losses of all time steps, that is:

[0087]

[0088] Wherein, T represents the number of tokens in the token sequence.

[0089] According to some embodiments, the first token sequence includes: a reference audio semantic token sequence obtained by discretizing the features of the reference audio, and a target audio semantic token obtained by discretizing the features of the target audio; the second token sequence includes: a reference audio acoustic token sequence obtained by discretizing the features of the reference audio; the token sequence includes: a target audio acoustic token obtained by discretizing the features of the target audio, wherein the target audio and the reference audio have the same audio features.

[0090] Specifically, the reference audio semantic token sequence, the target audio semantic token, and the reference audio acoustic token sequence are input into a fifth generative language model to predict the first target audio acoustic token and its corresponding probability; the reference audio semantic token sequence, the target audio semantic token, the reference audio acoustic token sequence, and the first target audio acoustic token in the token sequence are input into the fifth generative language model to predict the next target audio acoustic token and its corresponding probability; in response to determining that the next target audio acoustic token is not an end symbol, the reference audio semantic token sequence, the target audio semantic token, the reference audio acoustic token sequence, and the target audio acoustic tokens in the token sequence corresponding to all the currently predicted tokens are input into the fifth generative language model to predict a new next target audio acoustic token and its corresponding probability; in response to determining that the next target audio acoustic token is an end symbol, based on the token sequence, all the predicted tokens and their corresponding probabilities, a third loss value is determined by a third loss function; and the parameters of the fifth generative language model are adjusted based on the third loss value.

[0091] According to some embodiments, the trained language model is used to implement the method described in any one of the above embodiments.

[0092] According to an embodiment of the present disclosure, as Figure 9As shown, a target audio generation agent 900 is also provided, including: an input module 910 configured to: obtain a reference audio, a reference text, and a target text, where the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; a processing module 920 configured to perform the following operations: based on discretizing the features of the reference audio, obtain a reference audio acoustic marker sequence and a reference audio semantic marker sequence; discretize the features of the target text and the reference text respectively to obtain their respective corresponding text semantic marker sequences; based on the text semantic marker sequences corresponding to the target text and the reference text respectively, and the reference audio semantic marker sequence, obtain a target audio semantic marker sequence through a first generative language model; based on the reference audio semantic marker sequence, the target audio semantic marker sequence, and the reference audio acoustic marker sequence, obtain a target audio acoustic marker sequence through a second generative language model; and based on the target audio acoustic marker sequence, decode to obtain the target audio; and an output module 930 configured to output the target audio.

[0093] According to some embodiments, the processing module is further configured to perform the method described in any one of the above embodiments.

[0094] According to an embodiment of the present disclosure, as Figure 10 As shown, a target audio generation device 1000 is also provided, including: a first acquisition unit 1010 configured to obtain a reference audio, a reference text, and a target text, where the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; a first feature discretization unit 1020 configured to, based on discretizing the features of the reference audio, obtain a reference audio acoustic marker sequence and a reference audio semantic marker sequence; a second feature discretization unit 1030 configured to discretize the features of the target text and the reference text respectively to obtain their respective corresponding text semantic marker sequences; a first processing unit 1040 configured to, based on the text semantic marker sequences corresponding to the target text and the reference text respectively, and the reference audio semantic marker sequence, obtain a target audio semantic marker sequence through a first generative language model; a second processing unit 1050 configured to, based on the reference audio semantic marker sequence, the target audio semantic marker sequence, and the reference audio acoustic marker sequence, obtain a target audio acoustic marker sequence through a second generative language model; and a decoding unit 1060 configured to, based on the target audio acoustic marker sequence, decode to obtain the target audio.

[0095] Here, the operations of the above units 1010 to 1060 of the target audio generation device 1000 are respectively similar to the operations of steps 210 to 260 described above, and will not be elaborated here.

[0096] According to an embodiment of the present disclosure, as Figure 11 shown, there is also provided a model training device 1100, including: a second acquisition unit 1110 configured to acquire a first token sequence, a second token sequence, and a label sequence; a first prediction unit 1120 configured to input the first token sequence and the second token sequence into a fifth generative language model to predict a first token and its corresponding probability; a second prediction unit 1130 configured to input the first token sequence, the second token sequence, and the first token in the label sequence into the fifth generative language model to predict a next token and its corresponding probability; a first determination unit 1140 configured to, in response to determining that the next token is not an end token, input the first token sequence, the second token sequence, and a token sequence formed by tokens corresponding to all currently predicted tokens in the label sequence into the fifth generative language model to predict a new next token and its corresponding probability; a second determination unit 1150 configured to, in response to determining that the next token is an end token, determine a third loss value based on the label sequence, all predicted tokens, and their corresponding probabilities through a third loss function; and an adjustment unit 1160 configured to adjust the parameters of the fifth generative language model based on the third loss value.

[0097] Here, the operations of the above units 1110 to 1160 of the model training device 1100 are respectively similar to the operations of steps 810 to 860 described above, and will not be elaborated here.

[0098] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved are all in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0099] According to an embodiment of the present disclosure, there is also provided an electronic device, a readable storage medium, and a computer program product.

[0100] Refer to Figure 12, a block diagram of an electronic device 1200 that can be a server or a client of the present disclosure will now be described. It is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0101] As Figure 12 shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the electronic device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0102] A plurality of components in the electronic device 1200 are connected to the I / O interface 1205, including: an input unit 1206, an output unit 1207, a storage unit 1208, and a communication unit 1209. The input unit 1206 can be any type of device that can input information into the electronic device 1200. The input unit 1206 can receive input digital or character information, and generate key signal inputs related to the user settings and / or function controls of the electronic device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and / or a remote control. The output unit 1207 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1208 can include, but is not limited to, magnetic disks, optical disks. The communication unit 1209 allows the electronic device 1200 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0103] The computing unit 1201 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 executes the various methods and processes described above, such as method 200 or 800. For example, in some embodiments, method 200 or 800 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of method 200 or 800 described above may be executed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute method 200 or 800 in any other suitable manner (e.g., by means of firmware).

[0104] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), systems-on-a-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0107] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0108] The systems and techniques described herein can be implemented in a computing system that includes backend components (such as, for example, a data server), or a computing system that includes middleware components (such as, for example, an application server), or a computing system that includes frontend components (such as, for example, a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (such as, for example, a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0109] A computer system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0110] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitation is imposed herein.

[0111] Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only defined by the authorized claims and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, the steps can be executed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples can be combined in various ways. Importantly, with the evolution of technology, many of the elements described herein can be replaced by equivalent elements that emerge after the present disclosure.

Claims

1. A target audio generation method, comprising: Acquire reference audio, reference text and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; Based on discretizing the features of the reference audio, a reference audio acoustic label sequence and a reference audio semantic label sequence are obtained; Discretize the target text and the reference text respectively to obtain their corresponding text semantic tag sequences; Based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence, obtaining a target audio semantic tag sequence through a first generative language model; Based on the reference audio semantic label sequence, the target audio semantic label sequence and the reference audio acoustic label sequence, obtaining a target audio acoustic label sequence through a second generative language model; and Based on the target audio acoustic marker sequence, the target audio is decoded and obtained.

2. The method of claim 1, wherein: The reference audio acoustic label sequence includes N layers of reference audio acoustic label sequences, and the N layers of reference audio acoustic label sequences are obtained by discretizing the reference audio based on N preset audio acoustic vector tables, where N≥2 and N is a positive integer, and wherein the second generative language model includes a third generative language model and a fourth generative language model, and the target audio acoustic label sequence includes N layers of target audio acoustic label sequences, wherein obtaining the target audio acoustic label sequence through the second generative language model includes: Based on the reference audio semantic label sequence, the target audio semantic label sequence, and the first layer reference audio acoustic label sequence in the reference audio acoustic label sequence, obtaining a first layer target audio acoustic label sequence through a third generative language model; Based on the N layers of reference audio acoustic label sequences and the first layer of target audio acoustic label sequences, obtaining a next layer of target audio acoustic label sequences through prediction by a fourth generative language model; In response to determining that the next-layer target audio acoustic label sequence is not the N-th layer target audio acoustic label sequence, based on the N-layer reference audio acoustic label sequence and the currently obtained multi-layer target audio acoustic label sequence, obtain a new next-layer target audio acoustic label sequence through the fourth generative language model prediction; and In response to determining that the next layer of target audio acoustic label sequence is the Nth layer of target audio acoustic label sequence, using the currently obtained multi-layer target audio acoustic label sequence as the Nth layer of target audio acoustic label sequence; The step of decoding the target audio based on the target audio acoustic marker sequence includes: decoding the target audio based on the N layers of target audio acoustic marker sequences.

3. The method of claim 2, wherein: The N preset audio acoustic vector tables correspond to different types of audio features respectively.

4. The method of claim 1, wherein: Discretizing the features of the reference audio to obtain a corresponding reference audio semantic tag sequence includes: Divide the reference audio into frames to obtain multiple frames of reference audio; For each frame of reference audio in the multiple frames of reference audio, determining a reference audio vector corresponding to the frame of reference audio; For each frame of reference audio in the multiple frames of reference audio, determine, in a preset audio semantic vector table, an audio semantic vector that is closest to the reference audio vector corresponding to the frame of reference audio and an index corresponding to the audio semantic vector that is closest, so as to use the index as the corresponding reference audio semantic tag; Determining the reference audio semantic tag sequence based on the indexes corresponding to the multiple frames of reference audio, The preset audio semantic vector table includes a plurality of audio semantic vectors, each of the plurality of audio semantic vectors corresponds to a corresponding pronunciation unit, and each of the audio semantic vectors has a corresponding index.

5. The method according to any one of claims 1 to 3, wherein: Discretizing the features of the reference audio to obtain a corresponding reference audio acoustic label sequence includes: Divide the reference audio into frames to obtain multiple frames of reference audio; For each frame of reference audio in the multiple frames of reference audio, determining a reference audio vector corresponding to the frame of reference audio; For each frame of reference audio in the multiple frames of reference audio, determine, in a preset audio acoustic vector table, an audio acoustic vector that is closest to the reference audio vector corresponding to the frame of reference audio and an index corresponding to the audio acoustic vector that is closest to the reference audio vector, so as to use the index as the corresponding reference audio acoustic label; Determining the reference audio acoustic marker sequence based on the indexes corresponding to the multiple frames of reference audio, The preset audio acoustic vector table includes a plurality of audio acoustic vectors, each of the plurality of audio acoustic vectors corresponds to a corresponding audio feature, and each of the audio acoustic vectors has a corresponding index.

6. The method of claim 4, wherein: The preset audio semantic vector table is obtained by training through the following operations: Acquire audio to be processed, multiple audio frames corresponding to the audio to be processed, and a text label corresponding to the audio to be processed; Determine an audio vector corresponding to each audio frame in the plurality of audio frames; For each audio frame, determining in the audio semantic vector table an audio semantic vector that is closest to the audio vector corresponding to the audio frame, so as to use the audio vector corresponding to the audio frame and the audio semantic vector that is closest to the audio vector as a vector pair; Determine a first loss value by a first loss function based on the vector pairs corresponding to each of the plurality of audio frames; Predicting a target text corresponding to the audio to be processed based on the audio semantic vectors with the closest distance corresponding to each of the multiple audio frames; Based on the target text and the text label, determining a second loss value by a second loss function; as well as The corresponding audio semantic vector in the audio semantic vector table is adjusted based on the first loss value and the second loss value.

7. The method of claim 6, wherein: The first loss function includes a mean square error loss function, and the second loss function includes a maximum likelihood loss function.

8. The method of claim 5, wherein: Based on the target audio acoustic marker sequence, decoding to obtain the target audio includes: Based on the preset audio acoustic vector table, determining the audio acoustic vector corresponding to each target audio acoustic marker in the target audio acoustic marker sequence in the preset audio acoustic vector table; and Based on all audio acoustic vectors corresponding to the target audio acoustic marker sequence, the target audio is decoded to obtain the target audio.

9. A model training method, comprising: Obtaining a first marker sequence, a second marker sequence, and a tag sequence; Inputting the first tag sequence and the second tag sequence into a fifth generative language model to predict a first tag and its corresponding probability; Inputting the first tag sequence, the second tag sequence, and the first tag in the tag sequence into the fifth generative language model to predict the next tag and its corresponding probability; In response to determining that the next token is not a terminator, inputting the first token sequence, the second token sequence, and a token sequence formed by tokens corresponding to all currently predicted tokens in the tag sequence into the fifth generative language model to predict a new next token and its corresponding probability; In response to determining that the next token is a terminator, determining a third loss value by a third loss function based on the tag sequence, all predicted tokens and their corresponding probabilities; and Parameters of the fifth generative language model are adjusted based on the third loss value.

10. The method of claim 8, wherein: The first tag sequence includes: a text semantic tag sequence obtained by discretizing the features of the target text and the reference text; The second tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the features of the reference audio; The label sequence includes: a target audio semantic tag obtained after discretizing the features of the target audio, wherein the reference text is the text corresponding to the reference audio, the target text is the text corresponding to the target audio, and the target audio and the reference audio have the same audio features.

11. The method of claim 8, wherein: The first tag sequence includes: a reference audio semantic tag sequence obtained by discretizing the features of the reference audio, and a target audio semantic tag obtained by discretizing the features of the target audio; The second label sequence includes: a reference audio acoustic label sequence obtained by discretizing the features of the reference audio; The label sequence includes: a target audio acoustic label obtained after discretizing the features of the target audio, wherein the target audio and the reference audio have the same audio features.

12. The method according to any one of claims 9 to 11, wherein: The trained language model is used to implement the method described in any one of claims 1-8.

13. A target audio generation agent, comprising: An input module is configured to: obtain reference audio, reference text and target text, wherein the reference text is the text corresponding to the reference audio, and the target text is the text corresponding to the target audio to be generated; A processing module configured to perform the following operations: Based on discretizing the features of the reference audio, a reference audio acoustic label sequence and a reference audio semantic label sequence are obtained; Discretize the target text and the reference text respectively to obtain their corresponding text semantic tag sequences; Based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence, obtaining a target audio semantic tag sequence through a first generative language model; Based on the reference audio semantic label sequence, the target audio semantic label sequence and the reference audio acoustic label sequence, obtaining a target audio acoustic label sequence through a second generative language model; and Based on the target audio acoustic marker sequence, decoding to obtain the target audio; An output module is configured to output the target audio.

14. An agent as claimed in claim 13, wherein: The processing module is further configured to execute the method of any one of claims 2-8.

15. A target audio generation device, comprising: A first acquisition unit is configured to acquire a reference audio, a reference text, and a target text, wherein the reference text is a text corresponding to the reference audio, and the target text is a text corresponding to the target audio to be generated; A first feature discretization unit is configured to obtain a reference audio acoustic label sequence and a reference audio semantic label sequence based on discretizing the features of the reference audio; A second feature discretization unit is configured to discretize the features of the target text and the reference text respectively to obtain respective corresponding text semantic tag sequences; A first processing unit is configured to obtain a target audio semantic tag sequence through a first generative language model based on the text semantic tag sequences corresponding to the target text and the reference text, and the reference audio semantic tag sequence; a second processing unit configured to obtain a target audio acoustic label sequence through a second generative language model based on the reference audio semantic label sequence, the target audio semantic label sequence and the reference audio acoustic label sequence; and The decoding unit is configured to decode and obtain the target audio based on the target audio acoustic marker sequence.

16. A model training device, comprising: A second acquisition unit is configured to acquire the first marking sequence and the second marking sequence, and the tag sequence; A first prediction unit is configured to input the first tag sequence and the second tag sequence into a fifth generative language model to predict a first tag and a corresponding probability thereof; A second prediction unit is configured to input the first tag sequence, the second tag sequence, and the first tag in the tag sequence into the fifth generative language model to predict a next tag and a corresponding probability thereof; A first determining unit is configured to, in response to determining that the next token is not a terminator, input the first token sequence, the second token sequence, and a token sequence formed by tokens corresponding to all currently predicted tokens in the tag sequence into the fifth generative language model to predict a new next token and its corresponding probability; A second determining unit is configured to determine, in response to determining that the next token is a terminator, a third loss value based on the tag sequence, all predicted tokens and their corresponding probabilities by using a third loss function; as well as An adjustment unit is configured to adjust parameters of the fifth generative language model based on the third loss value.

17. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

19. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images