Task processing method, audio generation model training method, and text processing method

Abstract

A task processing method, an audio generation model training method, and a text processing method, relating to the technical field of computers. The task processing method comprises: determining task data to be processed and target voiceprint data; performing semantic decoding on said task data to obtain an audio discrete feature; and on the basis of the audio discrete feature and the target voiceprint feature, determining a target audio corresponding to said task data. The audio discrete feature, represented as a series of discrete integers, is capable of representing semantic information of said task data, so that the target audio determined on the basis of the audio discrete feature and the target voiceprint feature can have semantic consistency and a correspondence between an audio and a text, thereby ensuring the stability and accuracy of an audio synthesis effect.

Description

Task processing method, audio generation model training method and text processing method

[0001] The present disclosure claims priority to Chinese Patent Application No. 2024108980302, filed on July 4, 2024 with the Chinese Patent Office, entitled "Task processing method, audio generation model training method and text processing method", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present specification relate to the field of computer technology, in particular to a task processing method, an audio generation model training method and a text processing method. BACKGROUND

[0003] Speech synthesis technology is a technology that can convert text into speech. Speech synthesis technology can be used in human-computer interaction systems, such as enabling human-robot communication.

[0004] Currently, text-to-speech conversion can be achieved through a task model. However, the current task model usually only focuses on the acoustic information of the data, without considering the semantic information of the text data and the speech data, resulting in a lack of semantic consistency in the synthesized speech and poor speech synthesis effect. Therefore, there is an urgent need for an effective technical solution to solve the above problems. SUMMARY

[0005] Therefore, the embodiments of the present specification provide two task processing methods. One or more embodiments of the present specification simultaneously relate to two task processing devices, an audio generation model training method, an audio generation model training device, a text processing method, a text processing device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects in the prior art.

[0006] According to a first aspect of the embodiments of the present specification, a task processing method is provided, comprising:

[0007] determining task data to be processed and target voiceprint features;

[0008] performing semantic decoding on the task data to be processed to obtain audio discrete features;

[0009] determining target audio corresponding to the task data to be processed according to the audio discrete features and the target voiceprint features.

[0010] According to a second aspect of the embodiments of the present specification, a task processing device is provided, comprising:

[0011] a first determining module configured to determine task data to be processed and target voiceprint features;

[0012] a decoding module, configured to perform semantic decoding on the to-be-processed task data to obtain audio discrete features;

[0013] a second determining module, configured to determine, according to the audio discrete features and the target voiceprint features, a target audio corresponding to the to-be-processed task data.

[0014] According to a third aspect of an embodiment of the present specification, an audio generation model training method is provided, including:

[0015] determining a machine learning model to be trained and training data associated with an audio generation task, wherein the machine learning model to be trained includes a semantic decoding layer, a spectrum prediction layer, and a spectrum processing layer, and the training data includes sample text, sample audio, sample voiceprint features corresponding to the sample audio, and sample audio discrete features;

[0016] processing the sample text, the sample audio, the sample voiceprint features, and the sample audio discrete features according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result;

[0017] adjusting model parameters of the machine learning model according to the audio generation result to obtain a trained audio generation model.

[0018] According to a fourth aspect of an embodiment of the present specification, an audio generation model training apparatus is provided, including:

[0019] a determining module, configured to determine a machine learning model to be trained and training data associated with an audio generation task, wherein the machine learning model to be trained includes a semantic decoding layer, a spectrum prediction layer, and a spectrum processing layer, and the training data includes sample text, sample audio, sample voiceprint features corresponding to the sample audio, and sample audio discrete features;

[0020] a processing module, configured to process the sample text, the sample audio, the sample voiceprint features, and the sample audio discrete features according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result;

[0021] an adjusting module, configured to adjust model parameters of the machine learning model according to the audio generation result to obtain a trained audio generation model.

[0022] According to a fifth aspect of an embodiment of the present specification, a text processing method is provided, including:

[0023] receiving a text reading instruction for a target text sent by a client, and determining target voiceprint features;

[0024] perform semantic decoding on the target text to obtain an audio discrete feature;

[0025] determine target audio corresponding to the target text according to the audio discrete feature and the target voiceprint feature;

[0026] send the target audio to the client to enable the client to play the target audio.

[0027] According to a sixth aspect of an embodiment of the present specification, a text processing apparatus is provided, comprising:

[0028] a receiving module configured to receive a text reading instruction for a target text sent by a client and determine a target voiceprint feature;

[0029] a decoding module configured to perform semantic decoding on the target text to obtain an audio discrete feature;

[0030] a determining module configured to determine target audio corresponding to the target text according to the audio discrete feature and the target voiceprint feature;

[0031] a sending module configured to send the target audio to the client to enable the client to play the target audio.

[0032] According to a seventh aspect of an embodiment of the present specification, a task processing method is provided, comprising:

[0033] determining conference task data corresponding to a target voiceprint feature and a target conference;

[0034] performing semantic decoding on the conference task data to obtain an audio discrete feature;

[0035] determining target conference data corresponding to the conference task data according to the audio discrete feature and the target voiceprint feature;

[0036] in response to a playing instruction of a client, playing the target conference data according to a preset playing rule.

[0037] According to an eighth aspect of an embodiment of the present specification, a task processing apparatus is provided, comprising:

[0038] a first determining module configured to determine conference task data corresponding to a target voiceprint feature and a target conference;

[0039] a decoding module configured to perform semantic decoding on the conference task data to obtain an audio discrete feature;

[0040] a second determining module configured to determine target conference data corresponding to the conference task data according to the audio discrete feature and the target voiceprint feature;

[0041] The playing module is configured to play the target conference data according to a preset playing rule in response to a playing instruction of the client.

[0042] According to a ninth aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0043] a memory and a processor;

[0044] The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which realize the steps of the above method when executed by the processor.

[0045] According to a tenth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer programs / instructions, which realize the steps of the above method when executed by the processor.

[0046] According to an eleventh aspect of an embodiment of the present specification, a computer program product is provided, comprising computer programs / instructions, which realize the steps of the above method when executed by the processor.

[0047] One embodiment of the present specification provides a task processing method. After determining the to-be-processed task data and the target voiceprint feature, the to-be-processed task data is semantically decoded to obtain an audio discrete feature corresponding to the to-be-processed task data. The audio discrete feature is a series of discrete integer representations, which can represent the semantic information of the to-be-processed task data. The target audio determined according to the audio discrete feature and the target voiceprint feature can have semantic consistency, have a corresponding relationship between the audio and the text, ensure the stability and accuracy of the audio synthesis effect, and enable the target audio to be played in the tone corresponding to the target voiceprint feature when played subsequently. BRIEF DESCRIPTION OF DRAWINGS

[0048] FIG. 1 is a schematic diagram of an application scenario of a task processing method according to one embodiment of the present specification;

[0049] FIG. 2 is a flowchart of a task processing method according to one embodiment of the present specification;

[0050] FIG. 3 is a schematic diagram of a feature sequence in a task processing method according to one embodiment of the present specification;

[0051] FIG. 4 is a structural schematic diagram of an audio generation model in a task processing method according to one embodiment of the present specification;

[0052] FIG. 5 is a processing flowchart of an audio generation model in a task processing method according to one embodiment of the present specification;

[0053] FIG. 6 is a flowchart of a method for training an audio generation model according to an embodiment of the present specification;

[0054] FIG. 7 is a schematic diagram of a training process for a semantic decoding layer in a method for training an audio generation model according to an embodiment of the present specification;

[0055] FIG. 8 is a schematic diagram of a training process for a spectrum prediction layer in a method for training an audio generation model according to an embodiment of the present specification;

[0056] FIG. 9 is a flowchart of a processing procedure of a task processing method according to an embodiment of the present specification;

[0057] FIG. 10 is a flowchart of a text processing method according to an embodiment of the present specification;

[0058] FIG. 11 is a flowchart of another task processing method according to an embodiment of the present specification;

[0059] FIG. 12 is a schematic diagram of a structure of a task processing apparatus according to an embodiment of the present specification;

[0060] FIG. 13 is a schematic diagram of a structure of an audio generation model training apparatus according to an embodiment of the present specification;

[0061] FIG. 14 is a schematic diagram of a structure of a text processing apparatus according to an embodiment of the present specification;

[0062] FIG. 15 is a schematic diagram of a structure of another task processing apparatus according to an embodiment of the present specification;

[0063] FIG. 16 is a block diagram of a structure of a computing device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0064] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, which are not described in the present specification, and it is understood that the scope of the present specification is not limited to the details of the description. In other instances, well-known methods, procedures, components, and networks have not been described in detail for the sake of brevity.

[0065] The terms used in one or more embodiments of the present specification are used only for the purpose of describing particular embodiments and are not intended to limit one or more embodiments of the present specification. The singular forms "a," "an," and "the" used in one or more embodiments of the present specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It will be further understood that the term "and / or" used in one or more embodiments of the present specification means and includes any or all possible combinations of one or more associated listed items.

[0066] It should be understood that, although the terms first, second, etc. can be employed in describing various information in one or more embodiments of the present specification, such information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information of the same type. For example, without departing from the scope of one or more embodiments of the present specification, first can also be referred to as second, and similarly, second can also be referred to as first. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "upon" or "in response to determining".

[0067] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0068] In one or more embodiments of the present specification, a large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, thousands of billions or even tens of billions of model parameters. The large model can also be called a foundation model. Through large-scale unlabeled corpus pre-training, a pre-trained model with hundreds of millions of parameters is output. Such a model can adapt to a wide range of downstream tasks, and the model has good generalization ability. For example, large language model (LLM, Large Language Model), multi-modal pre-training model, etc.

[0069] In actual application, the large model only needs a small amount of sample to fine-tune the pre-trained model and can be applied to different tasks. The large model can be widely applied to natural language processing (NLP, Natural Language Processing) and computer vision fields. Specifically, it can be applied to computer vision field tasks such as visual question answering (VQA, Visual Question Answering), image caption (IC, Image Caption), image generation, and natural language processing field tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0070] First, the nomenclature terms related to one or more embodiments of the present specification are explained.

[0071] Audio discrete features: a series of integer representations obtained by discretizing audio, and each second of speech can correspond to a fixed number of integers.

[0072] Paralanguage: can be understood as non-verbal sounds, such as laughter, breathing, coughing, emphasis, etc.

[0073] Transformer: a deep learning model based on self-attention mechanism, which can consider all positions of input in parallel when processing sequence data.

[0074] Convolutional Transformer U-Net: a deep learning model that combines the advantages of convolutional neural networks (CNN), Transformers, and U-Net architecture. This model is commonly used for computer vision tasks such as image segmentation, object detection, and semantic understanding.

[0075] Classifier-free guidance: a technique used in generative models to improve the performance of conditional generation tasks while reducing the dependence on additional classifiers. It allows the model to learn both conditional and unconditional (i.e., unconditional) generation during training. This means that the model not only accepts samples with condition signals during training, but also accepts samples without condition signals. When the model receives a condition signal during the generation phase, it can use the difference between the previously learned conditional and unconditional generation to generate data in a more fine-grained and controllable manner.

[0076] Teacher forcing: a model training strategy that uses the core idea of using the true target output as input at each time step during training, rather than the model's own prediction at the previous time step.

[0077] In practical applications, the main purpose of speech synthesis technology is to synthesize the corresponding speech of the text synthesis content, which has poor effects in terms of naturalness, prosody, and other aspects, and lacks the ability to rapidly replicate, control emotions, and control granularity. In the stage of large model generation, people's demand for natural speech interaction with large models is increasing, and they hope that the speech output by large models can be rich in emotion, personification, and naturalness. In this context, the existing speech synthesis system cannot meet the above requirements.

[0078] To solve the above technical problems, in the present specification, two task processing methods are provided. One or more embodiments of the present specification simultaneously relate to two task processing devices, an audio generation model training method, an audio generation model training device, a text processing method, a text processing device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0079] Referring to FIG. 1, FIG. 1 shows a schematic diagram of an application scenario of a task processing method according to an embodiment of the present specification, the task processing method comprising the following steps: determining to-be-processed task data and target voiceprint features; performing semantic decoding on the to-be-processed task data to obtain audio discrete features; and determining target audio corresponding to the to-be-processed task data according to the audio discrete features and the target voiceprint features.

[0080] The end-side device 102 and the cloud-side device 104 are included in FIG. 1.

[0081] A user can log in to a certain website through the end-side device 102, and the website can be used to provide an audio synthesis service. After the user logs in to the website, the user can input to-be-processed text on a display interface of the end-side device 102. The end-side device 102 sends the to-be-processed text to the cloud-side device 104. The cloud-side device 104 can determine target voiceprint features of a preset voice and perform semantic decoding on the to-be-processed text to obtain audio discrete features corresponding to the to-be-processed text. The cloud-side device 104 determines target audio corresponding to the to-be-processed text according to the audio discrete features and the target voiceprint features. The cloud-side device 104 sends the target audio to the end-side device 102. The display interface of the end-side device 102 displays a play control for the target audio. The end-side device 102 plays the target audio according to a point selection instruction of the user for the play control, thereby realizing the audio synthesis service.

[0082] The terminal-side device 102 can include a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5) application, or a light application (also known as a small program, a lightweight application program), or a cloud application, and the like, which can be developed based on a software development kit (SDK) of a corresponding service provided by the server-side, such as a real-time communication (RTC) SDK, and the like. The terminal-side device can be deployed in an electronic device, and needs to be run in dependence on a device or an APP in the device, and the like. The electronic device can have a display screen and support information browsing, and the like, and can be a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, and the like. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant communication tools, mailbox clients, social platform software, and the like.

[0083] The cloud-side device 104 can be understood as a server providing various services, including a physical server, a cloud server, such as a server providing communication services for multiple clients, such as a server for background training supporting a model used on a client, such as a server processing data sent by a client, and the like. It should be noted that the cloud-side device 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The cloud-side device 104 can also be a server of a distributed system, or a server combined with a blockchain. The cloud-side device 104 can also be a cloud server of cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and the like basic cloud computing services, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0084] It should be noted that the text processing method provided in the embodiments of the present specification can be executed by the cloud-side device 104, and in other embodiments of the present specification, the audio generation model can be deployed in the terminal-side device 102, so that the terminal-side device 102 can also have similar functions as the cloud-side device 104, thereby executing the text processing method provided in the embodiments of the present specification; in other embodiments, the text processing method provided in the embodiments of the present specification can also be executed by the terminal-side device 102 and the cloud-side device 104 together.

[0085] Referring to FIG. 2, FIG. 2 shows a flowchart of a task processing method according to an embodiment of the present specification, which specifically includes the following steps.

[0086] Step 202: determining task data to be processed and target voiceprint features.

[0087] Specifically, the task processing method provided by the embodiments of the present specification can be applied to an application program, a website or an applet with a text processing function, and a text processing task can be performed on the application program, the website or the applet. For example, a website deploying an audio generation model can perform a task of synthesizing audio from text. For another example, a novel reading application program can provide a novel reading service, and can perform an audio synthesis task on novel text to obtain audio and play the audio. For another example, a robot can convert a set reply text into audio and play the audio to realize human-computer interaction. For another example, in the field of film production, a script text can be converted into audio to realize tasks such as dubbing of characters in a movie, production of a dubbed film and simultaneous interpretation, which are not limited by the embodiments of the present specification.

[0088] Among them, the task data to be processed can be understood as task data that needs to be synthesized into audio, and the task data to be processed can be task text to be processed or task audio to be processed. In order to facilitate understanding, the embodiments of the present specification take the task data to be processed as the task text to be processed for illustration.

[0089] The task text to be processed can be understood as text data that needs to be synthesized into audio. The task text to be processed can be text data input by a user or text data selected by a user. For example, a user can input a piece of text data “I want a five-fruit moon cake”, or a user selects a novel in a novel reading application program, and the text data corresponding to the novel can be the task text to be processed.

[0090] The target voiceprint feature can be understood as a feature of a sound wave spectrum carrying speech information. The target voiceprint feature is in the form of a feature vector. The target voiceprint feature can represent pitch, intensity, duration and timbre of audio. The target voiceprint feature can be obtained according to a reference audio uploaded by a user. For example, a user wants to synthesize target audio using his own voice when playing the target audio. Therefore, the user can upload audio of his own speech as a reference audio, and extract target voiceprint features according to the reference audio.

[0091] Based on this, the text data to be processed and the reference audio uploaded by the user can be determined, and the target voiceprint features can be extracted according to the reference audio.

[0092] For example, a website provides an audio synthesis service for a user. The user uploads task text to be processed and a reference audio to the website through a client. The website can extract target voiceprint features according to the reference audio.

[0093] In particular implementation, the determining the to-be-processed task data and the target voiceprint feature comprises:

[0094] receiving the to-be-processed task data and the reference audio sent by the client;

[0095] extracting a voiceprint feature from the reference audio to obtain a target voiceprint feature corresponding to the reference audio; or

[0096] receiving the to-be-processed task data and the audio selection instruction sent by the client;

[0097] determining a reference audio according to the audio selection instruction, and extracting a voiceprint feature from the reference audio to obtain a target voiceprint feature corresponding to the reference audio.

[0098] The reference audio can be understood as a reference of a target audio obtained by performing audio synthesis on the to-be-processed task data. It can be understood that the timbre of the target audio can be the same as that of the reference audio. For example, the timbre of the reference audio is "soft female voice", and the timbre of the target audio is also "soft female voice".

[0099] The audio selection instruction can be understood as an audio selection instruction of selecting a candidate audio on a display interface of the client by a user. For example, the display interface of the client displays a plurality of candidate audios 1 "soft female voice", a candidate audio 2 "soft male voice", and a candidate audio 3 "old male voice" to the user. The user can select the candidate audios by clicking. The candidate audio selected by the user can be played through the audio playing device of the client. The user can select a candidate audio as the reference audio from the candidate audios. The instruction of selecting the candidate audio as the reference audio by the user is the audio selection instruction.

[0100] Based on this, in an embodiment of the present specification, the user can directly send the to-be-processed task data and the reference audio to the server through the client. The server can extract a voiceprint feature from the reference audio to obtain a target voiceprint feature corresponding to the reference audio. In another embodiment of the present specification, the user can send the to-be-processed task data to the server through the client. The server can send a plurality of candidate audios to the client. The plurality of candidate audios are displayed on the display interface of the client. The user can perform a clicking operation on the display interface of the client. The client generates an audio selection instruction based on the clicking operation and sends the audio selection instruction to the server. The server determines the candidate audio selected by the user as the reference audio from the plurality of candidate audios according to the audio selection instruction, and extracts a voiceprint feature from the reference audio to obtain a target voiceprint feature corresponding to the reference audio.

[0101] In specific implementation, the voiceprint feature of the reference audio can be extracted by using a deep learning model or a neural network model. Alternatively, the reference audio can be segmented into multiple frames of audio signals, a Fourier transform is performed on each frame of audio signal to obtain frequency domain information of each frame of audio signal, and the voiceprint feature of the reference audio is extracted according to the frequency domain information of each frame of audio signal.

[0102] In summary, by interacting with the client to obtain the to-be-processed task data and the reference audio, the target voiceprint feature based on the reference audio is obtained, and the to-be-processed task data is synthesized into an audio with the same timbre as the reference audio.

[0103] Step 204: performing semantic decoding on the to-be-processed task data to obtain audio discrete features.

[0104] The audio discrete features can be understood as a series of discrete integer representations. The audio discrete features can be used to represent standard features of acoustic information and semantic information in the to-be-processed task data, and are in the form of a feature vector. The audio discrete features are obtained by quantizing the semantic features. In the case of preserving acoustic information and speech information, the standardization and / or compression of the features are realized, and high-quality and / or efficient coding of the features is realized. For example, for the audio "don't be late for the meeting tomorrow", the audio discrete features not only contain the acoustic information of the basic pronunciation of each word, but also contain the semantic information about the time (tomorrow), the time (meeting), the attitude (don't be late), and the like. The audio discrete features accurately represent the text semantic information of the to-be-processed task data, so that there is enough semantic information for subsequent audio synthesis, and the understanding of the text semantic information of the to-be-processed task data is ensured.

[0105] Specifically, the to-be-processed task data can be decoded based on the target voiceprint feature to obtain audio discrete features corresponding to the to-be-processed task data.

[0106] In actual application, the semantic decoding of the to-be-processed task data can be realized based on an autoregressive Transformer model, and the autoregressive Transformer model is trained according to sample audio discrete features.

[0107] In specific implementation, the to-be-processed task data includes texts of multiple language categories.

[0108] Before the to-be-processed task data is decoded to obtain audio discrete features, the method further includes:

[0109] The to-be-processed task data, the multiple language categories, and the target voiceprint feature are used to construct a feature sequence.

[0110] The semantic decoding of the to-be-processed task data obtains audio discrete features, including:

[0111] The semantic decoding of the feature sequence obtains audio discrete features.

[0112] Wherein, the plurality of language categories include but are not limited to Chinese, English, Japanese, Korean, etc., and then the to-be-processed task data can include text of only one language category, such as Chinese to-be-processed task data, the to-be-processed task data can include text of multiple language categories, such as to-be-processed task data including Chinese text and English text, or to-be-processed task data including Chinese text, English text and Japanese text.

[0113] Based on this, the to-be-processed task data, the plurality of language categories and the target voiceprint features can be constructed into a feature sequence, which can include a sequence start symbol, a target voiceprint feature, to-be-processed task data, a category label of a language category and a sequence end symbol in turn. The sequence start symbol can be used to indicate the beginning of the feature sequence, and the sequence end symbol can be used to indicate the end of the feature sequence.

[0114] In specific implementation, when constructing the feature sequence, the category label of the language category can be before the text corresponding to the language category, such as the to-be-processed task data being “I want to climb a mountain”, and the language category of the to-be-processed task data being Chinese, then the feature sequence can include “

Chinese

I want to climb a mountain

English

Mary

Chinese

I want to climb a mountain

[0115] In actual application, the constructed feature sequence can be input into an autoregressive Transformer model, and the autoregressive Transformer model is used to autoregressively decode the audio discrete features of each text unit in the to-be-processed task data step by step until the autoregressive Transformer model detects the sequence end symbol of the feature sequence, and the audio discrete features of each text unit output by the autoregressive Transformer model are determined as the audio discrete features corresponding to the to-be-processed task data.

[0116] In addition, the prompt text, the reference audio discrete feature corresponding to the prompt text, and the description text can be input into the autoregressive Transformer model according to actual needs. The prompt text can be used to prompt the audio effect synthesized by the autoregressive Transformer model. The autoregressive Transformer model can synthesize audio carrying corresponding emotions or styles according to the input description text.

[0117] In an embodiment of the present specification, the to-be-processed task data can further include text corresponding to non-verbal sounds, which can include laughter, breathing, coughing, crying, humming, and the like. Therefore, the to-be-processed task data can be "Hello [breathing sound] My name is Xiaoming", so that the subsequent synthesized target audio can be more natural and personalized, and the communication experience in the field of human-computer interaction can be further improved.

[0118] Referring to FIG. 3, FIG. 3 shows a schematic diagram of a feature sequence in a task processing method according to an embodiment of the present specification. As shown in FIG. 3, S represents a sequence start symbol, E represents a sequence end symbol, and T represents a start symbol of audio discrete features. In an embodiment of the present specification, the feature sequence 302 can include target voiceprint features 1, to-be-processed task data 2, and audio discrete features 3. Specifically, the feature sequence including the target voiceprint features and the to-be-processed task data can be input into the autoregressive Transformer model, and the next audio discrete feature can be predicted by the Transformer model step by step until the sequence end symbol of the feature sequence is detected. The audio discrete features in the feature sequence 302 are the output of the autoregressive Transformer model. Therefore, the target audio determined according to the audio discrete features and the target voiceprint features can realize the synthesis of a specified timbre, so that the timbre of the target audio is the same as that of the reference audio corresponding to the target voiceprint features.

[0119] In summary, by performing step-by-step autoregressive semantic decoding on the to-be-processed task data, the extraction of the audio discrete features is realized, so that the audio discrete features contain the semantic information of the to-be-processed task data, the corresponding relationship between the audio and the text is ensured, and the audio synthesis effect is further ensured.

[0120] Step 206: determining a target audio corresponding to the to-be-processed task data according to the audio discrete features and the target voiceprint features.

[0121] In specific implementation, the determination of the target audio corresponding to the to-be-processed task data according to the audio discrete features and the target voiceprint features includes:

[0122] performing spectrum prediction on the audio discrete features and the target voiceprint features to obtain predicted spectrum features;

[0123] perform spectrum conversion on the predicted spectrum feature to obtain the target audio corresponding to the to-be-processed task data.

[0124] The predicted spectrum feature can be understood as a spectrum feature of the target audio, which can be a mel-spectrum speech feature for example.

[0125] In actual applications, the diffusion model can be used to perform spectrum prediction on the audio discrete feature and the target voiceprint feature to obtain the predicted spectrum feature, and the vocoder can be used to perform spectrum conversion on the predicted spectrum feature to convert the predicted spectrum feature into a target audio waveform, thereby obtaining the target audio corresponding to the to-be-processed task data.

[0126] Optionally, the diffusion model can use an ordinary differential diffusion model, in addition, the diffusion model can also use a denoising diffusion model (Denoising Diffusion Model) and a generative adversarial network model (Generative Adversarial Network), and the vocoder can use a big variational GAN (Big Variational GAN) and a spectrum generation network (Vocos), etc., which are not limited by the embodiments of the present specification.

[0127] In actual applications, the ordinary differential diffusion model is shown in the following formula. φ 1 (X) ~ p 1 (X)

[0128] Where t represents a time step, v t φ 1 (X) represents the predicted direction of the initial spectrum feature at t steps, φ t (X) represents the spectrum feature at t steps, and φ 0 (X) represents the initial spectrum feature (i.e., the prior distribution randomly sampled), which can be a standard normal distribution φ 1 (X) represents the predicted spectrum feature (i.e., the target distribution to be fitted), which can be the distribution of the audio mel-spectrum in the embodiments of the present specification.

[0129] Optionally, the diffusion model and the vocoder can be integrated into one network to directly complete the synthesis from the audio discrete feature and the target voiceprint feature to the audio waveform.

[0130] In an embodiment of the present specification, the reference audio feature of the reference audio corresponding to the reference text (i.e., the prompt text) can also be input into the diffusion model to realize spectrum prediction.

[0131] In summary, by performing spectrum prediction and spectrum conversion, the synthesis of the target audio is realized. Moreover, by using the ordinary differential diffusion model to synthesize the predicted spectrum feature from the audio discrete feature, the number of iteration steps is less, and the synthesis efficiency and synthesis quality are higher.

[0132] In actual application, the audio generation model can be used for audio synthesis of the to-be-processed task data. Referring to FIG. 4, FIG. 4 shows a structural schematic diagram of an audio generation model in a task processing method according to one embodiment of the present specification. As shown in FIG. 4, the audio generation model can include a semantic decoding layer 402, a spectrum prediction layer 404, and a spectrum processing layer 406. The semantic decoding layer can be the autoregressive Transformer model described above, the spectrum prediction layer can be the diffusion model described above, and the spectrum processing layer can be the vocoder described above. The specific implementation is as follows: after the to-be-processed task data and the target voiceprint feature are determined, the method further includes:

[0133] inputting the to-be-processed task data and the target voiceprint feature into the audio generation model to obtain target audio corresponding to the to-be-processed task data, wherein the audio generation model includes a semantic decoding layer, a spectrum prediction layer, and a spectrum processing layer;

[0134] The inputting the to-be-processed task data and the target voiceprint feature into the audio generation model to obtain target audio corresponding to the to-be-processed task data includes:

[0135] inputting the to-be-processed task data and the target voiceprint feature into the semantic decoding layer for semantic decoding to obtain audio discrete features;

[0136] inputting the audio discrete features and the target voiceprint feature into the spectrum prediction layer for spectrum prediction to obtain predicted spectrum features;

[0137] inputting the predicted spectrum features into the spectrum processing layer for spectrum conversion to obtain the target audio corresponding to the to-be-processed task data.

[0138] Specifically, after the to-be-processed task data and the target voiceprint feature are input into the audio generation model, the to-be-processed task data and the target voiceprint feature can be subjected to semantic decoding by the semantic decoding layer to obtain audio discrete features corresponding to the to-be-processed task data, the audio discrete features and the target voiceprint feature can be subjected to spectrum prediction by the spectrum prediction layer to obtain predicted spectrum features, and the predicted spectrum features can be subjected to spectrum conversion by the spectrum processing layer to obtain the target audio corresponding to the to-be-processed task data.

[0139] In actual application, after the semantic decoding layer outputs the audio discrete features, the audio discrete features can be input into the spectrum prediction layer after being classified by a classifier.

[0140] Further, referring to FIG. 5, FIG. 5 shows a processing flow diagram of an audio generation model in a task processing method according to one embodiment of the present specification. As shown in FIG. 5, in the scenario of timbre reproduction, the input of the audio generation model can include reference audio and reference audio discrete features of the reference audio, which can be extracted by an audio quantization model (speech tokenizer), or in the scenario of controlling timbre by instructions, the description text needs to be input, which can be extracted by a text quantization model (text tokenizer) and a text encoder to extract description text features of the description text, and the description text and the description text features are input into the audio generation model, or in the scenario of multi-language generation, the language category can also be input. That is, the necessary input of the audio generation model is the to-be-processed text, and the optional input can include the description text, the prompt text, the reference audio, and the reference audio discrete features of the reference audio. In the case that the to-be-processed text contains text corresponding to non-verbal sounds, the to-be-processed text can also be input into the text quantization model and the text encoder to obtain text features of the to-be-processed text input into the audio generation model. Specifically, the content to be input, such as the to-be-processed text and the target voiceprint features, can be input into the audio generation model, and then the semantic decoding layer 502 can be used to perform semantic decoding on the to-be-processed text and the target voiceprint features to obtain audio discrete features corresponding to the to-be-processed text, the spectral prediction layer 504 can be used to perform spectral prediction on the audio discrete features and the target voiceprint features to obtain predicted spectral features, and the spectral processing layer 506 can be used to perform spectral conversion on the predicted spectral features to obtain target audio corresponding to the to-be-processed text.

[0141] In summary, by combining the semantic decoding layer, the spectral prediction layer, and the spectral processing layer, three-stage synthesis of the audio synthesis task is realized, that is, step-by-step generation from the semantic level to the feature level and then to the audio signal level, which guarantees semantic consistency and significantly reduces the computational amount of the audio generation model, balances the computational cost and synthesis effect, and guarantees the quality, stability, and inference efficiency of audio synthesis.

[0142] In specific implementation, the inputting the to-be-processed task data and the target voiceprint features into the semantic decoding layer to perform semantic decoding to obtain audio discrete features includes:

[0143] The to-be-processed task data and the target voiceprint features are input into the semantic decoding layer, and a feature sequence is constructed for the to-be-processed task data and the target voiceprint features.

[0144] The feature sequence is subjected to semantic decoding to obtain audio discrete features.

[0145] It can be understood that the process of constructing the feature sequence and the semantic decoding by the semantic decoding layer on the to-be-processed task data and the target voiceprint feature is similar to the process of performing semantic decoding on the feature sequence to obtain the audio discrete feature, which will not be repeated here.

[0146] In an embodiment of the present specification, the input of the autoregressive Transformer model (i.e., the semantic decoding layer) can further include reference text and reference audio discrete features. The reference text can be understood as the text corresponding to the reference audio, and the reference audio discrete features can be understood as the audio discrete features corresponding to the reference audio. Then, the feature sequence can be constructed based on the target voiceprint feature, the reference text, the to-be-processed task data, and the reference audio discrete features, and the feature sequence is subjected to semantic decoding to obtain the audio discrete feature corresponding to the to-be-processed task data. Based on this, the target audio similar to the specified reference audio in timbre, prosody, emotion, etc. can be synthesized according to the reference audio without retraining or fine-tuning the audio generation model, thereby realizing zero-shot contextual learning and rapid replication of the reference audio. Optionally, the reference audio is relatively short, such as 5 to 15 seconds.

[0147] In combination with the above FIG. 3, the feature sequence 304 can include the target voiceprint feature 1, the reference text 4, the to-be-processed task data 2, the reference audio discrete features 5, and the audio discrete feature 3. Specifically, the feature sequence including the target voiceprint feature, the reference text, the to-be-processed task data, and the reference audio discrete features can be input into the semantic decoding layer, and the next audio discrete feature is predicted by the semantic decoding layer step by step until the end-of-sequence symbol of the feature sequence is detected. Then, the audio discrete feature in the feature sequence 304 is the output of the semantic decoding layer.

[0148] In another embodiment of the present specification, the input of the semantic decoding layer can further include the language category of the to-be-processed task data. Then, the feature sequence can be constructed based on the target voiceprint feature, the to-be-processed task data, and the language category, and the feature sequence is subjected to semantic decoding to obtain the audio discrete feature corresponding to the to-be-processed task data, thereby enabling the audio generation model to rapidly replicate according to the reference voice of different language categories and realizing cross-language sound reproduction.

[0149] In combination with the above FIG. 3, the feature sequence 306 can include the target voiceprint feature 1, the to-be-processed task data 2, the language category 6, and the audio discrete feature 3. Specifically, the feature sequence including the target voiceprint feature, the to-be-processed task data, and the language category can be input into the semantic decoding layer, and the next audio discrete feature is predicted by the semantic decoding layer step by step until the end-of-sequence symbol of the feature sequence is detected. Then, the audio discrete feature in the feature sequence 306 is the output of the semantic decoding layer.

[0150] In another embodiment of the present specification, the to-be-processed task data input into the semantic decoding layer also includes text corresponding to non-verbal sounds, such as laughter, coughing, breathing, etc. Based on this, the feature sequence can be constructed based on the target voiceprint feature, the to-be-processed task data including the text corresponding to the non-verbal sound, and the language category, and the semantic decoding is performed on the feature sequence to obtain the audio discrete feature corresponding to the to-be-processed task data, so that the audio generation model can synthesize natural and harmonious audio according to actual needs, so that the audio has natural personification, and fine-grained paralanguage control is realized.

[0151] In combination with the above FIG. 3, the feature sequence 308 can include the target voiceprint feature 1, the to-be-processed task data 7 including the text corresponding to the non-verbal sound, the language category 6, and the audio discrete feature 3. Specifically, the feature sequence including the target voiceprint feature, the to-be-processed task data including the text corresponding to the non-verbal sound, and the language category can be input into the semantic decoding layer, and the next audio discrete feature is gradually predicted by the semantic decoding layer until the end-of-sequence symbol of the feature sequence is detected, and then the audio discrete feature in the feature sequence 308 is taken as the output of the semantic decoding layer.

[0152] In addition, the to-be-processed task data corresponding descriptive text can also be input into the semantic decoding layer. Based on this, the feature sequence can be constructed based on the target voiceprint feature, the to-be-processed task data, and the descriptive text. It can be understood that the present specification can determine the text content input into the audio generation model according to actual needs, and the audio generation model can construct the feature sequence according to the input text content, which is not limited by the present specification.

[0153] In summary, by constructing the feature sequence according to the input text content, the semantic decoding layer performs step-by-step decoding on the feature sequence in an autoregressive manner to obtain the audio discrete feature corresponding to the to-be-processed task data, and provides accurate and natural semantic information for subsequent audio synthesis.

[0154] In actual application, the audio discrete feature and the target voiceprint feature are input into the spectrum prediction layer to perform spectrum prediction and obtain a predicted spectrum feature, including:

[0155] Through the spectrum prediction layer, the spectrum change direction of the initial spectrum feature is predicted based on the audio discrete feature and the target voiceprint feature to obtain a predicted spectrum change direction.

[0156] The initial spectrum feature is adjusted according to the predicted spectrum change direction to obtain a predicted spectrum feature.

[0157] Wherein, the spectrum change direction can be understood as the direction of the initial spectrum feature, representing the step-by-step change process from the initial spectrum feature to the predicted spectrum feature.

[0158] Specifically, the advancing direction of the initial spectrum feature is predicted by the spectrum prediction layer to obtain a predicted spectrum advancing direction, and the initial spectrum feature is adjusted according to the predicted spectrum advancing direction, so as to obtain a predicted spectrum feature.

[0159] In actual application, the spectrum prediction layer can include a Convolutional Transformer U-Net network. In the spectrum prediction layer, the advancing direction of the initial spectrum feature can be predicted by the Convolutional Transformer U-Net network, and the process is iterated until a preset iteration step number is reached according to the advancing direction to further reach a new position by a preset step size, so as to obtain the predicted spectrum feature.

[0160] It can be understood that the spectrum prediction process by the spectrum prediction layer is similar to the spectrum prediction process based on the diffusion model.

[0161] In summary, the spectrum feature of the target audio is predicted by predicting the spectrum advancing direction, which facilitates subsequent generation of the target audio.

[0162] In specific implementation, the spectrum change direction of the initial spectrum feature is predicted according to the audio discrete feature and the target voiceprint feature to obtain a predicted spectrum change direction, including:

[0163] The spectrum change direction of the initial spectrum feature at a plurality of preset step sizes is sequentially predicted according to the audio discrete feature and the target voiceprint feature until a preset iteration step number is reached to obtain a predicted spectrum change direction corresponding to a target preset step size, wherein the target preset step size is the last step in the plurality of preset step sizes.

[0164] The preset iteration step number can be understood as an iteration step number of predicting the spectrum change direction of the initial spectrum feature, such as 100 steps, 1000 steps, etc., which is not limited in the embodiments of the present application. The preset step size can be understood as a step length of the time step of the initial spectrum feature. The spectrum change direction of the initial spectrum feature at a plurality of preset step sizes is sequentially predicted, which can be understood as predicting the spectrum change direction of the initial spectrum feature at a first preset step size, the spectrum change direction of the initial spectrum feature at a second preset step size, and so on until the spectrum change direction of the initial spectrum feature at a last preset step size (i.e., the target preset step size) is predicted. It can be understood that the spectrum change direction prediction process is an iterative process.

[0165] Based on this, the initial spectral feature can be predicted in the spectral change direction of the first preset step, in the spectral change direction of the second preset step, and so on until the preset iteration step is reached, to obtain the predicted spectral change direction of the initial spectral feature in the last preset step.

[0166] For example, taking 3 preset steps as an example, the first step of the initial spectral feature can be predicted, the second step of the initial spectral feature can be predicted, and the third step of the initial spectral feature can be predicted, and so on, until the last preset step is reached, to obtain the predicted spectral change direction.

[0167] In summary, by iteratively predicting the spectral change direction, the spectrum of the target audio is predicted, and in the case of using a diffusion model based on an ordinary differential equation as the spectrum prediction layer, the model inference speed is faster and the number of iterations required when sampling the target distribution is less.

[0168] In one embodiment of the present specification, the first preset step in the plurality of preset steps is smaller than the second preset step, and the first preset step precedes the second preset step.

[0169] Specifically, since the randomness of the initial stage of spectral prediction is strong and the determinacy is poor, the preset step at this time can be small, while in the later stage of spectral prediction, as the determinacy increases and the randomness decreases, the preset step at this time can be increased. Based on this, the first preset step can be the preset step of the initial stage of spectral prediction, and the second preset step can be the preset step of the later stage of spectral prediction. For example, if 100 steps of iteration are required in the spectral prediction process, that is, there are 100 preset steps, the first 50 preset steps can be smaller than the last 50 preset steps.

[0170] In actual application, the preset step can be determined according to the cosine function, and the specific formula is as follows.

[0171] Where t is the time step.

[0172] In summary, by making the preset step of the initial stage smaller than the preset step of the later stage in the spectral prediction process, the determinacy of the spectral prediction is realized.

[0173] In addition, before the audio discrete feature and the target voiceprint feature are input into the spectrum prediction layer for spectral prediction to obtain the predicted spectral feature, the method further comprises:

[0174] Sampling the feature distribution of the audio spectrum feature, and determining the initial spectral feature according to the sampling result.

[0175] The audio spectrum feature can be understood as an audio spectrum feature of the reference audio.

[0176] In practical applications, a random number can be determined as the initial spectrum feature by randomly sampling from the prior distribution.

[0177] In summary, by determining the initial spectrum feature, a starting point for spectrum prediction is provided, thereby realizing spectrum prediction of the target audio

[0178] In another embodiment of the present specification, the inputting of the audio discrete feature and the target voiceprint feature into the spectrum prediction layer for spectrum prediction to obtain a predicted spectrum feature comprises:

[0179] The spectrum prediction layer is used to predict a spectrum change direction of the initial spectrum feature according to the audio discrete feature and the target voiceprint feature, and a first predicted spectrum change direction is obtained.

[0180] The spectrum prediction layer is used to predict a spectrum change direction of the initial spectrum feature according to the audio discrete feature and the target voiceprint feature, and a first predicted spectrum change direction is obtained.

[0181] The target predicted spectrum change direction is determined according to the first predicted spectrum change direction and the second predicted spectrum change direction.

[0182] The target predicted spectrum change direction is determined according to the first predicted spectrum change direction and the second predicted spectrum change direction.

[0183] The additional guidance condition can be understood as a guidance condition for inputting the prediction spectrum layer. This guidance condition can be used to adjust the processing process of the prediction spectrum layer, so that the spectrum prediction layer can focus more on meeting the condition. In practical applications, noise can be mixed in the input condition of the prediction spectrum layer, so that the spectrum prediction layer can generate data consistent with the condition without explicit guidance. The first predicted spectrum change direction can be understood as the predicted spectrum change direction predicted by the spectrum prediction layer without inputting the additional guidance condition. The second predicted spectrum change direction can be understood as the predicted spectrum change direction predicted by the spectrum prediction layer with inputting the additional guidance condition.

[0184] Based on this, the spectrum change direction of the initial spectrum feature can be predicted according to the audio discrete feature and the target voiceprint feature through the spectrum prediction layer without inputting additional guidance conditions to obtain a first predicted spectrum change direction, and the spectrum change direction of the initial spectrum feature can be predicted according to the additional guidance conditions, the audio discrete feature and the target voiceprint feature through the spectrum prediction layer with the input of the additional guidance conditions to obtain a second predicted spectrum change direction. The difference between the first predicted spectrum change direction and the second predicted spectrum change direction is determined as a target predicted spectrum change direction, and the initial spectrum feature is adjusted according to the target predicted spectrum change direction to obtain a predicted spectrum feature.

[0185] In practical applications, the introduction of additional guidance conditions can be implemented through Classifier-free guidance, and the specific process is shown in the following formula.

[0186] Wherein, β represents the certainty strength, and the value in practical applications can be 0.6 to 0.9, and ψ represents the additional guidance condition. The sampling process of the Classifier-free guidance can be understood as calculating the network output with the additional guidance condition and the network output without the additional guidance condition at each time step respectively, and then linearly combining the two network outputs (i.e. the first predicted spectrum change direction and the second predicted spectrum change direction) according to the certainty strength, so as to obtain a more accurate target predicted spectrum change direction.

[0187] In summary, by introducing the additional guidance condition, the randomness of spectrum prediction can be reduced, thereby improving the quality of audio synthesis.

[0188] In practical applications, after determining the target audio corresponding to the to-be-processed task data, the method further includes:

[0189] The target audio is sent to an audio playback device to enable the audio playback device to play the target audio.

[0190] Wherein, the audio playback device can be understood as a device in communication connection with the server, and the audio playback device includes but is not limited to a sound equipment, a playback device integrated with a client, etc.

[0191] In practical applications, after synthesizing the target audio corresponding to the to-be-processed task data, the target audio can be sent to the client, and the target audio can be played through the audio playback device integrated with the client.

[0192] In summary, by sending the target audio to the audio playback device, the playing of the target audio is realized.

[0193] In actual application, before the input of the to-be-processed task data and the target voiceprint feature into the audio generation model, the method further includes the following steps.

[0194] determining a machine learning model to be trained and training data associated with an audio generation task, wherein the machine learning model to be trained includes a semantic decoding layer, a spectrum prediction layer, and a spectrum processing layer, and the training data includes sample text, sample audio, sample voiceprint features corresponding to the sample audio, and sample audio discrete features.

[0195] processing the sample text, the sample audio, the sample voiceprint features, and the sample audio discrete features according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result.

[0196] adjusting model parameters of the machine learning model according to the audio generation result to obtain a trained audio generation model.

[0197] Specifically, the training process of the audio generation model is consistent with the training process of the audio generation model training method provided in the embodiments of the present specification, and can be referred to the audio generation model training method, which will not be repeated here.

[0198] In summary, one embodiment of the present specification provides a task processing method. After determining to-be-processed task data and target voiceprint features, the to-be-processed task data is subjected to semantic decoding to obtain audio discrete features corresponding to the to-be-processed task data. The audio discrete features, as a series of discrete integer representations, can represent semantic information of the to-be-processed task data, so that a target audio determined according to the audio discrete features and the target voiceprint features can have semantic consistency, have a corresponding relationship between the audio and the text, ensure the stability and accuracy of the audio synthesis effect, and also enable the target audio to be played in a tone corresponding to the target voiceprint feature in subsequent playing.

[0199] Referring to FIG. 6, FIG. 6 shows a flowchart of an audio generation model training method according to one embodiment of the present specification, which specifically includes the following steps.

[0200] Step 602: determining a machine learning model to be trained and training data associated with an audio generation task, wherein the machine learning model to be trained includes a semantic decoding layer, a spectrum prediction layer, and a spectrum processing layer, and the training data includes sample text, sample audio, sample voiceprint features corresponding to the sample audio, and sample audio discrete features.

[0201] The machine learning model to be trained can be understood as a blank machine learning model that has not been trained and can include, but is not limited to, a neural network model, a deep learning model, a reinforcement learning model, and a generative model, etc. The semantic decoding layer, the spectrum prediction layer, and the spectrum processing layer are similar to the aforementioned semantic decoding layer, the spectrum prediction layer, and the spectrum processing layer.

[0202] The sample text can be understood as a text as a training sample and can include texts of various language categories, such as Chinese texts, English texts, Japanese texts, etc. The sample audio can be audio corresponding to the sample text or can be any piece of audio, such as audio of a speaker speaking. The sample audio discrete feature can be understood as an audio discrete feature of the sample audio.

[0203] In actual applications, the sample audio discrete feature can be obtained by processing the sample audio through an audio quantization model. The audio quantization model can include a speech recognition coding layer and a vector quantizer. The speech recognition coding layer includes a position encoder and a first encoder. Specifically, the sample audio can be input into the audio quantization model, the position encoder can be used to perform position coding on the sample audio, and the first encoder can be used to perform coding to obtain a hidden layer representation of the sample audio. The hidden layer representation can be input into the vector quantizer, and the vector quantizer can be used to quantize the hidden layer representation to obtain the sample audio discrete feature of the sample audio. Since the audio quantization model is obtained based on supervised training, the correspondence between the sample audio discrete feature and the text is more explicit, the synthesis effect is more stable, and the quality requirement for the training data is lower, thereby reducing the training cost of the audio generation model.

[0204] The extraction process of the sample audio discrete feature is shown in the following formula. H = Encoder1 (PosEnc (X))

[0205] Wherein, X is an audio spectrum feature (in actual applications, it is a mel spectrum feature) of the sample audio, PosEnc is used to represent position coding, Encoder1 represents the aforementioned first encoder (in actual applications, it can include 6 layers of coding layers), H is used to represent a hidden layer representation of the sample audio, h l is used to represent a hidden layer representation corresponding to the lth frame of speech, VQ is used to represent the vector quantizer, C is used to represent a codebook in the vector quantizer, containing N code words, c n is used to represent the nth code word vector in the codebook of the vector quantizer, μ l is used to represent an audio discrete feature (i.e., a speech token) obtained after quantization of the lth frame of speech.

[0206] The update process of a code word in the codebook of the vector quantizer is shown in the following formula.

[0207] wherein a represents the update of the sparse parameter, and the value in actual application can be 0.9 to 0.99. Specifically, the code word is used to update the hidden layer representation h assigned to the code word. l The linear combination is performed to update.

[0208] Step 604: processing the sample text, the sample audio, the sample voiceprint feature, and the sample audio discrete feature according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result.

[0209] In specific implementation, before the processing the sample text, the sample audio, the sample voiceprint feature, and the sample audio discrete feature according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result, the method further includes:

[0210] extracting a feature of the sample text to obtain a sample text feature;

[0211] constructing a sample feature sequence according to a preset sequence start symbol and a preset sequence end symbol, the sample voiceprint feature, the sample text feature, and the sample audio discrete feature.

[0212] The preset sequence start symbol is used to represent the start of the feature sequence, which can be represented by sos, and the preset sequence end symbol is used to represent the end of the feature sequence, which can be represented by eos. The sample text feature can be understood as a hidden layer representation of the sample text. The sample text can include a description text and a text to be synthesized.

[0213] Specifically, the sample text can be input into a text quantization model (text tokenizer) to obtain a text discrete feature corresponding to the sample text, and the sample text feature can be obtained through a text encoder and a text decoder. The preset sequence start symbol, the sample voiceprint feature, the sample text feature, the sample audio discrete feature, and the preset sequence end symbol are sequentially constructed into a sample feature sequence.

[0214] In actual application, the process of constructing the sample feature sequence (i.e., the LM sequence) is shown in the following formula.

[0215] wherein, sos represents the preset sequence start symbol, eos represents the preset sequence end symbol, V represents the sample voiceprint feature, sos represents the preset sequence start symbol, sos represents the preset sequence start symbol, l sos represents the preset sequence start symbol,

[0216] Specifically, in the training process of the semantic decoding layer, the input of the semantic decoding layer is part of the above formula except the preset sequence end symbol, and the output is part of the above formula except the preset sequence start symbol, wherein the first model loss value is only calculated μ l and Part.

[0217] The processing process of the sample text is shown in the following formula.

[0218] Wherein, BPE is a word segmentation model, TextEncoder is a text encoder, Y is a sample text, is a sample text feature (i.e. text hidden layer representation).

[0219] In summary, by constructing a sample feature sequence, the subsequent training of the semantic decoding layer is facilitated.

[0220] In specific implementation, before the processing of the sample text, the sample audio, the sample voiceprint feature and the sample audio discrete feature according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result, further comprising:

[0221] extracting features from the sample audio to obtain sample audio features;

[0222] masking the sample audio features to obtain masked sample audio features.

[0223] Wherein, the masking process can be understood as setting the features to 0.

[0224] Specifically, in order to enable the audio generation model to obtain more additional information during training, audio features can be extracted from the sample audio to obtain sample audio features of the sample audio, and the sample audio features can be masked to obtain masked sample audio features.

[0225] In specific implementation, the sample audio features can be randomly masked, such as randomly masking multiple frames of features contained in the sample audio features, thereby obtaining the masked sample audio features.

[0226] In an embodiment of the present specification, when the sample audio features are masked, the first random number of frames of features of the sample audio features can be used as additional information, and all frames of features after the sample audio features are set to 0, i.e. all frames of features after the sample audio features are masked. For example, the sample audio features include 10 frames of features, and the last 4 frames of features can be set to 0, so that the masked sample audio features are the first 6 frames of features in the original sample audio features.

[0227] In summary, by performing masking processing on the sample audio features, the masked sample audio features are used as additional information for model training, so that the audio generation model can learn more additional information during the training process, thereby ensuring the processing performance of the audio generation model.

[0228] In practical applications, the training data further includes sample initial spectrum features and a sample stride;

[0229] The processing of the sample text, the sample audio, the sample voiceprint features, and the sample audio discrete features according to the semantic decoding layer and the spectrum prediction layer obtains an audio generation result, including:

[0230] According to the semantic decoding layer, autoregressive semantic decoding is performed on the sample feature sequence to obtain predicted audio discrete features;

[0231] According to the spectrum prediction layer, spectrum prediction is performed on the sample audio discrete features, the sample voiceprint features, the masked sample audio features, the sample initial spectrum features, and the sample stride to obtain sample predicted spectrum features.

[0232] The sample initial spectrum features can be understood as initial spectrum features of a training sample, and the acquisition manner of the sample initial spectrum features is similar to the acquisition manner of the initial spectrum features, which will not be repeated here. The sample stride can be understood as the step length of the time step of the initial spectrum features.

[0233] Specifically, the sample feature sequence can be input into the semantic decoding layer for autoregressive semantic decoding to obtain predicted audio discrete features. The sample discrete audio features, the sample voiceprint features, the masked sample audio features, the sample initial spectrum features, and the sample stride can be input into the spectrum prediction layer for spectrum prediction to predict the travel direction of the current sample stride in the transport vector field (i.e., the sample predicted spectrum change direction), thereby obtaining the sample predicted spectrum features.

[0234] In practical applications, the feature sequence input into the semantic decoding layer is a left-shift sequence, and the semantic decoding layer performs step-by-step semantic decoding on the feature sequence. The feature sequence output by the semantic decoding layer is a right-shift sequence. The left-shift sequence can include other parts except the preset sequence end symbol, and the right-shift sequence can include other parts except the preset sequence start symbol. It can be understood that when calculating the model loss value subsequently, the loss is calculated for the predicted audio discrete features and the preset sequence end symbol part in the right-shift sequence, and the loss is not calculated for other parts.

[0235] In summary, the training data is processed by the semantic decoding layer and the spectrum prediction layer, which facilitates the subsequent model training process.

[0236] Step 606: model parameter adjustment is performed on the machine learning model according to the audio generation result, and a trained audio generation model is obtained.

[0237] In a specific implementation, the model parameter adjustment performed on the machine learning model according to the audio generation result, and the trained audio generation model obtained, include:

[0238] The model parameter adjustment is performed on the semantic decoding layer according to the predicted audio discrete feature and the label audio discrete feature corresponding to the sample audio;

[0239] The model parameter adjustment is performed on the spectrum prediction layer according to the sample predicted spectrum feature and the label spectrum feature corresponding to the sample audio, and a trained audio generation model is obtained.

[0240] The label audio discrete feature corresponding to the sample audio can be understood as the sample audio discrete feature input into the semantic decoding layer. In actual applications, the training can be performed based on a Teacher forcing manner.

[0241] Specifically, a first model loss value can be calculated according to the predicted audio discrete feature and the label audio discrete feature corresponding to the sample audio, and the model parameter adjustment is performed on the semantic decoding layer according to the first model loss value. A second model loss value can be calculated according to the sample predicted spectrum feature and the label spectrum feature corresponding to the sample audio, and the model parameter adjustment is performed on the spectrum prediction layer according to the second model loss value, and a trained audio generation model is obtained.

[0242] The formula for calculating the first model loss value is as follows.

[0243] wherein, is the first model loss value, L represents the total length of the sample feature sequence of the sample audio, and +1 represents a preset sequence end symbol that needs to be predicted.

[0244] The formula for calculating the second model loss value is as follows.

[0245] wherein, is the second model loss value, is the sample predicted spectrum change direction, is the real spectrum change direction corresponding to the sample predicted spectrum change direction, is the expectation of the L1 loss, and the expectation can be estimated by randomly sampling from multiple different time steps, calculating the L1 loss respectively, and then averaging.

[0246] The calculation method of the mel spectrum at the time step t and the calculation method of the real spectrum change direction are shown in the following formula.

[0247] wherein, X0 represents a random number sampled from a prior distribution (i.e., a sample initial spectrum feature), and X1 represents a mel spectrum feature extracted from the sample speech, X1 represents the real spectrum change direction, and σ represents a parameter set for numerical stability, which can be 0.001 in actual application.

[0248] The prediction method of the sample predicted spectrum change direction is shown in the following formula.

[0249] wherein, NN represents a neural network, θ represents a parameter of the neural network, t represents a time step, v represents a sample voiceprint feature, {μ l} 1:L X0 represents a sample audio discrete feature, X1 represents a masked target speech mel spectrum (i.e., a sample predicted spectrum feature).

[0250] In actual application, the label spectrum feature corresponding to the sample audio can be an interpolation of the audio mel spectrum of the sample audio and the mel spectrum obtained by linear combination of the audio mel spectrum at a certain time step t. The input spectrum prediction layer also includes the linear combination of the audio mel spectrum at a certain time step t.

[0251] In addition, when training the spectrum prediction layer, the spectrum prediction layer can be trained according to the predicted audio discrete feature output by the semantic decoding layer, the sample voiceprint feature, the masked sample audio feature, the sample initial spectrum feature and the sample step, and the specific training process is similar to the aforementioned training process according to the sample audio discrete feature, the sample voiceprint feature, the masked sample audio feature, the sample initial spectrum feature and the sample step, which will not be repeated here.

[0252] Further, after obtaining the trained audio generation model, the audio generation model can be fine-tuned according to the audio of a specific person speaking, so that the audio synthesized by the audio generation model can have the speaking style, timbre and habits of the specific person.

[0253] In summary, through the model training process of the above-mentioned audio generation model, the training of the audio generation model is realized, so that the audio generation model has good audio synthesis capability and small training cost, ensuring the stability and accuracy of the audio synthesis result.

[0254] It can be understood that in actual application, taking a novel application program as an example for illustration, the novel application program provides a novel reading service, and then the novel application program side can train an audio generation model according to the audio generation model training method, and provide the novel reading service for users according to the audio generation model, or the novel application program side can send a model training request to a model training platform, the model training platform can receive the model training request, train an audio generation model according to the audio generation model training method according to the actual demand of the novel application program side and the input training data, and provide the audio generation model to the novel application program, which can directly integrate and use the audio generation model, or the model training platform can provide a calling interface of the audio generation model to the novel application program, and the novel application program can call the audio generation model through the calling interface.

[0255] The audio generation model is further described below by taking the application of the audio generation model training method provided in the specification to model training as an example in combination with FIG. 7 and FIG. 8. FIG. 7 shows a training process diagram for a semantic decoding layer in an audio generation model training method provided in an embodiment of the specification, and FIG. 8 shows a training process diagram for a spectrum prediction layer in an audio generation model training method provided in an embodiment of the specification.

[0256] As shown in FIG. 7, the constructed sample feature sequence can be input into the semantic decoding layer, and autoregressive semantic decoding is performed by the semantic decoding layer to output a predicted feature sequence, which can include a predicted audio discrete feature. The semantic decoding layer is adjusted in model parameters according to the predicted audio discrete feature and the sample audio discrete feature. The sample text feature can be obtained by inputting the sample text into a text encoder, and the sample audio discrete feature can be obtained by inputting the sample audio into an audio quantization model. S is a preset sequence start symbol, E is a preset sequence end symbol, and T is a start symbol of the audio discrete feature. In addition, the predicted audio discrete feature can be input into the spectrum prediction layer after being classified by a classifier to obtain a sample predicted spectrum feature, so as to complete the training of the subsequent spectrum prediction layer.

[0257] As shown in FIG. 8, when training the spectrum prediction layer, the sample audio discrete feature μ, the sample voiceprint feature V, the masked sample audio feature X~t, the sample initial spectrum feature Xt, and the sample step t are input into the spectrum prediction layer as additional information to obtain a sample predicted spectrum feature through a plurality of (N) ResNet1D (one-dimensional residual network) and a Transformer Block (sequence data processing network).

[0258] The text processing method is further described below in combination with FIG. 9, taking the application of the task processing method provided in the specification to novel reading as an example. FIG. 9 shows a process flowchart of a task processing method provided in an embodiment of the specification, specifically including the following steps.

[0259] Step 902: receiving a novel reading instruction sent by a client, and determining a novel text corresponding to the novel reading instruction.

[0260] Specifically, a novel application program can be installed on the client, and a user can enter the novel application program on the client to browse multiple novels in the novel application program. The client generates a novel reading instruction according to a selection operation of the user on one of the novels and sends the novel reading instruction to the server. The server determines a novel text selected by the user according to the novel reading instruction.

[0261] Step 904: determining a target voiceprint feature.

[0262] Specifically, the novel application program can provide multiple candidate tones for the user to select one as a target tone, and the server can extract a target voiceprint feature of the target tone. Alternatively, only one tone is set in the novel application program, in which case the user does not need to select, and the target voiceprint feature of the tone can be directly determined.

[0263] Step 906: inputting the novel text and the target voiceprint feature into an audio generation model to obtain a target audio corresponding to the novel text.

[0264] Specifically, the novel text and the target voiceprint feature can be input into the audio generation model. In the audio generation model, the novel text is semantically decoded by a semantic decoding layer to obtain audio discrete features, the target voiceprint feature and the audio discrete features are spectrally predicted by a spectrum prediction layer to obtain predicted spectrum features, and the predicted spectrum features are spectrally converted by a spectrum processing layer to obtain the target audio corresponding to the novel text.

[0265] Step 908: sending the target audio to the client and playing the target audio through the client.

[0266] Specifically, the audio generation model is used to synthesize the audio of the novel text, thereby realizing the novel reading service.

[0267] Referring to FIG. 10, FIG. 10 shows a flowchart of a text processing method according to an embodiment of the specification, specifically including the following steps.

[0268] Step 1002: receiving a text reading instruction for a target text sent by a client, and determining a target voiceprint feature.

[0269] Step 1004: performing semantic decoding on the target text to obtain audio discrete features;

[0270] Step 1006: determining target audio corresponding to the target text according to the audio discrete features and the target voiceprint features;

[0271] Step 1008: sending the target audio to the client to enable the client to play the target audio.

[0272] The text reading instruction can be understood as an instruction to read the target text. Reading the target text through the client can be understood as playing out the audio corresponding to the target text through the client. In the field of audiobooks, the target text can be a novel, an article, a news report, or the like. The target voiceprint features can be the voiceprint features of the audio input by the user, or the voiceprint features of the target voice selected by the user from multiple candidate voices.

[0273] Specifically, the user can select a target text that the user wants to listen to through the client. The client sends a text reading instruction for the target text to the server. The server determines target voiceprint features and performs semantic decoding on the target text to obtain audio discrete features. The target audio corresponding to the target text is determined according to the target voiceprint features and the audio discrete features. The target audio is sent to the client, and the client can play the target audio.

[0274] It can be understood that the process of processing the target text to obtain the target audio in the embodiments of the present specification is similar to the process of processing the task data to obtain the target audio described above, that is, the process of processing the target text can also be implemented through the audio generation model described above. The embodiments of the present specification will not be repeated here.

[0275] In an optional embodiment, after receiving the text reading instruction for the target text sent by the client, the method further includes:

[0276] calling a text processing model, inputting the target text into the text processing model to obtain an abstract text corresponding to the target text;

[0277] performing semantic decoding on the abstract text to obtain audio discrete features;

[0278] determining abstract audio corresponding to the abstract text according to the audio discrete features and the target voiceprint features;

[0279] sending the abstract audio to the client to enable the client to play the abstract audio.

[0280] The summary text corresponding to the target text can be understood as a summary of the text content of the target text. For example, the target text is a novel, and the summary text corresponding to the target text can briefly describe what the novel is about. The text processing model can be a model for summarizing input text, such as a neural network model, a deep learning model, or a large model, which is not limited in the embodiments of the present specification.

[0281] Specifically, after receiving the text reading instruction for the target text sent by the client, the text processing model can be called to summarize the target text, obtain the summary text corresponding to the target text, and perform semantic decoding on the summary text to obtain audio discrete features. According to the audio discrete features and the target voiceprint features, the summary audio corresponding to the summary text is determined, and the summary audio is sent to the client to play the summary audio through the client.

[0282] It can be understood that the process of processing the summary text to obtain the summary audio in the embodiments of the present specification is similar to the aforementioned process of processing the task data to obtain the target audio, that is, the process of processing the summary text can also be implemented through the audio generation model described above, which is not repeated in the embodiments of the present specification.

[0283] In summary, by generating a summary text for a target text first, and synthesizing the summary text into a summary audio, the user can understand the general content of the target text when playing the summary audio, thereby facilitating the user to choose whether to continue reading the target text, improving the user experience, and reducing the computational consumption of the server.

[0284] It can be understood that the summary generation service can be started or stopped according to the user's settings.

[0285] In an optional embodiment, after the summary audio is sent to the client, the method further includes:

[0286] In the case where the confirmation instruction sent by the client is received, the step of performing semantic decoding on the target text to obtain audio discrete features is executed.

[0287] The confirmation instruction can be understood as a confirmation reading instruction of the target text received by the client from the user during the playing of the summary audio or after the playing of the summary audio is completed.

[0288] Specifically, after receiving the confirmation instruction sent by the client, it indicates that the user wants to continue reading the target text, and at this time the server can continue to execute the steps of processing the target text to obtain the target audio corresponding to the target text, and sending the target audio to the client to play the target audio through the client.

[0289] Then, in the case where the confirmation instruction sent by the client is not received, it is indicated that the user may not be interested in the target text after listening to the summary audio and does not select to continue reading, and at this time, the subsequent processing process of performing semantic decoding on the target text can not be performed, so as to save the computing consumption of the server.

[0290] One embodiment of the present specification provides a text processing method, after determining the target text and the target voiceprint feature, performing semantic decoding on the to-be-processed task data to obtain audio discrete features corresponding to the target text. The audio discrete features are a series of discrete integer representations and can represent the semantic information of the target text. The target audio determined according to the audio discrete features and the target voiceprint feature can have semantic consistency, have a corresponding relationship between the audio and the text, ensure the stability and accuracy of the audio synthesis effect, and enable the target audio to be played in the target voiceprint feature corresponding tone in subsequent playing.

[0291] Referring to FIG. 11, FIG. 11 shows a flowchart of another task processing method provided by one embodiment of the present specification, which specifically includes the following steps.

[0292] Step 1102: Determine the target voiceprint feature and the conference task data corresponding to the target conference;

[0293] Step 1104: Perform semantic decoding on the conference task data to obtain audio discrete features;

[0294] Step 1106: Determine the target conference data corresponding to the conference task data according to the audio discrete features and the target voiceprint feature;

[0295] Step 1108: In response to a playing instruction of a client, play the target conference data according to a preset playing rule.

[0296] The conference task data corresponding to the target conference can be understood as a summary text summarizing the conference content of the target conference. The conference task data can include specific content discussed in the target conference, next step plans, and the like. The target conference data can include the target audio corresponding to the conference task data, and can also include other text content. The playing instruction can be understood as a trigger instruction for playing the target audio. Before receiving the playing instruction, the target audio is not played. The preset playing rule can be understood as a rule for playing the target audio. For example, the preset playing rule can be to play the complete target audio, or the preset playing rule can also be to divide the target audio into multiple audio paragraphs according to the content of the conference task data, and play the multiple audio paragraphs in a preset time interval, such as dividing the target audio into 3 audio paragraphs, playing 1 audio paragraph every 5 minutes. The participants can record and supplement the conference content in the preset time interval of pausing the playing. A person skilled in the art can set the preset playing rule according to actual needs, and the embodiments of the present specification do not limit this.

[0297] Specifically, the target voiceprint feature and the conference task data corresponding to the target conference can be determined, the conference task data is semantically decoded to obtain an audio discrete feature, the target audio corresponding to the conference task data is determined according to the audio discrete feature and the target voiceprint feature, the target audio is temporarily stored in the cache area, and the target audio is read from the cache area in response to the audio playing instruction, and the target audio is played according to the preset playing rule.

[0298] It can be understood that the process of processing the conference task data to obtain the target audio in the embodiments of the present specification is similar to the aforementioned process of processing the to-be-processed task data to obtain the target audio, that is, the process of processing the conference task data can also be implemented through the audio generation model described above, and the embodiments of the present specification do not repeat here.

[0299] One embodiment of the present specification provides a task processing method. After determining the conference task data and the target voiceprint feature, the conference task data is semantically decoded to obtain an audio discrete feature corresponding to the conference task data. The audio discrete feature, as a series of discrete integer representations, can represent the semantic information of the conference task data, so that the target audio determined according to the audio discrete feature and the target voiceprint feature can have semantic consistency, have a corresponding relationship between the audio and the text, ensure the stability and accuracy of the audio synthesis effect, and also enable the target audio to be played in the tone corresponding to the target voiceprint feature in subsequent playing.

[0300] Corresponding to the method embodiments described above, the specification also provides task processing device embodiments. FIG. 12 shows a structural schematic diagram of a task processing device according to an embodiment of the specification. As shown in FIG. 12, the device includes:

[0301] A first determination module 1202 configured to determine task data to be processed and target voiceprint features;

[0302] A decoding module 1204 configured to perform semantic decoding on the task data to be processed to obtain audio discrete features;

[0303] A second determination module 1206 configured to determine target audio corresponding to the task data to be processed according to the audio discrete features and the target voiceprint features.

[0304] In an optional embodiment, the second determination module 1206 is further configured to:

[0305] perform spectral prediction on the audio discrete features and the target voiceprint features to obtain predicted spectral features;

[0306] perform spectral conversion on the predicted spectral features to obtain the target audio corresponding to the task data to be processed.

[0307] The task data to be processed includes text of multiple language categories;

[0308] In an optional embodiment, the decoding module 1204 is further configured to:

[0309] construct a feature sequence from the task data to be processed, the multiple language categories, and the target voiceprint features;

[0310] perform semantic decoding on the feature sequence to obtain audio discrete features.

[0311] In an optional embodiment, the device further includes an input module configured to:

[0312] input the task data to be processed and the target voiceprint features into an audio generation model to obtain target audio corresponding to the task data to be processed, wherein the audio generation model includes a semantic decoding layer, a spectral prediction layer, and a spectral processing layer;

[0313] The inputting of the task data to be processed and the target voiceprint features into the audio generation model to obtain the target audio corresponding to the task data to be processed includes:

[0314] inputting the task data to be processed and the target voiceprint features into the semantic decoding layer to perform semantic decoding and obtain audio discrete features;

[0315] input the audio discrete feature and the target voiceprint feature into the spectrum prediction layer for spectrum prediction to obtain a predicted spectrum feature;

[0316] input the predicted spectrum feature into the spectrum processing layer for spectrum conversion to obtain a target audio corresponding to the to-be-processed task data.

[0317] In an optional embodiment, the input module is further configured to:

[0318] input the to-be-processed task data and the target voiceprint feature into the semantic decoding layer to construct a feature sequence for the to-be-processed task data and the target voiceprint feature;

[0319] perform semantic decoding on the feature sequence to obtain an audio discrete feature.

[0320] In an optional embodiment, the input module is further configured to:

[0321] predict, by the spectrum prediction layer, a spectrum change direction of an initial spectrum feature according to the audio discrete feature and the target voiceprint feature to obtain a predicted spectrum change direction;

[0322] adjust the initial spectrum feature according to the predicted spectrum change direction to obtain a predicted spectrum feature.

[0323] In an optional embodiment, the input module is further configured to:

[0324] sequentially predict spectrum change directions of the initial spectrum feature at a plurality of preset steps according to the audio discrete feature and the target voiceprint feature until a preset iteration step number is reached to obtain a predicted spectrum change direction corresponding to a target preset step, wherein the target preset step is the last step in the plurality of preset steps.

[0325] In an optional embodiment, a first preset step in the plurality of preset steps is smaller than a second preset step, and the first preset step is before the second preset step.

[0326] In an optional embodiment, the input module is further configured to:

[0327] sample a feature distribution of an audio spectrum feature, and determine an initial spectrum feature according to a sampling result.

[0328] In an optional embodiment, the input module is further configured to:

[0329] The spectrum prediction layer is configured to predict a spectrum change direction of the initial spectrum feature according to the audio discrete feature and the target voiceprint feature, and obtain a first predicted spectrum change direction.

[0330] The spectrum prediction layer is configured to predict a spectrum change direction of the initial spectrum feature according to the audio discrete feature and the target voiceprint feature, and obtain a first predicted spectrum change direction.

[0331] The spectrum prediction layer is configured to predict a spectrum change direction of the initial spectrum feature according to the audio discrete feature and the target voiceprint feature, and obtain a first predicted spectrum change direction.

[0332] The spectrum prediction layer is configured to predict a spectrum change direction of the initial spectrum feature according to the audio discrete feature and the target voiceprint feature, and obtain a first predicted spectrum change direction.

[0333] In an optional embodiment, the first determining module 1202 is further configured to:

[0334] receive the to-be-processed task data and the reference audio sent by the client;

[0335] extract a voiceprint feature of the reference audio to obtain a target voiceprint feature corresponding to the reference audio; or

[0336] receive the to-be-processed task data and the audio selection instruction sent by the client;

[0337] determine a reference audio according to the audio selection instruction, extract a voiceprint feature of the reference audio to obtain a target voiceprint feature corresponding to the reference audio.

[0338] In an optional embodiment, the apparatus further includes a training module, which is further configured to:

[0339] determine a machine learning model to be trained and training data associated with an audio generation task, wherein the machine learning model to be trained includes a semantic decoding layer, a spectrum prediction layer, and a spectrum processing layer, and the training data includes sample text, sample audio, a sample voiceprint feature corresponding to the sample audio, and a sample audio discrete feature;

[0340] process the sample text, the sample audio, the sample voiceprint feature, and the sample audio discrete feature according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result;

[0341] adjust model parameters of the machine learning model according to the audio generation result to obtain a trained audio generation model.

[0342] In an optional embodiment, the apparatus further includes a sending module, which is further configured to:

[0343] send the target audio to an audio playing device, so that the audio playing device plays the target audio.

[0344] One embodiment of the present specification provides a task processing apparatus. After determining to-be-processed task data and a target voiceprint feature, the to-be-processed task data is subjected to semantic decoding to obtain an audio discrete feature corresponding to the to-be-processed task data. The audio discrete feature, as a series of discrete integer representations, can represent semantic information of the to-be-processed task data. The target audio determined according to the audio discrete feature and the target voiceprint feature can have semantic consistency, have a corresponding relationship between audio and text, ensure stability and accuracy of an audio synthesis effect, and enable the target audio to be played in a tone corresponding to the target voiceprint feature when played subsequently.

[0345] The above is a schematic scheme of the task processing apparatus of the present embodiment. It should be noted that the technical scheme of the task processing apparatus belongs to the same concept as the technical scheme of the task processing method described above. Details of the technical scheme of the task processing apparatus that are not described in detail can be referred to the description of the technical scheme of the task processing method.

[0346] Corresponding to the method embodiments described above, the present specification also provides an audio generation model training apparatus embodiment. FIG. 13 shows a structural schematic diagram of an audio generation model training apparatus according to one embodiment of the present specification. As shown in FIG. 13, the apparatus includes:

[0347] A determination module 1302 is configured to determine a machine learning model to be trained and training data associated with an audio generation task. The machine learning model to be trained includes a semantic decoding layer, a spectrum prediction layer, and a spectrum processing layer. The training data includes sample text, sample audio, a sample voiceprint feature corresponding to the sample audio, and a sample audio discrete feature.

[0348] A processing module 1304 is configured to process the sample text, the sample audio, the sample voiceprint feature, and the sample audio discrete feature according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result.

[0349] An adjustment module 1306 is configured to adjust model parameters of the machine learning model according to the audio generation result to obtain a trained audio generation model.

[0350] In an optional embodiment, the processing module 1304 is further configured to:

[0351] extract features from the sample text to obtain sample text features;

[0352] According to the preset sequence start symbol and the preset sequence end symbol, the sample feature sequence is constructed from the sample voiceprint feature, the sample text feature and the sample audio discrete feature.

[0353] In an optional embodiment, the processing module 1304 is further configured to:

[0354] The sample audio is subjected to feature extraction to obtain a sample audio feature;

[0355] The sample audio feature is subjected to masking processing to obtain a masked sample audio feature.

[0356] In an optional embodiment, the training data further includes a sample initial spectrum feature and a sample stride;

[0357] The processing module 1304 is further configured to:

[0358] According to the semantic decoding layer, the sample feature sequence is subjected to autoregressive semantic decoding to obtain a predicted audio discrete feature;

[0359] According to the spectrum prediction layer, the sample audio discrete feature, the sample voiceprint feature, the masked sample audio feature, the sample initial spectrum feature and the sample stride are subjected to spectrum prediction to obtain a sample predicted spectrum feature.

[0360] In an optional embodiment, the adjusting module 1306 is further configured to:

[0361] According to the predicted audio discrete feature and a label audio discrete feature corresponding to the sample audio, the model parameter of the semantic decoding layer is adjusted;

[0362] According to the sample predicted spectrum feature and a label spectrum feature corresponding to the sample audio, the model parameter of the spectrum prediction layer is adjusted to obtain a trained audio generation model.

[0363] The above is a schematic scheme of an audio generation model training device according to the present embodiment. It should be noted that the technical scheme of the audio generation model training device belongs to the same concept as the technical scheme of the audio generation model training method described above. The technical scheme of the audio generation model training device, which is not described in detail, can be referred to the description of the technical scheme of the audio generation model training method.

[0364] Corresponding to the method embodiments described above, the present specification also provides text processing device embodiments. FIG. 14 shows a structural schematic diagram of a text processing device according to an embodiment of the present specification. As shown in FIG. 14, the device includes:

[0365] The receiving module 1402 is configured to receive the text reading instruction for the target text sent by the client, and determine a target voiceprint feature;

[0366] The decoding module 1404 is configured to perform semantic decoding on the target text to obtain an audio discrete feature;

[0367] The determining module 1406 is configured to determine a target audio corresponding to the target text according to the audio discrete feature and the target voiceprint feature.

[0368] The sending module 1408 is configured to send the target audio to the client to enable the client to play the target audio.

[0369] In an optional embodiment, the apparatus further comprises a processing module configured to:

[0370] invoke a text processing model, input the target text into the text processing model to obtain an abstract text corresponding to the target text;

[0371] perform semantic decoding on the abstract text to obtain an audio discrete feature;

[0372] determine an abstract audio corresponding to the abstract text according to the audio discrete feature and the target voiceprint feature;

[0373] send the abstract audio to the client to enable the client to play the abstract audio.

[0374] In an optional embodiment, the receiving module 1402 is further configured to:

[0375] in a case where the confirmation instruction sent by the client is received, perform the step of performing semantic decoding on the target text to obtain an audio discrete feature.

[0376] An embodiment of the present specification provides a text processing apparatus, after determining a target text and a target voiceprint feature, performing semantic decoding on to-be-processed task data to obtain an audio discrete feature corresponding to the target text. The audio discrete feature, as a series of discrete integer representations, can represent semantic information of the target text, so that a target audio determined according to the audio discrete feature and the target voiceprint feature can have semantic consistency, have a corresponding relationship between the audio and the text, guarantee the stability and accuracy of the audio synthesis effect, and enable the target audio to be played in a tone corresponding to the target voiceprint feature in subsequent playing.

[0377] The above is a schematic scheme of the text processing device of the embodiment. It should be noted that the technical scheme of the text processing device belongs to the same concept as the technical scheme of the text processing method described above. Details of the technical scheme of the text processing device that are not described in detail can be referred to the description of the technical scheme of the text processing method.

[0378] Corresponding to the method embodiments described above, the present specification also provides task processing device embodiments. FIG. 15 shows a structural schematic diagram of another task processing device provided by an embodiment of the present specification. As shown in FIG. 15, the device includes:

[0379] The first determination module 1502 is configured to determine conference task data corresponding to the target voiceprint feature and the target conference.

[0380] The decoding module 1504 is configured to perform semantic decoding on the conference task data to obtain an audio discrete feature.

[0381] The second determination module 1506 is configured to determine target conference data corresponding to the conference task data according to the audio discrete feature and the target voiceprint feature.

[0382] The playing module 1508 is configured to play the target conference data according to a preset playing rule in response to a playing instruction of the client.

[0383] An embodiment of the present specification provides a task processing device. After determining the conference task data and the target voiceprint feature, the conference task data is subjected to semantic decoding to obtain an audio discrete feature corresponding to the conference task data. The audio discrete feature, as a series of discrete integer representations, can represent semantic information of the conference task data, so that the target audio determined according to the audio discrete feature and the target voiceprint feature can have semantic consistency, have a corresponding relationship between the audio and the text, ensure the stability and accuracy of the audio synthesis effect, and enable the target audio to be played in a tone corresponding to the target voiceprint feature in subsequent playing.

[0384] The above is a schematic scheme of the task processing device of the embodiment. It should be noted that the technical scheme of the task processing device belongs to the same concept as the technical scheme of the task processing method described above. Details of the technical scheme of the task processing device that are not described in detail can be referred to the description of the technical scheme of the task processing method.

[0385] FIG. 16 shows a structural block diagram of a computing device according to an embodiment of the present specification. The components of the computing device 1600 include but are not limited to a memory 1610 and a processor 1620. The processor 1620 is connected with the memory 1610 through a bus 1630, and a database 1650 is used to save data.

[0386] The computing device 1600 also includes an access device 1640 that enables the computing device 1600 to communicate via one or more networks 1660. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of such networks, such as the Internet. The access device 1640 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, or the like.

[0387] In one embodiment of the present disclosure, the above-mentioned components of the computing device 1600 and other components not shown in FIG. 16 can also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG. 16 is only for the purpose of example, and is not a limitation on the scope of the present disclosure. Those skilled in the art can add or replace other components as needed.

[0388] The computing device 1600 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1600 can also be a mobile or stationary server.

[0389] The memory 1610 is configured to store computer programs / instructions, and the processor 1620 is configured to execute the computer programs / instructions stored in the memory 1610, which implement the steps of the above-mentioned method when executed by the processor.

[0390] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the method.

[0391] An embodiment of the present specification also provides a computer readable storage medium storing computer programs / instructions, which, when executed by a processor, implement the steps of the above method.

[0392] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the method.

[0393] An embodiment of the present specification also provides a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the above method.

[0394] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the method described above belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be referred to the description of the technical scheme of the method.

[0395] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps in a claim can be performed in an order different than the order in which the acts or steps are recited and still accomplish the desired results. Also, the process depicted in the accompanying figures does not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0396] The computer program / instructions can include a computer program code, which can be in a form of source code, object code, executable file, or some intermediate form etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0397] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present specification.

[0398] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0399] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited by the claims and their entire scope and equivalents.

Claims

1. A task processing method, comprising: determining to-be-processed task data and target voiceprint features; performing semantic decoding on the to-be-processed task data to obtain audio discrete features; determining target audio corresponding to the to-be-processed task data according to the audio discrete features and the target voiceprint features.

2. The task processing method of claim 1, wherein the determining target audio corresponding to the to-be-processed task data according to the audio discrete features and the target voiceprint features comprises: performing spectral prediction on the audio discrete features and the target voiceprint features to obtain predicted spectral features; and performing spectral conversion on the predicted spectral features to obtain the target audio corresponding to the to-be-processed task data.

3. The task processing method of claim 1, wherein the to-be-processed task data comprises text of multiple language categories; and before the performing semantic decoding on the to-be-processed task data to obtain audio discrete features, the method further comprises: constructing a feature sequence from the to-be-processed task data, the multiple language categories, and the target voiceprint features; and the performing semantic decoding on the to-be-processed task data to obtain audio discrete features comprises: performing semantic decoding on the feature sequence to obtain audio discrete features.

4. The task processing method of claim 1, wherein after the determining to-be-processed task data and target voiceprint features, the method further comprises: inputting the to-be-processed task data and the target voiceprint features into an audio generation model to obtain target audio corresponding to the to-be-processed task data, wherein the audio generation model comprises a semantic decoding layer, a spectral prediction layer, and a spectral processing layer; and the inputting the to-be-processed task data and the target voiceprint features into the audio generation model to obtain target audio corresponding to the to-be-processed task data comprises: inputting the to-be-processed task data and the target voiceprint features into the semantic decoding layer to perform semantic decoding and obtain audio discrete features; inputting the audio discrete features and the target voiceprint features into the spectral prediction layer to perform spectral prediction and obtain predicted spectral features; and inputting the predicted spectral features into the spectral processing layer to perform spectral conversion and obtain the target audio corresponding to the to-be-processed task data.

5. The task processing method of claim 4, wherein the inputting the to-be-processed task data and the target voiceprint features into the semantic decoding layer to perform semantic decoding and obtain audio discrete features comprises: inputting the to-be-processed task data and the target voiceprint features into the semantic decoding layer to construct a feature sequence from the to-be-processed task data and the target voiceprint features; and performing semantic decoding on the feature sequence to obtain audio discrete features.

6. The task processing method of claim 4, wherein the inputting the audio discrete features and the target voiceprint features into the spectral prediction layer to perform spectral prediction and obtain predicted spectral features comprises: predicting, by the spectral prediction layer, a spectral change direction of initial spectral features according to the audio discrete features and the target voiceprint features to obtain a predicted spectral change direction. ​ According to the predicted spectrum change direction, the initial spectrum feature is adjusted to obtain a predicted spectrum feature.

7. The task processing method of claim 6, wherein the step of predicting a spectrum change direction of an initial spectrum feature according to the audio discrete feature and the target voiceprint feature to obtain a predicted spectrum change direction comprises: sequentially predicting spectrum change directions of the initial spectrum feature at a plurality of preset steps according to the audio discrete feature and the target voiceprint feature until a preset iteration step number is reached to obtain a predicted spectrum change direction corresponding to a target preset step, wherein the target preset step is the last step among the plurality of preset steps.

8. The task processing method of claim 7, wherein a first preset step among the plurality of preset steps is smaller than a second preset step, and the first preset step is before the second preset step.

9. The task processing method of claim 6, wherein before the step of inputting the audio discrete feature and the target voiceprint feature into the spectrum prediction layer to perform spectrum prediction to obtain a predicted spectrum feature, the method further comprises: sampling a feature distribution of an audio spectrum feature to determine an initial spectrum feature according to a sampling result.

10. The task processing method of claim 4, wherein the step of inputting the audio discrete feature and the target voiceprint feature into the spectrum prediction layer to perform spectrum prediction to obtain a predicted spectrum feature comprises: predicting a spectrum change direction of an initial spectrum feature according to the audio discrete feature and the target voiceprint feature through the spectrum prediction layer to obtain a first predicted spectrum change direction; predicting the spectrum change direction of the initial spectrum feature according to an additional guide condition, the audio discrete feature and the target voiceprint feature to obtain a second predicted spectrum change direction; determining a target predicted spectrum change direction according to the first predicted spectrum change direction and the second predicted spectrum change direction; and adjusting the initial spectrum feature according to the target predicted spectrum change direction to obtain a predicted spectrum feature.

11. The task processing method of claim 1, wherein the step of determining the task data to be processed and the target voiceprint feature comprises: receiving task data to be processed and reference audio sent by a client; and extracting a voiceprint feature of the reference audio to obtain a target voiceprint feature corresponding to the reference audio.

12. The task processing method of claim 4, wherein before the step of inputting the task data to be processed and the target voiceprint feature into the audio generation model, the method further comprises: determining a machine learning model to be trained and training data associated with an audio generation task, wherein the machine learning model to be trained comprises a semantic decoding layer, a spectrum prediction layer and a spectrum processing layer, and the training data comprises sample text, sample audio, a sample voiceprint feature corresponding to the sample audio and a sample audio discrete feature. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ According to the semantic decoding layer and the spectrum prediction layer, the sample text, the sample audio, the sample voiceprint feature and the sample audio discrete feature are processed to obtain an audio generation result; According to the audio generation result, model parameter adjustment is performed on the machine learning model to obtain a trained audio generation model.

13. The task processing method of claim 1, after the target audio corresponding to the task data to be processed is determined, further comprising: sending the target audio to an audio playing device to enable the audio playing device to play the target audio.

14. An audio generation model training method, comprising: determining a machine learning model to be trained and training data associated with an audio generation task, wherein the machine learning model to be trained comprises a semantic decoding layer, a spectrum prediction layer and a spectrum processing layer, and the training data comprises sample text, sample audio, sample voiceprint features corresponding to the sample audio and sample audio discrete features; according to the semantic decoding layer and the spectrum prediction layer, processing the sample text, the sample audio, the sample voiceprint features and the sample audio discrete features to obtain an audio generation result; according to the audio generation result, performing model parameter adjustment on the machine learning model to obtain a trained audio generation model.

15. The audio generation model training method of claim 14, before the sample text, the sample audio, the sample voiceprint features and the sample audio discrete features are processed according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result, further comprising: performing feature extraction on the sample text to obtain sample text features; according to a preset sequence start symbol and a preset sequence end symbol, constructing a sample feature sequence from the sample voiceprint features, the sample text features and the sample audio discrete features.

16. The audio generation model training method of claim 15, before the sample text, the sample audio, the sample voiceprint features and the sample audio discrete features are processed according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result, further comprising: performing feature extraction on the sample audio to obtain sample audio features; performing masking processing on the sample audio features to obtain masked sample audio features.

17. The audio generation model training method of claim 16, the training data further comprises sample initial spectrum features and sample steps; the processing of the sample text, the sample audio, the sample voiceprint features and the sample audio discrete features according to the semantic decoding layer and the spectrum prediction layer to obtain an audio generation result comprises: according to the semantic decoding layer, performing autoregressive semantic decoding on the sample feature sequence to obtain predicted audio discrete features; according to the spectrum prediction layer, performing spectrum prediction on the sample audio discrete features, the sample voiceprint features, the masked sample audio features, the sample initial spectrum features and the sample steps to obtain sample predicted spectrum features.

18. The audio generation model training method of claim 17, wherein the model parameter adjustment of the machine learning model according to the audio generation result to obtain a trained audio generation model comprises: performing model parameter adjustment on the semantic decoding layer according to the predicted audio discrete feature and a label audio discrete feature corresponding to the sample audio; and performing model parameter adjustment on the spectrum prediction layer according to the sample predicted spectrum feature and a label spectrum feature corresponding to the sample audio to obtain the trained audio generation model.

19. A text processing method, comprising: receiving a text reading instruction for a target text sent by a client and determining a target voiceprint feature; performing semantic decoding on the target text to obtain an audio discrete feature; determining a target audio corresponding to the target text according to the audio discrete feature and the target voiceprint feature; and sending the target audio to the client to enable the client to play the target audio.

20. The text processing method of claim 19, wherein after receiving the text reading instruction for the target text sent by the client, the method further comprises: invoking a text processing model, inputting the target text into the text processing model, and obtaining an abstract text corresponding to the target text; performing semantic decoding on the abstract text to obtain an audio discrete feature; determining an abstract audio corresponding to the abstract text according to the audio discrete feature and the target voiceprint feature; and sending the abstract audio to the client to enable the client to play the abstract audio.

21. The text processing method of claim 20, wherein after sending the abstract audio to the client, the method further comprises: in a case where an acknowledgement instruction sent by the client is received, performing the step of performing semantic decoding on the target text to obtain an audio discrete feature.

22. A task processing method, comprising: determining a conference task data corresponding to a target voiceprint feature and a target conference; performing semantic decoding on the conference task data to obtain an audio discrete feature; determining a target conference data corresponding to the conference task data according to the audio discrete feature and the target voiceprint feature; and playing the target conference data according to a preset playing rule in response to a playing instruction of a client.

23. A computing device, comprising: a memory and a processor; the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method of any one of claims 1 to 22.

24. A computer-readable storage medium storing computer programs / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 22.

25. A computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 22. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images