Audio generation method and device, electronic equipment and computer program product

Abstract

The invention provides an audio generation method and device, electronic equipment and a computer program product, and can obtain a to-be-processed text and audio style information which is used for indicating an audio style corresponding to a to-be-generated target audio. And inputting the to-be-processed text and the audio style information into a pre-trained audio editing model to obtain discrete codes output by the audio editing model and corresponding to the to-be-processed text and the audio style information, and decoding the discrete codes to obtain a target audio. Through the setting, the discrete codes corresponding to the to-be-processed text and the audio style information are output by using the audio editing model, and the audio of which the content style meets the requirements is generated by decoding the discrete codes, so that the audio and video editing difficulty is greatly reduced, and the creation efficiency of a user is improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio generation, and in particular to an audio generation method, device, electronic device and computer program product. Background Art

[0002] Publishing videos has become an important way for people to record their lives, share their views, and express their emotions. However, there are often a lot of pauses, blurts, inadequate words, or redundant content when shooting, and people need to record multiple times or do tedious editing work to get a satisfactory video. If the voice content in the video can be directly generated in the form of editing text, it will greatly reduce the difficulty of audio and video editing and improve the user's creation efficiency.

[0003] Therefore, how to directly generate the voice content in the video by editing text has become a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0004] In view of this, the present application proposes an audio generation method, device, electronic device and computer program product, which can directly generate voice content in a video by editing text.

[0005] The technical solutions proposed in this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides an audio generation method, comprising:

[0007] Acquire the text to be processed and audio style information; wherein the audio style information is used to indicate the audio style corresponding to the target audio to be generated;

[0008] Inputting the text to be processed and the audio style information into a pre-trained audio editing model to obtain a discrete code output by the audio editing model corresponding to the text to be processed and the audio style information;

[0009] The discrete code is decoded to obtain the target audio.

[0010] In a second aspect, an embodiment of the present application provides an audio generating device, comprising:

[0011] An acquisition module, used to acquire the text to be processed and audio style information; wherein the audio style information is used to indicate the audio style corresponding to the target audio to be generated;

[0012] An input module, used for inputting the text to be processed and the audio style information into a pre-trained audio editing model, and obtaining a discrete code output by the audio editing model corresponding to the text to be processed and the audio style information;

[0013] A decoding module is used to decode the discrete code to obtain the target audio.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0015] A memory and a processor; wherein the memory is used to store programs; and the processor is used to implement any of the methods described above by running the programs in the memory.

[0016] In a fourth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, any of the methods described above is implemented.

[0017] The audio generation method proposed in this application can obtain the text to be processed and the audio style information, wherein the audio style information is used to indicate the audio style corresponding to the target audio to be generated, and then input the text to be processed and the audio style information into a pre-trained audio editing model to obtain the discrete code output by the audio editing model corresponding to the text to be processed and the audio style information, and obtain the target audio by decoding the discrete code. In this way, the audio editing model is used to output the discrete code corresponding to the text to be processed and the audio style information, and the audio with the content style that meets the requirements is generated by decoding the discrete code, which greatly reduces the difficulty of audio and video editing and improves the user's creation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0019] Figure 1 It is a schematic diagram of a feasible application scenario of an audio generation method provided in an embodiment of the present application.

[0020] Figure 2 It is a flowchart of an audio generation method provided in an embodiment of the present application.

[0021] Figure 3 It is a schematic diagram of performing mask processing on discrete coding of samples provided in an embodiment of the present application.

[0022] Figure 4 It is a schematic diagram of the training process of the audio editing model provided in an embodiment of the present application.

[0023] Figure 5 It is a structural schematic diagram of an audio generating device provided in an embodiment of the present application.

[0024] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] Nowadays, more and more people are posting videos on social networking platforms as a way to record their lives, share their views and express their emotions. However, in the actual shooting process, there are often a lot of pauses, blurts, inadequate words or redundant content. People often need to remove such content after multiple recordings or tedious editing work to get a satisfactory video. If the voice content in audio and video can be edited directly in the form of text editing, it will greatly reduce the difficulty of audio and video editing and improve the user's creative efficiency. Therefore, it is necessary to develop a text-based voice editing system.

[0027] Based on this, the present application proposes an audio generation method, device, electronic device and computer program product. The method can directly generate voice content in a video by editing text, thereby reducing the difficulty of audio and video editing and improving the user's creative efficiency.

[0028] Figure 1 A feasible application scenario of the audio generation method is shown, such as Figure 1 In the scenario shown, a client and a server are set up.

[0029] The client can be an electronic device with network access capability. Specifically, for example, the client can be a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a smart wearable device, a shopping guide terminal, a television, etc. Among them, the smart wearable device includes but is not limited to a smart bracelet, a smart watch, a smart glasses, a smart helmet, a smart necklace, etc. Alternatively, the client can also be software that can run in an electronic device.

[0030] The server can be an electronic device with certain computing and processing capabilities. It can have a network communication module, a processor, and a memory, etc. Of course, the server can also refer to software running in an electronic device. The server can also be a distributed server, which can be a system with multiple processors, memories, network communication modules, etc. operating in collaboration. Alternatively, the server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the implementation method of the specification. For example, it can be a new form of "server" based on quantum computing.

[0031] The client and the server can communicate through the target network. The target network can be any type of network. For example, the target network can be a network that can be subdivided into multiple subnetworks. The target network or the multiple subnetworks contained in the target network can be at least one of a cellular mobile network (such as 2G, 3G, 4G or 5G), ZIGBEE, WIFI, Bluetooth, or any combination of at least one of these networks and other networks.

[0032] In the above feasible application scenario, the client collects the text and audio style information to be processed, and sends the text and audio style information to be processed to the server. After the server obtains the text and audio style information to be processed from the client, it inputs the text and audio style information to be processed into the pre-trained audio editing model, obtains the discrete code corresponding to the text and audio style information to be processed output by the audio editing model, decodes the discrete code, obtains the target audio, and sends the above target audio to the client. In this way, the audio editing model is used to output the discrete code corresponding to the text and audio style information to be processed, and the audio with the required content style is generated by decoding the discrete code, which greatly reduces the difficulty of audio and video editing and improves the user's creative efficiency.

[0033] Furthermore, an exemplary embodiment of the present specification provides an audio generation method, which is described by taking the method applied to an electronic device as an example. The electronic device may be a device having communication and computing functions such as the server in the above embodiment. Figure 2 As shown, the method includes:

[0034] S101: Obtain text and audio style information to be processed.

[0035] The above-mentioned text to be processed refers to the text that needs to be edited into audio, that is, in the embodiment of the present application, the target audio is generated based on the text to be processed. For example, if the text "This shampoo is a very easy-to-use shampoo, with a strong anti-dandruff function" needs to be edited into audio, then the text "This shampoo is a very easy-to-use shampoo, with a strong anti-dandruff function" is the above-mentioned text to be processed.

[0036] The audio style information is used to indicate the audio style corresponding to the target audio to be generated. Extracting the audio style information in advance and generating the target speech based on the audio style information helps to improve the timbre similarity between the target audio to be generated and the user audio, making the target audio to be generated more realistic and in line with user requirements.

[0037] In some embodiments, if it is necessary to edit a segment of user audio input by the user, delete part of the audio content in the segment of user audio, and generate a target audio to fill the deleted part, the user does not need to input additional audio, and the audio style information can be directly extracted from the segment of user audio; if the user's audio is not edited, but the target audio is generated directly, the user is required to input one or more segments of user audio in order to extract the audio style information therefrom. The length of the audio input by the user should be sufficient to extract the audio style information, and this embodiment does not limit this.

[0038] Furthermore, the audio style information can be extracted from the user audio by entering the following steps:

[0039] Extract the audio Mel spectrum of the user audio; encode the audio Mel spectrum into an acoustic latent vector, resample the acoustic latent vector, and obtain the audio style information corresponding to the user audio.

[0040] Specifically, the audio Mel spectrum of the user audio can be extracted as input, and then the audio style information can be extracted therefrom using a style encoder. The style encoder includes a Transformer-based Mel encoder and an Optimal Transport-based resampler. The Mel encoder encodes the audio Mel spectrum into an acoustic latent vector, and the resampler converts the acoustic latent vector into a style embedding of a fixed dimension that is independent of the length of the input audio as the audio style information. The above-mentioned fixed dimension can be set according to actual conditions, and this embodiment does not limit it. For example, a fixed 32-dimensional dimension is selected.

[0041] In some embodiments, the audio style information is extracted by optimizing the transmission, which can avoid the leakage of the audio content to a certain extent.

[0042] S102: Input the text to be processed and the audio style information into a pre-trained audio editing model to obtain a discrete code output by the audio editing model that corresponds to the text to be processed and the audio style information.

[0043] In the embodiment of the present application, an audio editing model is pre-trained, and the audio editing model has the ability to perform the task of converting the text to be processed and the audio style information into discrete codes (tokens). Discrete codes are generally obtained by encoding and compressing audio. Discrete codes can compress the length of the video. The ability to train the audio editing model to generate discrete codes can make the audio editing model not have to directly generate the video, reduce the data processing amount of the model, and facilitate modeling.

[0044] In a specific embodiment, a large amount of sample audio can be acquired in advance, and the text and audio style information corresponding to the sample audio can be used as training samples. The sample audio is compressed and encoded to obtain the sample discrete code as a training label, and the audio editing model is trained with the goal of predicting the discrete code corresponding to the training sample until the prediction ability of the audio editing model meets the requirements. The above-mentioned audio editing model can be trained based on any neural network model, or based on a pre-trained model, such as a pre-trained large model similar to ChatGPT.

[0045] After the audio editing model training is completed, the text to be processed and the audio style information are input into the audio editing model to obtain the discrete code output by the audio editing model corresponding to the text to be processed and the audio style information.

[0046] S103: Decode the discrete code to obtain target audio.

[0047] After the discrete code is obtained, the discrete code is decoded. For example, the discrete code can be input into a decoder to obtain the target audio.

[0048] In some embodiments, a codec can be pre-set, and the codec can be used to encode and compress the sample audio to obtain a discrete code corresponding to the sample audio, that is, to obtain a training label for the audio editing model; at the same time, the codec is used to decode the discrete code output by the audio editing model to obtain the target audio.

[0049] The neural audio codec can encode audio into discrete codes that can retain acoustic information about the speaker and recording conditions, and can directly reconstruct high-quality audio waveforms without the need for an additional vocoder, even if the speaker has never been seen during training. Based on this feature, in some embodiments, the neural audio codec is used to encode and compress the sample audio to obtain the discrete code corresponding to the sample audio, that is, to obtain the training label of the audio editing model; at the same time, the neural audio codec is used to decode the discrete code output by the audio editing model to obtain the target audio.

[0050] In the above embodiment, the text to be processed and the audio style information can be obtained, wherein the audio style information is used to indicate the audio style corresponding to the target audio to be generated, and then the text to be processed and the audio style information are input into the pre-trained audio editing model to obtain the discrete code corresponding to the text to be processed and the audio style information output by the audio editing model, and the target audio is obtained by decoding the discrete code. In this way, the audio editing model is used to output the discrete code corresponding to the text to be processed and the audio style information, and the audio with the content style that meets the requirements is generated by decoding the discrete code, which greatly reduces the difficulty of audio and video editing and improves the user's creation efficiency.

[0051] As an optional implementation, another embodiment of the present application discloses that the training process of the audio editing model in the above embodiment may specifically include the following steps:

[0052] Obtain training samples for the audio editing model; the training samples include sample text corresponding to the sample audio, sample audio style information corresponding to the sample audio, and mask codes corresponding to the sample audio; the mask codes are obtained by masking a set area in a sample discrete code of the sample audio; the training samples are input into the audio editing model, and the audio editing model is trained with the goal of predicting the content of the mask area in the mask code.

[0053] Specifically, a large amount of audio containing human voice can be acquired in advance as sample audio.

[0054] The text corresponding to the human voice in the sample audio is extracted as the sample text. The human voice in the sample audio can be converted into the corresponding text to obtain the sample text using the speech conversion technology in the prior art, which is not limited in this embodiment. The above-mentioned speech conversion technology is a very conventional prior art in this field, and those skilled in the art can refer to the prior art without consuming creativity, and will not be elaborated here.

[0055] Extract the sample audio style information corresponding to the sample audio. The sample audio style information can be extracted from the sample audio in the manner of extracting the audio style information from the user voice as described in the above embodiment. Specifically, the audio Mel spectrum of the sample audio can be extracted as input, and then the sample audio style information can be extracted therefrom using a style encoder. More specifically, the Mel encoder in the style encoder encodes the audio Mel spectrum into an acoustic latent vector, and the resampler converts the acoustic latent vector into a style embedding of a fixed dimension that is independent of the length of the input audio as the audio style information.

[0056] The sample audio is encoded and compressed to obtain a sample discrete code corresponding to the sample audio. In some embodiments, the sample audio is input into a neural audio codec, and the sample audio is encoded and compressed using the neural audio codec to obtain a sample discrete code corresponding to the sample audio. Specifically, the neural audio codec includes three modules: an encoder, a residual vector quantizer (RVQ), and a decoder. The encoder encodes the audio into a potential representation, and then the residual vector quantizer quantizes the potential representation into a discrete code, and finally the decoder network reconstructs the discrete code into an audio waveform. The residual vector quantizer includes multiple layers of vector quantizers (VQ). After each layer of quantization, the residual is input to the next layer of quantization. The entire system is trained in an end-to-end manner, and adversarial training is used to improve audio quality. More specifically, the sample audio is passed through the above-mentioned encoder and RVQ in turn to obtain a sample discrete code corresponding to the sample audio.

[0057] Then, a set area in the sample discrete code is masked. The set area may be designated and identified in advance, or may be a random area in the sample discrete code, which is not limited in this embodiment.

[0058] In this embodiment, the training sample includes the above-mentioned sample text, sample audio style information and mask code. The training sample is input into the audio editing model, and the audio editing model is trained with the goal of predicting the content of the mask area in the mask code. During the training process, the audio editing model can learn the ability to generate discrete codes corresponding to the sample text, and the discrete codes it generates also carry the sample audio style information. In this way, the discrete codes output by the audio editing model are decoded, and the obtained audio content and style meet the user's requirements.

[0059] In this embodiment, a large language model is used as the basic model of the audio editing model, such as ChatGPT. Discrete coding can compress the length to facilitate the modeling of the large language model. On the other hand, with discrete coding, the regression problem can be converted into a relatively simple classification problem, and the modeling ideas in the large language model can be used to implement the audio synthesis task, which is simpler.

[0060] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment to obtain the training sample of the audio editing model may specifically include the following steps:

[0061] A portion of continuous coding sequences is randomly selected in the sample discrete coding of the sample audio for masking to obtain a mask sequence; the mask sequence is moved to the end of the sample discrete coding, and a mask identifier is added to the position before and after the mask sequence is moved to obtain the mask code corresponding to the sample audio.

[0062] In an embodiment of the present application, a causal mask-based method is used to perform mask processing on the sample discrete code of the sample audio. Specifically, a portion of continuous coding sequences can be randomly selected from the sample discrete code of the sample audio for mask processing, and the portion of the sample discrete code that is masked is the mask sequence. The mask sequence is then moved to the end of the sample discrete code, and a mask identifier is added to the position before the mask sequence is moved and the position after the mask sequence is moved to identify the mask position, so that the mask code corresponding to the sample audio can be obtained. It should be noted that the mask code corresponding to the sample audio includes a coding sequence that has not been masked, a mask sequence that has been masked, and a mask identifier.

[0063] Figure 3 The figure shows the process of masking a set of sample discrete codes using causal masking. Figure 3 As shown, the sample discrete code includes X1, X2, X3, X4, X5, and X6. X3 and X4 are randomly selected as mask sequences, and X3 and X4 are moved to the end of the sample discrete code. Then, the mask identifier m is added to the position before the mask sequence is moved and the position after the mask sequence is moved, and the mask codes X1, X2, m, X5, X6, m, X3, and X4 are obtained, where X3 and X4 are masked masks.

[0064] In an embodiment of the present application, since the mask area is shifted backward, the audio editing model can consider all information before the current position when predicting the content of the mask area, where all information before the current position includes context information before the mask sequence moves. As a result, the audio editing model can obtain sufficient context information when predicting the content of the mask area, thereby achieving a smooth transition before and after the editing position.

[0065] With this setting, the audio editing model has the ability to predict the content of the masked area based on all the information. Even if it is at the very beginning or the very end of the text to be processed, sufficient contextual information can be obtained during the prediction, and inaccurate predictions will not occur due to lack of sufficient contextual information.

[0066] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment to obtain the training sample of the audio editing model may specifically include the following steps:

[0067] The sample audio is encoded into sample discrete codes using a residual vector quantizer.

[0068] As described in the above embodiment, the sample audio can be encoded into sample discrete codes by using RVQ. Specifically, the sample audio passes through the encoder of the neural audio codec, the encoder encodes the sample audio into sample potential representation, and then RVQ quantizes the sample potential representation into sample discrete codes.

[0069] Such a setting can obtain sample discrete encoding, so as to facilitate the training of the audio editing model so that the audio editing model has the ability to generate discrete encoding.

[0070] As an optional implementation method, another embodiment of the present application discloses that the residual vector quantizer of the above embodiment includes multiple layers of parallel sub-residual vector quantizers, the sample discrete coding includes sub-sample discrete coding output by each layer of sub-residual vector quantizers, and correspondingly, the mask coding includes sub-mask coding obtained by masking a set area in each sub-sample discrete coding.

[0071] Specifically, the RVQ in the prior art includes multiple layers of VQ quantizers, which causes the discrete codes to be generated to be too long, increasing the difficulty of learning the audio editing model. Considering that the working principle of RVQ is to input the residual to the next layer for quantization after each layer is quantized, there is a natural dependency between different layers, and the first layer contains the richest information, and the impact on the audio synthesis quality becomes smaller as time goes by. In this embodiment, multiple layers of parallel sub-residual vector quantizers are used to generate multiple layers of sub-sample discrete codes in parallel without significantly increasing the sequence length. Among them, one layer of sub-residual vector quantizer corresponds to generating one layer of sub-sample discrete codes. In the embodiment of the present application, after the sample audio is sent to the residual vector quantizer through the encoder, the sample audio is encoded into multiple layers of sub-sample discrete codes by multiple layers of parallel sub-residual vector quantizers to form sample discrete codes corresponding to the sample audio.

[0072] For example, if the sub-residual vector quantizer has 4 layers, after the sample audio is sent to the sub-residual vector quantizer through the encoder, 4 discrete groups of sub-sample discrete codes X are obtained. q ={X q1 ,X q2 ,……,X qT}, {q=1,2,3,4}, T represents the sequence length of the sample audio after being compressed by the sub-residual vector quantizer.

[0073] The steps of the above embodiment obtain the training samples of the audio editing model, including: performing mask processing on the set area in each sub-sample discrete code to obtain the sub-mask code corresponding to each sub-sample discrete code; stacking the sub-mask codes corresponding to each sub-sample discrete code respectively to obtain the sub-mask code corresponding to each sub-sample discrete code after stacking; summing the sub-mask codes corresponding to all the sub-sample discrete codes after stacking to obtain the mask code.

[0074] Specifically, after masking the set area in each sub-sample discrete code, the sub-mask code corresponding to each sub-sample discrete code is obtained. Specifically, according to the records of the above embodiment, each sub-sample discrete code can be processed in a causal masking manner to obtain a sub-mask code. For any group of sub-sample discrete codes, the following processing can be performed:

[0075] Randomly select a part of the continuous code sequence in the sub-sample discrete code for masking to obtain a mask sequence; move the mask sequence to the end of the sub-sample discrete code, and add a mask identifier at the position before and after the mask sequence is moved to obtain a sub-mask code corresponding to the sub-sample discrete code. For a more specific processing process, refer to the mask process recorded in the above embodiment, which will not be repeated here.

[0076] After obtaining the sub-mask codes corresponding to each sub-sample discrete code, the sub-mask codes corresponding to each sub-sample discrete code may be stacked in a stacked manner to obtain the sub-mask codes corresponding to each sub-sample discrete code after the stacking.

[0077] The stacking mentioned above means that each sub-mask code is controlled to move in the same direction, and the number of moving bits is consistent with the number of layers corresponding to each sub-mask code, where the number of layers represents the number of layers of the sub-residual vector quantizer or the number of layers of random coding that generates the sub-sample discrete coding corresponding to the sub-mask code, where the number of layers of random coding should start from 1 and be continuous. For example, if the sub-mask code is 10 layers, the sub-mask codes of each layer are randomly encoded as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Then <eos>The token is filled into the empty position in each sub-mask code to make the length of each sub-mask code consistent, and the sub-mask code corresponding to each sub-sample discrete code after stacking is obtained.

[0078] For example, if the sub-residual vector quantizer has 4 layers, after the sample audio is sent to the sub-residual vector quantizer through the encoder, 4 groups of sub-sample discrete codes X are obtained. q ={X q1 ,X q2 ,……,X qT }, {q=1,2,3,4}, T represents the sequence length of the sample audio after being compressed by the sub-residual vector quantizer. If T is 4, the sub-mask codes corresponding to the discrete codes of each sub-sample are stacked respectively, and the sub-mask codes corresponding to the discrete codes of each sub-sample after stacking are obtained as follows:

[0079] X1={E,X 11 ,X 12 ,X 13 ,X 14 ,E,E,E}

[0080] X2={E,E,X 21 ,X 22 ,X 23 ,X 24 ,E,E}

[0081] X3={E,E,E,X 31 ,X 32 ,X 33 ,X 34 ,E}

[0082] X4={E,E,E,E,X 41 ,X 42 ,X 43 ,X 44 }

[0083] E stands for <eos>The number of layers of X1 is 1, and it is shifted 1 bit to the right. The number of layers of X2 is 2, and it is shifted 2 bits to the right. The number of layers of X3 is 3, and it is shifted 3 bits to the right. The number of layers of X4 is 4, and it is shifted 4 bits to the right. <eos>The token is filled into the empty space in each sub-mask code so that the length of each sub-mask code is consistent and is 8 bits.

[0084] Finally, the sub-mask codes corresponding to the discrete codes of all the stacked sub-samples are summed to obtain the mask code. In some embodiments, the sub-mask codes corresponding to the discrete codes of each sub-sample after stacking can be respectively passed through the embedding layer to obtain the corresponding token representation, and then the four layers of token representation are summed to obtain the mask code X emb :

[0085]

[0086] In the above embodiment, by adopting multi-layer parallel sub-residual vector quantizers, multi-layer sub-sample discrete codes are generated in parallel without significantly increasing the sequence length, thereby reducing the learning difficulty of the audio editing model. The stacking method avoids the significant increase in the length of discrete codes caused by flattening, and does not need to use multiple large language models to generate different layers of discrete codes like VALL-E. Instead, the same large language model can be used to generate multiple layers of discrete codes in parallel, thereby simplifying the generation of the language model.

[0087] As an optional implementation, another embodiment of the present application discloses that the training sample of the above embodiment also includes the duration corresponding to each phoneme in the sample audio. The steps of the above embodiment to obtain the training sample of the audio editing model may specifically include the following steps:

[0088] By aligning the sample text and sample audio corresponding to the sample audio, the alignment duration corresponding to each phoneme is determined; if it is detected that the alignment duration of any phoneme is less than the set duration, the alignment duration is determined to be the duration corresponding to the phoneme; if it is detected that the alignment duration of any phoneme is greater than or equal to the set duration, the set duration is determined to be the duration corresponding to the phoneme.

[0089] Due to the inherent defects of the large language model architecture, it is easy to have multiple word omission problems, such as multiple word omission problems caused by incorrect prediction of the end symbol position. In order to alleviate this problem, duration information is introduced in the embodiment of the present application to help and stabilize the generation of discrete coding. Specifically, the sample text and sample audio corresponding to the sample audio can be aligned to determine the alignment duration corresponding to each phoneme. Specifically, the sample text and sample audio can be aligned in a hard alignment (Force Alignment, FA) manner, and the present embodiment is not limited. Then define a set duration as the maximum duration value, and all alignment durations exceeding the set duration will be staged as the maximum value. Specifically, the alignment duration corresponding to each factor and the set duration can be compared. If the alignment duration of any phoneme is detected to be less than the set duration, the alignment duration is determined to be the duration corresponding to the phoneme; if the alignment duration of any phoneme is detected to be greater than or equal to the set duration, the set duration is determined to be the duration corresponding to the phoneme.

[0090] This setting allows the audio editing model to learn the normal duration of each phoneme in the audio, and then constrain the predicted discrete code length, so that the duration of each phoneme in the final generated audio is consistent with common sense, reducing the trigger rate of multiple word omissions.

[0091] As an optional implementation, it is disclosed in another embodiment of the present application that the training sample of the above embodiment also includes each phoneme in the sample audio.

[0092] More specifically, the training samples of the above embodiment include sample audio style information, sample text, phonemes, set duration and mask code. During the training process, the sample audio style information, sample text, phonemes, set duration and mask code are all input into the audio editing model, and the audio editing model is trained with the goal of predicting the content of the mask area in the mask code. Among them, the sample audio style information enables the audio editing model to learn audio style generation, and the set duration enables the audio editing model to learn the duration information of the phonemes.

[0093] Figure 4 FIG. 4 shows the training process of the audio editing model. Figure 4 As shown, the audio editing model predicts X 11 ,E,E,E, and then according to X 11 ,E,E,E,The audio editing model can predict X 12 ,X 21 ,E,E, according to X 12 ,X 21 ,E,E,The audio editing model can predict X 13 ,X 22 ,X 31 ,E, and so on, until the prediction is completed, and the discrete code output by the audio editing model is obtained. In some embodiments, four classification layers can be used to obtain the posterior probability of the four layers of discrete code on each frame, and then the discrete code is selected from them according to the size of the posterior probability as the discrete code output by the above audio editing model. It should be noted that the category with the largest posterior probability is usually selected as the output of the model.

[0094] Then, by comparing the contents of the masked area of ​​the discrete code with the corresponding area in the predicted discrete code output by the audio editing model, the cross entropy loss between the masked area of ​​the discrete code and the corresponding area in the predicted discrete code output by the audio editing model is calculated, and the loss value of the audio editing model is determined. The parameters of the audio editing model are adjusted with the goal of reducing the loss value, and then the above training is repeated until the audio editing model is stable.

[0095] In an embodiment of the present application, the speech synthesis task is regarded as a conditional language modeling task. The corresponding discrete code is generated autoregressively based on the text to be synthesized, and the discrete code is then sent to the decoder of the codec to obtain the target audio.

[0096] Corresponding to the above-mentioned audio generation method, the present application embodiment also discloses an audio generation device, see Figure 5 As shown, the device comprises:

[0097] The acquisition module 100 is used to acquire the text to be processed and the audio style information; wherein the audio style information is used to indicate the audio style corresponding to the target audio to be generated;

[0098] An input module 110, used to input the text to be processed and the audio style information into a pre-trained audio editing model, and obtain a discrete code output by the audio editing model corresponding to the text to be processed and the audio style information;

[0099] The decoding module 120 is used to decode the discrete code to obtain the target audio.

[0100] As an optional implementation, another embodiment of the present application discloses that the audio generating device in the above embodiment may specifically include:

[0101] A training module is used to obtain training samples for an audio editing model; the training samples include sample text corresponding to the sample audio, sample audio style information corresponding to the sample audio, and mask codes corresponding to the sample audio; the mask codes are obtained by masking a set area in a sample discrete code of the sample audio; the training samples are input into the audio editing model to train the audio editing model with the goal of predicting the content of the mask area in the mask code.

[0102] As an optional implementation, another embodiment of the present application discloses that the training module of the above embodiment, when obtaining the training sample of the audio editing model, is specifically used to:

[0103] A portion of continuous coding sequences is randomly selected in the sample discrete coding of the sample audio for masking to obtain a mask sequence; the mask sequence is moved to the end of the sample discrete coding, and a mask identifier is added to the position before and after the mask sequence is moved to obtain the mask code corresponding to the sample audio.

[0104] As an optional implementation, another embodiment of the present application discloses that the training module of the above embodiment, when obtaining the training sample of the audio editing model, is specifically used to:

[0105] The sample audio is encoded into sample discrete codes using a residual vector quantizer.

[0106] As an optional implementation, another embodiment of the present application discloses that the residual vector quantizer includes multiple layers of parallel sub-residual vector quantizers; the sample discrete coding includes sub-sample discrete coding output by each layer of sub-residual vector quantizers; the training module of the above embodiment, when obtaining the training sample of the audio editing model, is specifically used to:

[0107] After masking the set area in each sub-sample discrete code, a sub-mask code corresponding to each sub-sample discrete code is obtained; the sub-mask codes corresponding to each sub-sample discrete code are stacked in a stacked manner to obtain the sub-mask code corresponding to each sub-sample discrete code after the stacking; the sub-mask codes corresponding to all the sub-sample discrete codes after the stacking are summed to obtain the mask code.

[0108] As an optional implementation, another embodiment of the present application discloses that the training sample also includes the duration corresponding to each phoneme in the sample audio; the training module of the above embodiment, when obtaining the training sample of the audio editing model, is specifically used to:

[0109] By aligning the sample text and sample audio corresponding to the sample audio, the alignment duration corresponding to each phoneme is determined; if it is detected that the alignment duration of any phoneme is less than the set duration, the alignment duration is determined to be the duration corresponding to the phoneme; if it is detected that the alignment duration of any phoneme is greater than or equal to the set duration, the set duration is determined to be the duration corresponding to the phoneme.

[0110] As an optional implementation, another embodiment of the present application discloses that the training sample also includes each phoneme in the sample audio.

[0111] Specifically, the device provided in this embodiment belongs to the same application concept as the method provided in the above embodiment of this application, can execute the method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not fully described in this embodiment, please refer to the specific processing content of the method provided in the above embodiment of this application, which will not be repeated here.

[0112] The functions implemented by the above modules can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0113] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0114] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0115] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0116] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0117] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input-output component through the interface circuit of the control device.

[0118] The input-output component specifically refers to a hardware component that enables a user to input information and output information to the user, such as a microphone, keyboard, handwriting tablet, touch screen, display, speaker, printer, etc.

[0119] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0120] The processor in the control device is a circuit with signal processing capability, which converts text into audio by executing any of the audio generation methods described in the above embodiments. The specific implementation of the processor can refer to the above processor implementation, and the embodiments of this application are not strictly limited.

[0121] When the control device is applied to a device with a human-computer interaction function, the input and output components of the control device may be input components and output components on the device, such as a microphone, a keyboard, a handwriting tablet, a touch screen, a display, an audio player, etc. At the same time, the processor of the control device may be a CPU or GPU, etc. provided by the device, and the interface circuit of the control device may be an interface circuit between the information input component of the device and a processor such as a CPU or GPU.

[0122] Corresponding to the above audio generation method, the present application embodiment also discloses an electronic device, see Figure 6 As shown, the electronic device includes:

[0123] Memory 200 and processor 210;

[0124] The memory 200 is connected to the processor 210 and is used to store programs;

[0125] The processor 210 is used to implement the audio generation method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0126] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0127] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0128] A bus may include a pathway that transfers information between components of a computer system.

[0129] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0130] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0131] The memory 200 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0132] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0133] Output device 240 may include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0134] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0135] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the audio generation method provided in the above embodiments of the present application.

[0136] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the various steps of the audio generation method provided in the above embodiments.

[0137] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0138] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0139] In addition, the embodiments of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes each step of the audio generation method provided in the above embodiments.

[0140] The computer readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0141] Specifically, the specific working contents of each part of the above-mentioned electronic device, computer program product and storage medium, as well as the specific processing contents when the computer program product or the computer program on the above-mentioned storage medium is executed by the processor, can be found in the contents of the various embodiments of the above-mentioned audio generation method, and will not be repeated here.

[0142] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0143] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0144] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0145] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0147] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0149] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0150] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0151] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0152] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / eos> < / eos> < / eos>

Claims

1. An audio generation method, characterized in that: include: Acquire the text to be processed and audio style information; wherein the audio style information is used to indicate the audio style corresponding to the target audio to be generated; Inputting the text to be processed and the audio style information into a pre-trained audio editing model to obtain a discrete code output by the audio editing model corresponding to the text to be processed and the audio style information; The discrete code is decoded to obtain the target audio.

2. The method according to claim 1, characterized in that: The training process of the audio editing model includes: Acquire a training sample of the audio editing model; the training sample includes a sample text corresponding to the sample audio, sample audio style information corresponding to the sample audio, and a mask code corresponding to the sample audio; the mask code is obtained by performing mask processing on a set area in a sample discrete code of the sample audio; The training samples are input into the audio editing model, and the audio editing model is trained with the goal of predicting the content of the mask area in the mask code.

3. The method according to claim 2, characterized in that The step of obtaining a training sample of the audio editing model comprises: Randomly selecting a portion of continuous coding sequences in the sample discrete coding of the sample audio for masking to obtain a masked sequence; The mask sequence is moved to the end of the sample discrete code, and mask identifiers are added to the position before and after the mask sequence is moved, so as to obtain the mask code corresponding to the sample audio.

4. The method according to claim 2, characterized in that: The step of obtaining a training sample of the audio editing model includes: The sample audio is encoded into the sample discrete code using a residual vector quantizer.

5. The method according to claim 4, characterized in that The residual vector quantizer includes multiple layers of parallel sub-residual vector quantizers; the sample discrete coding includes sub-sample discrete coding output by the sub-residual vector quantizers of each layer; The step of obtaining a training sample of the audio editing model comprises: After performing mask processing on the set area in each sub-sample discrete code, a sub-mask code corresponding to each sub-sample discrete code is obtained; The sub-mask codes corresponding to the discrete codes of each sub-sample are stacked in a stacked manner to obtain the sub-mask codes corresponding to the discrete codes of each sub-sample after the stacking; The sub-mask codes corresponding to all the stacked sub-sample discrete codes are summed to obtain the mask code.

6. The method according to claim 2, characterized in that The training sample also includes the duration corresponding to each phoneme in the sample audio; The step of obtaining a training sample of the audio editing model includes: Determine the alignment duration corresponding to each phoneme by aligning the sample text corresponding to the sample audio and the sample audio; If it is detected that the alignment duration of any phoneme is less than the set duration, the alignment duration is determined to be the duration corresponding to the phoneme; if it is detected that the alignment duration of any phoneme is greater than or equal to the set duration, the set duration is determined to be the duration corresponding to the phoneme.

7. The method according to claim 2, characterized in that The training sample also includes each phoneme in the sample audio.

8. An audio generating device, characterized in that: include: An acquisition module, used to acquire the text to be processed and audio style information; wherein the audio style information is used to indicate the audio style corresponding to the target audio to be generated; An input module, used for inputting the text to be processed and the audio style information into a pre-trained audio editing model to obtain a discrete code output by the audio editing model corresponding to the text to be processed and the audio style information; A decoding module is used to decode the discrete code to obtain the target audio.

9. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to implement the method according to any one of claims 1 to 7 by running the program in the memory.

10. A computer program product, characterized in that The computer program product comprises a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images