Audio generation model training method, device and electronic equipment

Abstract

The present application discloses a training method, device and electronic device for an audio generation model, and belongs to the field of artificial intelligence technology. The method includes: encoding sample audio through a neural network model to obtain at least two first sample features, the i-th first sample feature represents the feature information of each audio segment obtained after segmenting the sample audio according to the i-th time scale; generating reconstructed audio based on at least two first sample features through a neural network model; training the neural network model through the sample audio and the reconstructed audio to obtain an audio generation model, and the audio generation model is used to generate a second audio based on the first audio. Since different first sample features reflect the information of the sample audio at different time scales, the representation ability of each first sample feature is stronger. Based on this, a reconstructed audio with higher accuracy can be generated according to each first sample feature, so that an audio generation model with higher accuracy can be trained based on the reconstructed audio.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a training method, device and electronic device for an audio generation model. Background Art

[0002] In recent years, artificial intelligence technology has developed rapidly at an astonishing speed and has gradually been widely used in all walks of life.

[0003] Among them, audio is an indispensable modality in applications and is widely adopted in many scenarios. Taking the audio transmission scenario as an example, the audio can be encoded at the source end through an audio generation model to obtain audio features. Then, the source end transmits the audio features to the destination end. After that, the audio is reconstructed at the destination end based on the audio features through the audio generation model.

[0004] In this process, we hope that the reconstructed audio is as similar to the original audio as possible. Based on this, the training method of the audio generation model is particularly important. Summary of the invention

[0005] The present application provides a training method, device and electronic device for an audio generation model, which can be used to train an audio generation model with higher accuracy, so as to generate high-quality audio based on the audio generation model. The technical solution includes the following contents.

[0006] On the one hand, a training method for an audio generation model is provided, the method comprising: obtaining sample audio and a neural network model to be trained; encoding the sample audio through the neural network model to obtain at least two first sample features; the at least two first sample features correspond to different time scales, and the i-th first sample feature among the at least two first sample features is used to characterize feature information of each audio segment obtained after segmenting the sample audio according to the i-th time scale, where i is a positive integer; generating reconstructed audio based on the at least two first sample features through the neural network model; training the neural network model through the sample audio and the reconstructed audio to obtain an audio generation model, wherein the audio generation model is used to generate a second audio based on the first audio.

[0007] On the other hand, a training device for an audio generation model is provided, the device comprising: an acquisition module for acquiring sample audio and a neural network model to be trained; an encoding module for encoding the sample audio through the neural network model to obtain at least two first sample features; the at least two first sample features correspond to different time scales, and the i-th first sample feature among the at least two first sample features is used to characterize feature information of each audio segment obtained after segmenting the sample audio according to the i-th time scale, where i is a positive integer; a generation module for generating reconstructed audio based on the at least two first sample features through the neural network model; and a training module for training the neural network model through the sample audio and the reconstructed audio to obtain an audio generation model, wherein the audio generation model is used to generate a second audio based on the first audio.

[0008] In one possible implementation, the neural network model includes a quantizer and a decoder; the generation module is used to quantize each first sample feature of the at least two first sample features through the quantizer to obtain each first quantized feature; fuse the each first quantized feature through the quantizer to obtain a reconstructed audio feature; and decode the reconstructed audio feature through the decoder to obtain a reconstructed audio.

[0009] In a possible implementation, the generation module is used to quantize the first input information through the quantizer to obtain the i-th first quantized feature; wherein, when the i is 1, the first input information includes the first first sample feature of the at least two first sample features; when the i is greater than 1, the first input information includes the i-th first sample feature and the i-1th residual feature, and the i-1th residual feature represents the difference between the i-1th first quantized feature and the i-1th first sample feature.

[0010] In a possible implementation, the number of the first quantized features is N, where N is a positive integer; the generation module is used to upsample the second input information through the quantizer to obtain the Ni-th upsampling result, and to fuse the Ni-th upsampling result and the i-th first quantized feature through the quantizer to obtain the Ni-th fusion result; wherein, when i is N-1, the second input information includes the N-th first quantized feature, and when i is any positive integer from 1 to N-2, the second input information includes the Ni-1-th fusion result, and the N-1-th fusion result is the reconstructed audio feature.

[0011] In one possible implementation, the training module is used to determine the i-th first loss through the i-th first sample feature and the i-th first quantization feature, and the i-th first loss is used to characterize the difference between the i-th first sample feature and the i-th first quantization feature; based on the sample audio, the reconstructed audio and each first loss, the neural network model is trained to obtain an audio generation model.

[0012] In one possible implementation, the training module is used to determine a second loss through the at least two first sample features, where the second loss characterizes the correlation between every two first sample features; and train the neural network model based on the sample audio, the reconstructed audio and the second loss to obtain an audio generation model.

[0013] In one possible implementation, the neural network model includes a first encoder and a second encoder; the encoding module is used to encode the sample audio through the first encoder to obtain reference audio features; perform convolution processing on the third input information through the second encoder to obtain the i-th convolution result, and determine the i-th first sample feature based on the i-th convolution result; wherein, when i is 1, the third input information includes the reference audio features, and when i is greater than 1, the third input information includes the i-1-th convolution result.

[0014] In a possible implementation, the encoding module is used to perform feature extraction on the i-th convolution result to obtain local features and global features, wherein the local features represent local information of each audio segment, and the global features represent global information of each audio segment; and fuse the local features and the global features to obtain the i-th first sample feature.

[0015] In one possible implementation, the encoding module is used to downsample the fourth input information through the first encoder to obtain the jth downsampling result; wherein, j is a positive integer, when j is 1, the fourth input information includes the sample audio, when j is greater than 1, the fourth input information includes the j-1th downsampling result, and the last downsampling result is used to determine the reference audio feature.

[0016] In one possible implementation, the generation module is used to generate a second sample feature based on the j-th downsampling result through the neural network model, wherein the second sample feature represents time-independent information in the sample audio; and generate reconstructed audio based on the second sample feature and each first sample feature through the neural network model.

[0017] In a possible implementation, the generation module is used to divide the second sample feature to obtain at least two second sample sub-features; quantize each of the at least two second sample sub-features to obtain each second quantized feature; and generate the reconstructed audio based on the each second quantized feature and each first sample feature through the neural network model.

[0018] In one possible implementation, the training module is used to determine at least one of a third loss or a fourth loss based on the sample audio and the reconstructed audio, the third loss characterizing the difference between the sample audio and the reconstructed audio in the time domain, and the fourth loss characterizing the difference between the sample audio and the reconstructed audio in the frequency domain; based on at least one of the third loss or the fourth loss, train the neural network model to obtain the audio generation model.

[0019] In a possible implementation, the acquisition module is also used to acquire the first audio; the encoding module is also used to encode the first audio through the audio generation model to obtain at least two first audio features, the at least two first audio features correspond to different time scales, and the i-th first audio feature of the at least two first audio features is used to characterize the feature information of each audio segment obtained after segmenting the first audio according to the i-th time scale, where i is a positive integer; the generation module is also used to generate the second audio based on the at least two first audio features through the audio generation model.

[0020] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above-mentioned training methods for the audio generation model.

[0021] On the other hand, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor so that an electronic device can implement any of the above-mentioned training methods for the audio generation model.

[0022] On the other hand, a computer program is also provided, wherein at least one computer program is loaded and executed by a processor so that an electronic device can implement any of the above-mentioned training methods for an audio generation model.

[0023] On the other hand, a computer program product is also provided, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor so that an electronic device can implement any of the above-mentioned training methods for the audio generation model.

[0024] The technical solution provided by this application brings at least the following beneficial effects.

[0025] The technical solution provided by the present application is to encode sample audio through a neural network model to obtain at least two first sample features. Since the i-th first sample feature represents the information of each audio segment obtained after the sample audio is segmented according to the i-th time scale, different first sample features can reflect the information of the sample audio at different time scales, so that the representation ability of each first sample feature is stronger and can accurately reflect the sample audio. Based on this, a reconstructed audio with higher accuracy can be generated according to each first sample feature, so that when the neural network model is trained based on the reconstructed audio and the sample audio, the training effect can be improved, and the accuracy of the audio generation model can be improved, so that high-quality audio can be generated based on the audio generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 It is a schematic diagram of a computer system provided in an embodiment of the present application.

[0028] Figure 2 It is a schematic diagram of an audio generation process provided in an embodiment of the present application.

[0029] Figure 3 It is a flowchart of a training method for an audio generation model provided in an embodiment of the present application.

[0030] Figure 4 It is a structural schematic diagram of an adapter provided in an embodiment of the present application.

[0031] Figure 5 It is a structural diagram of a feature extraction module provided in an embodiment of the present application.

[0032] Figure 6 This is a flow chart of speech synthesis provided in an embodiment of the present application.

[0033] Figure 7 It is a training diagram of an audio generation model provided in an embodiment of the present application.

[0034] Figure 8 This is a schematic diagram of a speech synthesis interface provided in an embodiment of the present application.

[0035] Fig. 9 It is a structural schematic diagram of a training device for an audio generation model provided in an embodiment of the present application.

[0036] Fig.10 It is a structural diagram of a terminal device provided in an embodiment of the present application.

[0037] Fig.11 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0039] It should be noted that the terms "first", "second", etc. in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0040] like Figure 1 As shown, Figure 1 1 is a schematic diagram of a computer system provided in an embodiment of the present application, and the computer system includes: a terminal device 101 and a server 102. Among them, a client is installed and running in the terminal device 101, and the server 102 is used to provide background services for the client. The object 103 can use the terminal device 101 to achieve instant communication, media information viewing, speech synthesis and other functions through the interaction between the client and the server 102.

[0041] In a possible implementation, the server 102 undertakes the main computing work, and the terminal device 101 undertakes the secondary computing work. Alternatively, the server 102 undertakes the secondary computing work, and the terminal device 101 undertakes the main computing work. Alternatively, the terminal device 101 and the server 102 use a distributed computing architecture to perform collaborative computing.

[0042] Optionally, the terminal device 101 may be any electronic device product that can perform human-computer interaction with a user through one or more methods such as a keyboard, a touchpad, a remote controller, voice interaction, or a handwriting device. For example, the terminal device 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a PC (Personal Computer), a mobile phone, a PDA (Personal Digital Assistant), a wearable device, a PPC (Pocket PC), a smart car machine, a smart TV, etc.

[0043] The terminal device 101 may generally refer to one of a plurality of terminal devices, and this embodiment is only illustrated by taking the terminal device 101 as an example. Those skilled in the art may know that the number of the terminal devices 101 may be more or less. For example, the terminal device 101 may be only one, or the terminal devices 101 may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the terminal devices 101.

[0044] The server 102 is a single server, or a server cluster consisting of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server 102 is directly or indirectly connected to the terminal device 101 by means of a wired network or a wireless network. The server 102 has a data receiving function, a data processing function, and a data sending function. Of course, the server 102 may also have other functions, which are not limited in the embodiments of the present application.

[0045] Those skilled in the art should understand that the above-mentioned terminal device 101 and server 102 are only for illustration, and other existing or future terminal devices or servers, if applicable to the present application, should also be included in the scope of protection of the present application and are included here by reference.

[0046] In a possible implementation, an audio generation model is trained by at least one of the terminal device 101 or the server 102 to generate new audio through the audio generation model. The audio generation model is obtained by training a neural network model, and based on this, the audio generation model and the neural network model are similar in structure and function, and the two differ only in model parameters.

[0047] In this example, if Figure 2As shown, the neural network model includes a first encoder 206, a second encoder 207, a quantizer 208 and a decoder 209. The first encoder 206 is used to receive audio and encode the audio into features that represent its information. It is a network that has the function of encoding audio. The second encoder 207 is connected in series after the first encoder 206, and is used to receive the audio features output by the first encoder 206, and further encode the audio features based on at least two time scales. It is a network that has the function of encoding audio features at different time scales. The quantizer 208 is connected in series after the second encoder 207, and is used to receive the features corresponding to each time scale output by the second encoder 207, quantize the features corresponding to each time scale and fuse them. It is a network that has functions such as quantization and fusion of features. The decoder 209 is connected in series after the quantizer 208, and is used to receive the fused features output by the quantizer 208, and decode the fused features to obtain audio. It is a network that has the function of decoding features into audio. As shown Figure 2 As shown, the process includes at least the following steps.

[0048] Step 1: Encode the sample audio to obtain reference audio features. Specifically, the sample audio 201 can be input into the first encoder 206, and the sample audio can be encoded by the first encoder 206 to obtain the reference audio features 202, and the sample audio can be characterized by the reference audio features 202. characterizes the reference audio features 202, and , where R represents a real number, N represents the number of audio frames included in the sample audio, and d represents the dimension of the features of each audio frame.

[0049] Step 2: Encode the reference audio features to obtain features at multiple time scales. Specifically, the reference audio features 202 may be input into the second encoder 207, and the reference audio features may be encoded by the second encoder 207 to obtain features 203 at multiple time scales. The features 203 at multiple time scales include first sample features at multiple time scales. For ease of description, different reference numbers are used to represent first sample features at different time scales. For example, reference numbers 203-1, 203-2, and 203-3 are used to represent first sample features at three time scales, respectively.

[0050] Optionally, use Characterizes the first sample feature 203-1, and ,use Characterizes the first sample feature 203-2, and ,use Characterize the first sample feature 203-3, and It can be seen that the first sample feature 203-1 includes The first sample feature 203-2 includes The first sample feature 203-3 includes As the number of audio frames decreases, the time scale increases.

[0051] In addition, a mutual information can be calculated based on the first sample feature 203-1 and the first sample feature 203-2, another mutual information can be calculated based on the first sample feature 203-1 and the first sample feature 203-3, and another mutual information can be calculated based on the first sample feature 203-3 and the first sample feature 203-2. These three mutual information are used to train the first encoder 206, the second encoder 207, the quantizer 208 and the decoder 209.

[0052] Step 3: quantize the features of each time scale and fuse them to obtain the reconstructed audio feature. That is, the first sample feature 203-1 can be quantized by the quantizer 208 to obtain a first quantized feature, the first sample feature 203-2 can be quantized by the quantizer 208 to obtain another first quantized feature, and the first sample feature 203-3 can be quantized by the quantizer 208 to obtain another first quantized feature. The reconstructed audio feature 204 is obtained by fusing the three first quantized features. Optionally, use characterize the reconstructed audio features 204, and .

[0053] Step 4: Decode the reconstructed audio features to obtain the reconstructed audio. Specifically, the reconstructed audio features 204 are input to the decoder 209, and the decoder 209 decodes the reconstructed audio features 204 to obtain the reconstructed audio 205. In addition, the L2 loss can be calculated based on the reconstructed audio 205 and the sample audio 201 to train the first encoder 206, the second encoder 207, the quantizer 208 and the decoder 209 through the L2 loss.

[0054] It should be noted that the above steps 1 to 4 can be performed multiple times to train the first encoder 206, the second encoder 207, the quantizer 208 and the decoder 209 multiple times, and finally train the audio generation model. The audio generation model can generate the second audio according to the first audio according to the implementation principle of the above steps 1 to 4, and the generation process will not be repeated here.

[0055] The present application embodiment provides a method for training an audio generation model, which can be applied to the above-mentioned computer system and executed by at least one of the terminal device 101 or the server 102. For ease of description, the terminal device 101 or the server 102 is collectively referred to as an electronic device, that is, the method of the present application embodiment is executed by an electronic device. Figure 3 As shown, the method includes the following steps.

[0056] Step 301, obtain sample audio and a neural network model to be trained.

[0057] The sample audio may be any audio, and optionally, the sample audio is the voice of a certain object. The electronic device may obtain the sample audio through an audio acquisition device, or the electronic device may obtain the sample audio through the Internet. The method for obtaining the sample audio is not limited here.

[0058] In addition, the electronic device can also obtain a neural network model to be trained. Optionally, the neural network model is a pre-trained model that has been preliminarily trained and has audio generation capabilities. The embodiment of the present application does not limit the structure, size, network layers included, etc. of the neural network model.

[0059] Step 302: encode the sample audio through a neural network model to obtain at least two first sample features; the at least two first sample features correspond to different time scales, and the i-th first sample feature among the at least two first sample features is used to characterize feature information of each audio segment obtained after segmenting the sample audio according to the i-th time scale, where i is a positive integer.

[0060] Time scale is a measurement method for audio segmentation. The larger the time scale, the longer the audio segment obtained by segmentation. For audio with a fixed duration, the longer the audio segment, the fewer the number of audio segments. For example, if there are three time scales, the first time scale is twice the second time scale, and the second time scale is twice the third time scale, then the duration of the audio segment corresponding to the first time scale is twice the duration of the audio segment corresponding to the second time scale, and the duration of the audio segment corresponding to the second time scale is twice the duration of the audio segment corresponding to the third time scale. For audio with a fixed duration, the number of audio segments corresponding to the first time scale is 0.5 times the number of audio segments corresponding to the second time scale, and the number of audio segments corresponding to the second time scale is 0.5 times the number of audio segments corresponding to the third time scale.

[0061] In this example, the sample audio is input into the neural network model, and N first sample features are determined by the neural network model, where N is a positive integer. The N first sample features correspond to N time scales, and the i-th (i is a positive integer less than or equal to N) first sample feature represents the information of each audio segment obtained after the sample audio is segmented according to the i-th time scale. The information of the audio segment includes the semantics, emotion, timbre, etc. represented by the audio segment. It can be understood that the structure of the neural network model is different, and the way of extracting the first sample features is also different.

[0062] In an exemplary embodiment, the neural network model includes a first encoder and a second encoder. Step 302 includes steps 3021 to 3022 (not shown in the figure).

[0063] Step 3021: Encode the sample audio through a first encoder to obtain reference audio features.

[0064] The first encoder is any type of encoder that can encode audio. For example, the first encoder can be an autoencoder or a variational autoencoder. Represents the sample audio, where R represents a real number and T represents the duration of the sample audio. Input the first encoder, and obtain the reference audio features representing the sample audio by encoding through the first encoder . Wherein, N represents the number of audio frames included in the sample audio, and d represents the dimension of the features of each audio frame.

[0065] Optionally, step 3021 includes: downsampling the fourth input information through the first encoder to obtain the jth downsampling result; wherein j is a positive integer, when j is 1, the fourth input information includes sample audio, when j is greater than 1, the fourth input information includes the j-1th downsampling result, and the last downsampling result is used to determine the reference audio features.

[0066] The first encoder includes at least one first encoding module. If there are at least two first encoding modules, the first encoding modules are connected in series, and the structures of any two first encoding modules can be the same or different. The embodiment of the present application does not limit the structure of the first encoding module, and any network with a downsampling function can be used as the first encoding module, and the function of the first encoding module is not limited to downsampling.

[0067] For example, the first encoder includes four first encoding modules connected in series, and each first encoding module includes three residual units and a downsampling layer. Among them, the residual unit includes a convolution layer, a normalization layer and an activation layer. The convolution layer is used to perform convolution processing on the input of the residual unit, and the normalization layer is used to perform normalization processing on the processing result of the convolution layer. The residual unit is implemented in the form of a skip layer connection, and the input of the residual unit can be directly added to the output of the normalization layer, and then activated by the activation layer to obtain the output of the residual unit. The residual unit can reduce information loss, improve the representation ability of features, and thus improve model performance. The downsampling layer is used to downsample the input.

[0068] In this example, the first first encoding module is used to downsample the sample audio to obtain the first downsampling result. The second first encoding module is used to downsample the first downsampling result to obtain the second downsampling result. The third first encoding module is used to downsample the second downsampling result to obtain the third downsampling result. And so on, until the last downsampling result is obtained, so as to determine the reference audio feature based on the last downsampling result. By continuously downsampling the sample audio, the information of the sample audio is continuously extracted, so that the reference audio feature can accurately characterize the sample audio, so as to facilitate the subsequent generation of reconstructed audio based on the sample audio.

[0069] In actual application, network layers such as convolutional layers and normalization layers may be connected in series before or after any first encoding module, which is not limited here.

[0070] For example, the first encoder includes four first encoding modules, a one-dimensional convolution layer is connected in series before the first first encoding module, and a one-dimensional convolution layer is connected in series after the last first encoding module, and the convolution layer is used to perform convolution processing. Based on this, the electronic device inputs the sample audio into the first encoder, and first, performs convolution processing on the sample audio through a one-dimensional convolution layer to obtain a convolution processing result. Then, the convolution processing result is gradually downsampled through the four first encoding modules to obtain the last downsampling result. Then, the last downsampling result is convoluted through a one-dimensional convolution layer to obtain a reference audio feature. The downsampling factor of each first encoding module (i.e., the step size for performing downsampling) is not limited here. Exemplarily, the downsampling factors of the four first encoding modules are 2, 4, 5, and 8, respectively, and the total downsampling factor satisfies D=2×4×5×8=320.

[0071] Step 3022: Perform convolution processing on the third input information through the second encoder to obtain the i-th convolution result, and determine the i-th first sample feature based on the i-th convolution result. When i is 1, the third input information includes the reference audio feature, and when i is greater than 1, the third input information includes the i-1-th convolution result.

[0072] The second encoder is a hierarchical multi-scale encoder. The hierarchical multi-scale encoder is used to extract multi-scale features of audio features to accurately capture the information density differences of audio features at different time scales and improve the characterization capabilities of each first sample feature. The second encoder includes at least two second encoding modules, and the structures of any two second encoding modules can be the same or different. In practical applications, any network with a convolution function can be used as a second encoding module, that is, the second encoding module includes a convolution layer for performing convolution processing, and the output of the convolution layer can be regarded as the i-th convolution result. However, the function of the second encoding module may not be limited to convolution. For example, the second encoding module includes three residual units connected in series and a downsampling layer, and each residual unit includes a convolution layer. The residual unit performs convolution processing and normalization processing on the input, and activates the input and the normalization processing result after adding them together to obtain the output of the residual unit. The downsampling layer is used to perform downsampling on the input (that is, the output of the last residual unit), and the output of the downsampling layer can be regarded as the i-th convolution result.

[0073] In this example, the first second encoding module is used to perform convolution processing on the reference audio feature to obtain the first convolution result. Optionally, the first convolution result is used as the first first sample feature, or the first convolution result is further processed to obtain the first first sample feature. The second second encoding module is used to perform convolution processing on the first convolution result to obtain the second convolution result. Optionally, the second convolution result is used as the second first sample feature, or the second convolution result is further processed to obtain the second first sample feature. The third second encoding module is used to perform convolution processing on the second convolution result to obtain the third convolution result. Optionally, the third convolution result is used as the third first sample feature, or the third convolution result is further processed to obtain the third first sample feature. And so on, until the last first sample feature is obtained.

[0074] It can be seen from the above that the embodiment of the present application is to gradually perform convolution processing on the reference audio features. Through the convolution processing, the time scale is continuously expanded. Exemplarily, the second encoder includes three second encoding modules, and the convolution scales of these three second encoding modules are 2, 2, and 2 respectively. After each convolution processing, the time scale is doubled. Based on this, the time scale corresponding to the first first sample feature is twice the time scale corresponding to the second first sample feature, and the time scale corresponding to the second first sample feature is twice the time scale corresponding to the third first sample feature.

[0075] Optionally, the first sample feature , the second first sample feature , the third first sample feature . Where R represents a real number, The frame numbers of the audio frames corresponding to the three first sample features are represented in sequence, and d represents the feature dimension of each audio frame.

[0076] Since the number of audio frames corresponding to the three first sample features is halved successively, the time scales corresponding to the three first sample features are doubled successively. Characterizes the information of the sample audio captured at the finest time scale, the second first sample feature Characterizes the information of the sample audio captured at the intermediate time scale, the third first sample feature Characterizes the information of the sample audio captured at the coarsest time scale.

[0077] The second encoder determines each first sample feature based on the reference audio feature that characterizes the sample audio, and extracts information of the sample audio at different time scales, making each first sample feature more capable of characterizing and accurately reflecting the sample audio. Moreover, features at different time scales enhance feature diversity, which is conducive to improving the generalization ability of the audio generation model, so that the audio generation model has high performance in a variety of bit rate scenarios and can generate high-quality audio.

[0078] In actual application, network layers such as a normalization layer and a feedforward layer may be connected in series before or after any second encoding module, which is not limited here.

[0079] In an exemplary embodiment, based on the ith convolution result, the ith first sample feature is determined, including: performing feature extraction on the ith convolution result to obtain local features and global features, the local features representing local information of each audio segment, and the global features representing global information of each audio segment; fusing the local features and the global features to obtain the ith first sample feature.

[0080] In this example, the i-th second encoding module also includes an adapter, and the adapter can be connected in series after the convolution layer. The adapter includes a feature extraction module, and the feature extraction module is used to perform feature extraction on the i-th convolution result to obtain local features and global features. The embodiment of the present application does not limit the structure of the adapter. In the actual application process, the adapter can be flexibly built according to the application scenario.

[0081] Optionally, the adapter includes at least one Conformer layer, for example, the adapter includes two Conformer layers connected in series. For ease of description, the adapter including one Conformer layer is taken as an example to illustrate the structure and function of the adapter.

[0082] like Figure 4As shown, the adapter includes a backbone network, a projection layer and at least one feature extraction module, and the feature extraction module includes a Transformer block (Trans block for short) and a convolution block, and the Trans block and the convolution block are connected by a bridge unit. The Trans block is used to extract global features, and the convolution block is used to extract local features. For ease of description, the kth feature extraction module is identified by the reference numeral 40k (k is a positive integer). For example, the first feature extraction module is identified by the reference numeral 401, and the feature extraction module 401 includes a Trans block 401-1 and a convolution block 401-2 connected by a bridge unit; the second feature extraction module is identified by the reference numeral 402, and the feature extraction module 402 includes a Trans block 402-1 and a convolution block 402-2 connected by a bridge unit; and so on.

[0083] Among them, the backbone network is used to preliminarily process the i-th convolution result. The structure of the backbone network is different, and the processing method is also different. For example, the backbone network includes a convolution layer with a scale of 7×7 and a step size of 2. The convolution layer is used to perform convolution processing on the i-th convolution result. After the convolution layer, a maximum pooling layer with a scale of 3×3 is connected in series. The maximum pooling layer is used to perform maximum pooling processing on the convolution result of the convolution layer. The processing result of the maximum pooling layer is used to determine the processing result of the backbone network. Then, on the one hand, the processing result of the backbone network is used to input the convolution block 401-2, and on the other hand, the projection mapping is performed on the processing result of the backbone network through the projection layer, and the projection mapping result is input into the Trans block 401-1. Then, on the one hand, the output of the Trans block 401-1 is used to input the Trans block 402-1, and on the other hand, the output of the convolution block 401-2 is used to input the convolution block 402-2, and so on.

[0084] The structure of any feature extraction module is as follows Figure 5 The feature extraction module includes a convolution block 501, a Trans block 502 and a bridge unit 503, and their functions are described in turn below.

[0085] The feature 504 may be input into the convolution block 501, and the feature 504 includes the processing result of the backbone network or the output of the previous convolution block. The convolution block 501 includes 1×1Conv-BN (i.e., a convolution layer and a batch normalization layer with a scale of 1×1), 3×3Conv-BN (i.e., a convolution layer and a batch normalization layer with a scale of 3×3), and 1×1Conv-BN. Among them, the convolution layer is used to perform convolution processing, and the batch normalization layer is used to perform batch normalization processing. After the feature 504 is subjected to multiple convolution processing and batch normalization processing, the processing result and the feature 504 are added to obtain the feature 505.

[0086] Among them, the bridge unit 503 includes a downsampling unit 503-1, and the output features of the first 3×3Conv-BN in the convolution block 501 can be input to the downsampling unit 503-1. The downsampling unit 503-1 includes a 1×1Conv, a downsampling layer and an LN (i.e., a layer normalization layer). The downsampling layer is used to perform an average pooling operation and a reshaping operation on the features, and the LN is used to perform a layer normalization process on the features. Feature 507 and the output features of the downsampling unit 503-1 are added and input to the Trans block 502. Feature 507 includes a projection mapping result or an output feature of the previous Trans block. The Trans block 502 includes an LN and a multi-head self-attention layer, and the multi-head self-attention layer is used to perform a multi-head self-attention process. After the input features of the Trans block 502 are processed by layer normalization and multi-head self-attention, the processing results are added to the input features of the Trans block 502 to obtain an addition result. The Trans block 502 also includes an LN and an MLP (i.e., a multi-layer perceptron), and the MLP is used to perform a mapping operation. After the addition result is subjected to layer normalization and mapping operation, the operation result and the addition result are added to obtain feature 508. Feature 508 can be used as a global feature of the embodiment of the present application, or can be input to the next Trans block.

[0087] The bridge unit 503 also includes an upsampling unit 503-2, and the feature 508 can be input into the upsampling unit 503-2. The upsampling unit 503-2 includes an upsampling layer, a 1×1Conv, and a BN. The upsampling layer is used to perform reshaping operations and interpolation operations. After the feature 508 is reshaped, interpolated, convolved, and batch normalized, the output feature of the upsampling unit 503-2 is obtained. The convolution block 501 also includes a 1×1Conv-BN. After the feature 505 is convolved and batch normalized, the processing result is added to the output feature of the upsampling unit 503-2 to obtain an addition result. The convolution block 501 also includes a 3×3Conv-BN and a 1×1Conv-BN. After the addition result is subjected to multiple convolutions and batch normalization, the processing result is added to the feature 505 to obtain a feature 506. The feature 506 can be used as a local feature of an embodiment of the present application, or it can be input to the next convolution block.

[0088] Based on the above content, it can be seen that the adapter includes a backbone network, a projection layer and at least one feature extraction module, and the feature extraction module includes a Trans block and a convolution block. The backbone network is used to process the i-th convolution result, and the projection layer is used to perform projection mapping on the processing result of the backbone network. The k-th Trans block is used to perform global feature extraction on the fifth input information. Wherein, when k is 1, the fifth input information includes the projection mapping result of the projection layer, when k is a positive integer greater than 1, the fifth input information includes the output features of the k-1th Trans block, and the output features of the last Trans block are global features. The k-th convolution block is used to perform local feature extraction on the sixth input information. Wherein, when k is 1, the sixth input information includes the processing result of the backbone network, when k is a positive integer greater than 1, the sixth input information includes the output features of the k-1th convolution block, and the output features of the last convolution block are local features.

[0089] The local features represent the local information of each audio segment. The local information of the audio segment includes at least one information such as fundamental frequency, timbre, pitch, energy, etc., which is used to characterize the basic properties and changing rules of the sound. The global features represent the global information of each audio segment. The global information of the audio segment includes at least one information such as energy, zero-crossing rate, mean, variance, etc., which is used to characterize the overall properties and statistical characteristics of the sound. The adapter also includes a fusion network, which fuses the local features and the global features through the fusion network to obtain the i-th first sample feature. The structure of the fusion network is not limited here. By way of example, the fusion network includes a splicing layer or MLP, etc.

[0090] By fusing the local features and global features of the i-th convolution result, the local information and global information of the audio segment are fused, ensuring the information integrity of the audio segment, so that the i-th first sample feature can accurately characterize each audio segment, has a strong characterization ability, and is conducive to improving the training effect of the model.

[0091] Step 303: Generate reconstructed audio based on at least two first sample features through a neural network model.

[0092] Since at least two first sample features can accurately characterize the sample audio, the neural network model can decode each first sample feature to reconstruct the sample audio and obtain the reconstructed audio. It can be understood that the structure of the neural network model is different and the decoding method is also different. A possible implementation method is shown below. In this example, the neural network model includes a quantizer and a decoder. Step 303 includes steps 3031 to 3033 (not shown in the figure).

[0093] Step 3031: quantize each of the at least two first sample features by a quantizer to obtain each first quantized feature.

[0094] The present application embodiment does not limit the structure of the quantizer, and any network with a quantization function can be used as a quantizer. The quantizer quantizes each first sample feature to obtain a first quantized feature. For example, the quantizer quantizes the first sample feature , get the first quantitative feature , quantize the first sample feature through the quantizer , get the first quantitative feature , quantize the first sample feature through the quantizer , get the first quantitative feature By quantizing the first sample feature, the amount of data is compressed, which is beneficial to increasing the speed of subsequent decoding and improving the efficiency of audio generation.

[0095] In an exemplary embodiment, step 3031 includes: quantizing the first input information through a quantizer to obtain the i-th first quantized feature. When i is 1, the first input information includes the first first sample feature of at least two first sample features; when i is greater than 1, the first input information includes the i-th first sample feature and the i-1th residual feature, and the i-1th residual feature represents the difference between the i-1th first quantized feature and the i-1th first sample feature.

[0096] In this example, the quantizer includes a first quantization module, and the first quantization module is used to quantize the first first sample feature to obtain the first first quantized feature. The quantizer also includes a second quantization module and a first downsampling module. The first downsampling module is used to calculate the difference between the first first quantized feature and the first first sample feature, and the first downsampling module includes an average pooling layer, and the average pooling operation is performed on the difference through the average pooling layer to obtain the first residual feature. The first residual feature and the second first sample feature are added and input into the second quantization module, and quantized through the second quantization module to obtain the second first quantized feature. The quantizer may also include a third quantization module and a second downsampling module. The second downsampling module is used to calculate the difference between the second first quantized feature and the second first sample feature, and the second downsampling module includes an average pooling layer, and the average pooling operation is performed on the difference through the average pooling layer to obtain the second residual feature. The second residual feature and the third first sample feature are added and input into the third quantization module, and quantized by the third quantization module to obtain the third first quantized feature. And so on, until the last first quantized feature is obtained. Among them, the downsampling module is used to calculate the feature difference between the first quantized feature and the first sample feature, and align the dimension of the feature difference with the dimension of the next first sample feature, so as to facilitate subsequent quantization by the quantizer.

[0097] By directly quantizing the first sample feature, the data volume of the feature is reduced, and the speed of subsequent processing of the first quantized feature is improved. Subsequently, in addition to quantizing the first sample feature of the current time, the residual features before and after the previous quantization are also quantized to minimize the information loss caused by feature quantization. Under the premise of reducing information loss, the data volume of the feature is reduced, so that the first quantized feature has a stronger representation ability and can improve the speed of subsequent processing of the first quantized feature. The aforementioned characteristics of the first quantized feature are conducive to improving the generation efficiency and accuracy of the reconstructed audio, thereby improving the training effect of the model.

[0098] Step 3032: The first quantized features are fused through a quantizer to obtain a reconstructed audio feature.

[0099] The embodiment of the present application does not limit the fusion method of the first quantitative features. For example, the reconstructed audio feature can be obtained by splicing various first quantitative features.

[0100] In an exemplary embodiment, the number of first quantized features is N, where N is a positive integer. Step 3032 includes: upsampling the second input information through a quantizer to obtain the Ni-th upsampling result, fusing the Ni-th upsampling result and the i-th first quantized feature through a quantizer to obtain the Ni-th fusion result. Wherein, when i is N-1, the second input information includes the N-th first quantized feature, and when i is any positive integer from 1 to N-2, the second input information includes the Ni-1-th fusion result, and the N-1-th fusion result is the reconstructed audio feature.

[0101] In this example, the quantizer includes a first upsampling module, through which the Nth first quantized feature is upsampled to obtain a first upsampling result, and the first upsampling result is added to the N-1th first quantized feature to obtain a first fusion result. Similarly, the quantizer may include a second upsampling module, through which the first fusion result is upsampled to obtain a second upsampling result, and the second upsampling result is added to the N-2th first quantized feature to obtain a second fusion result. The quantizer may also include a third upsampling module, through which the second fusion result is upsampled to obtain a third upsampling result, and the third upsampling result is added to the N-3th first quantized feature to obtain a third fusion result. And so on, until the N-1th fusion result is obtained, and the N-1th fusion result is the reconstructed audio feature. The upsampling module is used to align the dimension of the fusion result or the dimension of the Nth first quantized feature with the dimension of the previous first quantized feature to facilitate subsequent fusion.

[0102] By upsampling the second input information and fusing the upsampling result with the first quantized feature, the feature dimension is aligned and the features are fused, so that the reconstructed audio feature can accurately characterize the information of the sample audio to reconstruct the sample audio and improve the accuracy of the reconstructed audio.

[0103] Step 3033: decode the reconstructed audio features through a decoder to obtain reconstructed audio.

[0104] The present application embodiment does not limit the structure of the decoder, and any network with decoding function can be used as a decoder. Characterize and reconstruct audio features. Input decoder, decoded by decoder to get reconstructed audio . Where R represents a real number, represents the number of audio frames included in the reconstructed audio, d represents the dimension of the audio frame, and T represents the reconstructed audio It should be noted that the decoding methods are different due to different decoder structures, which will not be described here.

[0105] By quantizing each first sample feature to obtain each first quantized feature, and fusing each first quantized feature to obtain a reconstructed audio feature, the amount of data of the reconstructed audio feature is reduced, and the speed of subsequent decoding of the reconstructed audio feature is increased, thereby improving the generation efficiency of the reconstructed audio. In addition, the representation ability of the reconstructed audio feature can be improved, thereby improving the accuracy of the reconstructed audio, and further improving the training effect of the model.

[0106] In another possible implementation, step 303 includes steps 3034 to 3035 (not shown in the figure).

[0107] Step 3034: Generate a second sample feature based on the j-th downsampling result through a neural network model, where the second sample feature represents time-independent information in the sample audio.

[0108] The neural network model also includes a time-invariant representation extraction (TIRE) module. As mentioned above, the first encoder includes at least one first encoding module, and the jth first encoding module is used to determine the jth downsampling result. The jth downsampling result can be input into the TIRE module, for example, the second downsampling result can be input into the TIRE module. The TIRE module includes a convolution layer and an activation layer, the convolution layer is used to perform convolution processing, and the activation layer is used to perform activation operations. The TIRE module can perform convolution processing and activation operations on the jth downsampling result multiple times, and perform time-based average calculation on the execution results to obtain a time-invariant feature, which is a feature that is independent of time. The TIRE module obtains a second sample feature by performing a mapping operation and an activation operation on the time-invariant feature, and the second sample feature is also a feature that is independent of time.

[0109] In the sample audio, some information is stable and will not change over time. For example, the spectrum of the sample audio, the timbre of the object to which it belongs, and other information will not change over time. The second sample feature is extracted by the TIRE module, so that the second sample feature can represent the time-independent information in the sample audio, for example, the second sample feature can represent the timbre, spectrum, and other information. You can use represents the second sample feature, where R represents a real number and d represents the dimension of the second sample feature.

[0110] Step 3035: Generate reconstructed audio based on the second sample features and each first sample feature through a neural network model.

[0111] According to the implementation contents of step 3031 to step 3032, the reconstructed audio features can be determined based on each first sample feature. , the audio features will be reconstructed The second sample feature m is input into the decoder, and the reconstructed audio is obtained by decoding the decoder. The decoding process is not described here. The reconstructed audio is obtained by decoding the second sample feature and each first sample feature, and the sample audio is reconstructed according to the time-independent information in the sample audio and the information of different time scales in the sample audio. Since the information used in the reconstruction is more comprehensive, the accuracy of the reconstructed audio can be improved, thereby improving the model training effect.

[0112] In an exemplary embodiment, step 3035 includes: dividing the second sample feature to obtain at least two second sample sub-features; quantizing each of the at least two second sample sub-features to obtain each second quantized feature; and generating reconstructed audio based on each second quantized feature and each first sample feature through a neural network model.

[0113] The neural network model also includes a grouping quantizer. First, the second sample feature is divided into at least two second sample sub-features by the grouping quantizer. For example, the second sample feature is divided into 8 second sample sub-features. The division method is not limited here. Then, each second sample sub-feature is quantized by the grouping quantizer to obtain a second quantized feature. The fused feature is obtained by fusing the second quantized features. represents the fused features, where R represents a real number and d represents the dimension of the fused features.

[0114] According to the implementation contents of step 3031 to step 3032, the reconstructed audio features can be determined based on each first sample feature. , the audio features will be reconstructed and fusion features Input decoder, decoded by decoder to get reconstructed audio By dividing the second sample features, the second sample features at the segment level are converted into second sample sub-features at the frame level, so as to align each second sample sub-feature with the first sample feature at the frame level, which is convenient for subsequent decoding. By quantizing the second sample sub-features, the data volume of the features is reduced, thereby improving the speed of subsequent decoding.

[0115] The decoder structures are different and the decoding methods are also different. A possible implementation method is shown below.

[0116] The decoder includes at least one decoding module. If there are at least two decoding modules, the decoding modules are connected in series, and the structures of any two decoding modules can be the same or different. The embodiment of the present application does not limit the structure of the decoding module, and any network with an upsampling function can be used as a decoding module, and the function of the decoding module is not limited to upsampling.

[0117] For example, the decoder includes four decoding modules connected in series, each of which includes three residual units and an upsampling layer. The residual unit includes a transposed convolution layer (also called a deconvolution layer), a normalization layer, and an activation layer. The transposed convolution layer is used to perform a transposed convolution process on the input of the residual unit, and the normalization layer is used to perform a normalization process on the processing result of the transposed convolution layer. The residual unit is implemented in the form of a skip layer connection, and the input of the residual unit can be directly added to the output of the normalization layer, and then activated by the activation layer to obtain the output of the residual unit. The upsampling layer is used to upsample the input.

[0118] In this example, the fusion feature Any decoding module can be input. The first decoding module is used to reconstruct the audio features. Upsample to get the first upsampled result. If the fusion feature Input the first decoding module, then the first decoding module will reconstruct the audio features and fusion features Add, upsample the added result, and obtain the first upsampled result. The second decoding module is used to upsample the first upsampled result to obtain the second upsampled result. Input the second decoding module, then the second decoding module will combine the first upsampling result and the fusion feature The third decoding module is used to upsample the second upsampled result to obtain the third upsampled result. Input the third decoding module, then the third decoding module combines the second upsampling result and the fusion feature The sum is added, and the sum is upsampled to obtain the third upsampled result. This process is repeated until the last upsampled result is obtained, so as to determine the reconstructed audio based on the last upsampled result.

[0119] In actual applications, network layers such as convolutional layers and normalization layers can be connected in series before or after any decoding module, which is not limited here.

[0120] For example, the decoder includes four decoding modules, and two one-dimensional convolutional layers are connected in series before the first decoding module. The convolutional layer is used to perform convolution processing. Based on this, the electronic device will reconstruct the audio features and fusion features Input decoder, first, reconstruct the audio features through a one-dimensional convolution layer Perform convolution processing to obtain the convolution processing result. Then, the convolution processing result and the fusion feature are gradually processed through four decoding modules. Upsampling is performed to obtain the final upsampling result. The upsampling factor of each decoding module (i.e., the step size of upsampling) is not limited here. For example, the upsampling factors of the four decoding modules are 8, 5, 4, and 2, respectively, and the total downsampling factor satisfies D=2×4×5×8=320.

[0121] Optionally, the structures of the decoder and the first encoder are symmetrical, and the processing performed by the decoder is the inverse operation of the processing performed by the first encoder. For example, the first encoder includes a Transformer encoding module, and the decoder includes a Transformer decoding module. Based on this, the first encoder can encode the sample audio to obtain the reference audio features and the jth downsampling result, and the decoder can decode the reconstructed audio features and the jth downsampling result to obtain the reconstructed audio.

[0122] Step 304: Train a neural network model using sample audio and reconstructed audio to obtain an audio generation model, where the audio generation model is used to generate a second audio based on the first audio.

[0123] In an embodiment of the present application, the total loss of the neural network model can be determined by the sample audio and the reconstructed audio, and the neural network model can be trained once by the total loss. In actual application, steps 301 to 304 can be performed multiple times, that is, the neural network model can be trained multiple times according to the implementation principle of the embodiment of the present application, and finally an audio generation model is obtained. The structure and function of the audio generation model and the neural network model are similar, and the two only differ in the model parameters. The number of training times of the neural network model is not limited here. For example, the number of training times can be set according to manual experience, or the number of training times is the number of times that the performance of the model meets the requirements. It can be understood that there are many ways to implement step 304, and several possible implementations are shown below.

[0124] In implementation method A, step 304 includes: determining the i-th first loss through the i-th first sample feature and the i-th first quantization feature, the i-th first loss being used to characterize the difference between the i-th first sample feature and the i-th first quantization feature; training a neural network model based on the sample audio, the reconstructed audio and each first loss to obtain an audio generation model.

[0125] The i-th first loss can be calculated based on the i-th first quantitative feature and the i-th first sample feature according to the first loss function. The type of the first loss function is not limited here. Exemplarily, the first loss function is an L1 loss function or an L2 loss function. Among them, the L1 loss function is also called the Mean Absolute Error (MAE) function, and the L2 loss function is also called the Mean Squared Error (MSE) function. Since there are at least two first sample features, each first sample feature corresponds to a first quantitative feature, and each first sample feature and the corresponding first quantitative feature can determine a first loss, so the first loss is also at least two.

[0126] In addition, at least one of the third loss or the fourth loss can be calculated based on the sample audio and the reconstructed audio according to the implementation method C described below. At least one of the third loss or the fourth loss and each first loss are weighted to obtain a total loss, and the neural network model is trained by the total loss to obtain an audio generation model.

[0127] By training the model with sample audio and reconstructed audio, the model can be optimized in the direction of making the reconstructed audio close to the sample audio, thereby improving the accuracy of the audio generation model. Furthermore, by training the model with the first loss, the model can be optimized in the direction of making the first quantized feature close to the first sample feature, thereby reducing information loss when the model quantizes features, improving the accuracy of the quantized features, and thus improving the accuracy of the audio generated by the model.

[0128] In implementation method B, step 304 includes: determining a second loss through at least two first sample features, the second loss characterizing the correlation between every two first sample features; training a neural network model based on the sample audio, the reconstructed audio and the second loss to obtain an audio generation model.

[0129] The correlation between each two first sample features may be determined according to a correlation function. The type of the correlation function is not limited herein. Exemplarily, the correlation function is a cosine similarity function, a mutual information function, a KL divergence function, etc. Optionally, the mutual information function is a contrast log-ratio upper bound (CLUB) function or a variational contrast log-ratio upper bound (VCLUB) function. The two functions are introduced in turn below.

[0130] When variables u are independent, variables v are independent, and the conditional distribution is known , the CLUB function is defined as: .in, Characterizes the probability of generating the distribution of variable u based on the distribution of variable v. Represents the expected mutual information between variables v and u. log is the logarithmic symbol and E is the mean symbol. represents the joint distribution of variables u and v, Characterizes the marginal distribution of variable u, Characterizes the marginal distribution of variable v. In practical applications, variables u and v can be independent of each other or have a correlation. Based on this, yes The upper bound of , Characterizes the actual mutual information between the variable v and the variable u. When the mutual information function is a CLUB function, the variable u and the variable v are two different first sample features, and the mutual information between the two first sample features can be calculated according to the above CLUB function.

[0131] The VCLUB function is a variational form of the CLUB function. When is unknown, a neural network is used to determine the conditional distribution. Assume that the conditional distribution determined by the neural network is: , then the VCLUB function can be expressed as: .in, Represents the probability of generating the distribution of variable u based on the distribution of variable v through a neural network, that is, yes The variational approximation of The network parameters that characterize the neural network, Characterizes the mutual information between variable v and variable u. The meanings of other parameters in the VCLUB function can be found in the description of the CLUB function and will not be repeated here. Optionally, When the mutual information function is a VCLUB function, the variable u and the variable v are two different first sample features, and the mutual information between the two first sample features can be calculated according to the VCLUB function.

[0132] Optionally, there are at least two sample audios. When calculating the mutual information between the two first sample features, the two first sample features may correspond to the same sample audio or to different sample audios, and the two first sample features correspond to different time scales. Assume that the two first sample features correspond to the time scales , then the two first sample features corresponding to the same sample audio at the two time scales constitute a positive sample pair, and the two first sample features corresponding to different sample audio at the two time scales constitute a negative sample pair. Based on each positive sample pair and each negative sample pair, the mutual information between the first sample features corresponding to the two time scales is calculated.

[0133] Optionally, since different first sample features correspond to different time scales, different first sample features correspond to different numbers of audio frames. Based on this, when calculating the mutual information between two first sample features, one of the first sample features can be expanded or compressed to align with the other first sample feature, thereby facilitating the calculation of the mutual information. For example, the first sample feature The audio frame number is the first sample feature Twice the number of audio frames, the first sample feature The audio frame number is the first sample feature Based on this, Repeat the audio frame twice. Repeat the audio frame four times so that The audio frames, Audio frame alignment audio frames, so as to facilitate the subsequent calculation of the mutual information between every two first sample features.

[0134] Based on this, we can follow , calculation time scale The corresponding feature (i.e., a first sample feature) and the time scale The mutual information between the corresponding features (i.e., another first sample feature). Among them, Characterization time scale The corresponding features, Characterization time scale The corresponding features, Characterization and The mutual information between them. Characterizes the number of sample audio, Characterizes the number of audio frame features in the first sample feature after alignment. Characterize the first sample feature corresponding to the k-th sample audio The i-th audio frame feature in, Characterize the first sample feature corresponding to the k-th sample audio The i-th audio frame feature in, Characterization l The first sample feature corresponding to the sample audio The i-th audio frame feature in . Among them, and constitute a positive sample pair, and and Form negative sample pairs. Characterizes the conditional distribution of positive sample pairs determined by the neural network, Characterize the conditional distribution of negative sample pairs determined by the neural network, Characterize the network parameters of the neural network.

[0135] According to the above principle, the mutual information between each two first sample features can be determined. Taking three time scales as an example, each sample audio corresponds to three first sample features, and these three first sample features are respectively According to the above implementation principle, the first sample feature can be determined With the first sample characteristics Mutual information between , the first sample characteristics With the first sample characteristics Mutual information between , and the first sample feature With the first sample characteristics Mutual information between .

[0136] In this example, the mutual information between each two first sample features can be summed or averaged to obtain the second loss For example, the second loss In addition, according to implementation C, at least one of the third loss or the fourth loss can be calculated based on the sample audio and the reconstructed audio. At least one of the third loss or the fourth loss and the second loss are weighted to obtain a total loss, and the neural network model is trained by the total loss to obtain an audio generation model.

[0137] By training the model with sample audio and reconstructed audio, the model can be optimized in the direction of making the reconstructed audio close to the sample audio, thereby improving the accuracy of the audio generation model. Furthermore, by training the model with the second loss, the model can be optimized in the direction of reducing the correlation between the two first sample features, thereby making the correlation between the features of different time scales extracted by the audio generation model lower, so that the features of different time scales can independently, accurately, and fully characterize the audio. In other words, features of different time scales can characterize information of audio in different aspects, reduce the redundancy of features of each time scale, improve the effectiveness of coding and the characterization ability of features, and thus improve the quality of generated audio.

[0138] In implementation method C, step 304 includes: based on the sample audio and the reconstructed audio, determining at least one of a third loss or a fourth loss, the third loss characterizing the difference between the sample audio and the reconstructed audio in the time domain, and the fourth loss characterizing the difference between the sample audio and the reconstructed audio in the frequency domain; based on at least one of the third loss or the fourth loss, training a neural network model to obtain an audio generation model.

[0139] The third loss function may be used to calculate the third loss based on the waveform of the sample audio and the waveform of the reconstructed audio. The type of the third loss function is not limited herein, and illustratively, the third loss function is an L1 loss function or an L2 loss function.

[0140] And / or, a fourth loss function may be used to calculate a fourth loss based on the spectrum of the sample audio and the spectrum of the reconstructed audio. The type of the fourth loss function is also not limited herein. Exemplarily, the fourth loss function is an L1 loss function or an L2 loss function.

[0141] Based on at least one of the third loss or the fourth loss, a total loss is determined. For example, the third loss and the fourth loss are weighted to obtain a total loss. The neural network model is trained by the total loss to obtain an audio generation model. By training the model by the third loss and the fourth loss, the model can be optimized in the direction of making the reconstructed audio close to the sample audio in both the time domain and the frequency domain, thereby improving the accuracy of the audio generation model.

[0142] It is understandable that the total loss includes but is not limited to the losses mentioned above. For example, the total loss also includes at least one of the fifth loss or the sixth loss determined in the following manner: the sample audio is input into the discriminator, the features of the sample audio are extracted by the discriminator, and a first discrimination result is determined based on the features of the sample audio, and the first discrimination result represents the probability that the sample audio is real data. Similarly, the reconstructed audio is input into the discriminator, the features of the reconstructed audio are extracted by the discriminator, and a second discrimination result is determined based on the features of the reconstructed audio, and the second discrimination result represents the probability that the reconstructed audio is real data. Among them, the real data refers to the original data such as the sample audio, and the non-real data (i.e., false data) refers to the generated data such as the reconstructed audio. The fifth loss is the loss determined according to the fifth loss function based on the features of the sample audio and the features of the reconstructed audio, which is used to characterize the difference between the two features. Optionally, the fifth loss function is an L2 loss. The sixth loss is the loss determined according to the sixth loss function based on the second discrimination result, which is used to characterize the difference between the second discrimination result and the specified data. Optionally, the sixth loss is determined according to the following formula: ,in, represents the sixth loss, K represents the number of sample audios in a batch, Represents the reconstructed audio corresponding to the k-th sample audio The second discrimination result.

[0143] In addition, the total loss is determined based on at least one of the third loss or the fourth loss, and can also be determined in combination with at least one of the first loss, the second loss, the fifth loss, the sixth loss, etc. For example, the total loss satisfies: . Where L represents the total loss. Characterize the third loss, Characterize the weight of the third loss. Characterize the fourth loss, Characterize the weight of the fourth loss. Characterizing the sixth loss, Characterize the weight of the sixth loss. Characterize the fifth loss, Characterize the weight of the fifth loss. Characterize the first loss, Characterize the weight of the first loss. Characterize the second loss, The weight representing the second loss. The neural network model is trained by the total loss to obtain an audio generation model, and subsequently, the audio generation model can be used to generate the second audio based on the first audio.

[0144] That is to say, the method of the embodiment of the present application also includes: obtaining a first audio; encoding the first audio through an audio generation model to obtain at least two first audio features, the at least two first audio features corresponding to different time scales, the i-th first audio feature of the at least two first audio features being used to characterize feature information of each audio segment obtained after segmenting the first audio according to the i-th time scale, where i is a positive integer; generating a second audio based on the at least two first audio features through the audio generation model.

[0145] The first audio may be any audio, and optionally, the first audio is the voice of a certain object. The electronic device may acquire the first audio through an audio acquisition device, or the electronic device acquires the first audio through the Internet. The acquisition method of the first audio is limited here.

[0146] According to the implementation principle of step 302, the first audio can be encoded by the audio generation model to obtain at least two first audio features, and the encoding method is not described in detail here. Next, according to the implementation principle of step 303, the second audio can be generated based on the at least two first audio features by the audio generation model, and the generation method is not described in detail here.

[0147] The audio generation model of the embodiment of the present application can be applied in a variety of scenarios. For example, the audio generation model can be applied in an audio transmission scenario. Exemplarily, the source device (an electronic device) and the destination device (another electronic device) both deploy an audio generation model, the source device obtains the first audio, encodes the first audio through the audio generation model to obtain each first audio feature, and quantizes each first audio feature to obtain each third quantized feature. The source device transmits each third quantized feature to the destination device. The destination device fuses each third quantized feature, and reconstructs the first audio based on the fused feature to obtain the second audio. Since each third quantized feature can accurately characterize the first audio, the reconstructed second audio is close to the first audio, that is, the quality of the second audio can be guaranteed, and the transmission of the third quantized feature in the audio transmission scenario reduces the amount of transmitted data and saves transmission resources.

[0148] For another example, the audio generation model can be applied in a speech synthesis scenario. Exemplarily, the electronic device is deployed with an audio generation model, and the electronic device can obtain the first audio, encode the first audio through the audio generation model to obtain each first audio feature, and quantize each first audio feature to obtain each third quantized feature. The electronic device also deploys other trained models, which can update each third quantized feature based on the target text to obtain each fourth quantized feature. Among them, the third quantized feature represents the semantics, timbre, frequency and other information of the first audio, while the fourth quantized feature represents the timbre, frequency, etc. of the first audio and the semantics of the target text. In other words, the third quantized feature and the fourth quantized feature represent different text semantics, but represent the same sound information. The electronic device can fuse each fourth quantized feature through the audio generation model, and generate a second audio based on the fused features, so that the second audio is the voice of the target text spoken in the voice of the first audio, thereby improving the quality of the synthesized speech.

[0149] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions. For example, the sample audio and first audio involved in this application are all obtained with full authorization.

[0150] The above method encodes the sample audio through a neural network model to obtain at least two first sample features. Since the i-th first sample feature represents the information of each audio segment obtained after the sample audio is segmented according to the i-th time scale, different first sample features can reflect the information of the sample audio at different time scales, so that the representation ability of each first sample feature is stronger and can accurately reflect the sample audio. Based on this, a reconstructed audio with higher accuracy can be generated according to each first sample feature, so that when the neural network model is trained based on the reconstructed audio and the sample audio, the training effect can be improved, and the accuracy of the audio generation model can be improved, so that high-quality audio can be generated based on the audio generation model.

[0151] The above describes the training method of the audio generation model from the perspective of method steps. The following is further described in combination with scenarios. As mentioned above, the audio generation model of the embodiment of the present application can be applied to scenarios such as audio transmission and speech synthesis. Based on this, taking the speech synthesis scenario as an example, Figure 6 As shown, the speech synthesis process includes the following steps.

[0152] Step 601, use sample audio to train a neural network model to obtain an audio generation model.

[0153] like Figure 7 As shown, Figure 7 is a training diagram of an audio generation model provided in an embodiment of the present application, and the audio generation model is obtained based on the training of a neural network model. The neural network model includes a first encoder, and the first encoder includes four first encoding modules connected in series and a convolution layer. The sample audio 701 is input into the first encoder, and the sample audio is encoded by the first encoder to obtain a reference audio feature 702. The reference audio feature 702 can be expressed as , where R represents a real number, N represents the number of audio frames, and d represents the feature dimension of each audio frame.

[0154] The neural network model also includes a second encoder, which is used to encode the reference audio feature 702 to obtain multiple first sample features 703. Specifically, the second encoder includes three convolution layers connected in series, and each convolution layer is followed by an adapter in series. After the reference audio feature 702 is input into the second encoder, first, the reference audio feature 702 is convolved by convolution layer 1, and the convolution result of convolution layer 1 is processed by adapter 1 to obtain a first sample feature. Then, the convolution result of convolution layer 1 is convolved by convolution layer 2, and the convolution result of convolution layer 2 is processed by adapter 2 to obtain another first sample feature. , then the convolution result of convolution layer 2 is convolved through convolution layer 3, and the convolution result of convolution layer 3 is processed through adapter 3 to obtain another first sample feature .

[0155] The neural network model also includes a quantizer, which is used to quantize each first sample feature 703 and fuse them to obtain a reconstructed audio feature 704. Specifically, the quantizer includes three quantization modules. After the multiple first sample features 703 are input into the quantizer, firstly, the first sample features are quantized by quantization module 1. Get a first quantitative feature, such as Figure 7 As shown, the first quantization feature of the quantization module 1 is 9, 16, 52, 84, ...; for the first sample feature The difference between the first quantized feature and the corresponding first quantized feature is downsampled, and the downsampled result and the first sample feature are quantized by quantization module 2. The addition result of , obtains another first quantitative feature, such as Figure 7 As shown, the first quantitative features of the quantization module 2 are 71, 67, ...; for the first sample feature The difference between the first quantized feature and the corresponding first quantized feature is downsampled, and the downsampled result and the first sample feature are quantized by the quantization module 3. The addition result of , obtains another first quantitative feature, such as Figure 7As shown, the first quantized feature of quantization module 3 is 12... Then, the first quantized feature of quantization module 3 is upsampled, the upsampled result is added to the first quantized feature of quantization module 2, the added result is upsampled, and the upsampled result is added to the first quantized feature of quantization module 1 to obtain the reconstructed audio feature 704. Optionally, the reconstructed audio feature 704 can be expressed as .

[0156] The neural network model also includes a time-invariant representation extraction module and a grouping quantizer. The time-invariant representation extraction module extracts features from the encoding result of the second first encoding module (i.e., the jth downsampling result mentioned above) to obtain second sample features, and the grouping quantizer groups and quantizes the second sample features and fuses them to obtain fused features. The neural network model also includes a decoder, which includes two convolutional layers in series and four decoding modules. The decoder processes the fusion features And reconstruct the audio features 704 to obtain the reconstructed audio 705.

[0157] Next, on the one hand, the mutual information between each two first sample features in the plurality of first sample features 703 is calculated, and on the other hand, the L2 loss between the sample audio 701 and the reconstructed audio 705 is calculated. The neural network model is trained by the mutual information and the L2 loss to obtain the audio generation model. The implementation principle of step 601 is described above and will not be repeated here.

[0158] Step 602: Input the first speech into the audio generation model to obtain various quantitative features corresponding to the first speech.

[0159] In the speech synthesis scenario, the electronic device may install a speech synthesis application, which may provide an interface for speech synthesis. The embodiment of the present application does not limit the style, size, content, etc. of the interface. For example, the speech synthesis interface may be Figure 8 In this example, the speech synthesis interface may include an audio input area 802, in which a user may upload audio so that the electronic device may acquire the first speech. The speech synthesis interface may also include an audio selection area 804, in which a user may select audio so that the electronic device may acquire the first speech.

[0160] The electronic device is deployed with an audio generation model. The first speech can be input into the audio generation model, and the first speech is processed by the first encoder, the second encoder and the quantizer to obtain corresponding quantitative features. Among them, the method for determining the quantitative features corresponding to the first speech can be seen in the description of the first quantitative features above. The implementation principles of the two are similar and will not be repeated here.

[0161] In practical applications, more information can also be used to determine the various quantitative features corresponding to the first voice. For example, the various quantitative features corresponding to the first voice can be determined based on the first voice and its voice recognition result through an audio generation model. Optionally, the speech synthesis interface can also include a correction area 803, which is used to display the recognition result of the first voice, and display the corrected recognition result according to the user's operation of correcting the recognition result. The electronic device can determine the various quantitative features corresponding to the first voice based on the first voice and its corrected voice recognition result through an audio generation model.

[0162] Step 603: Update each quantitative feature corresponding to the first speech based on the target text to obtain each updated quantitative feature.

[0163] In the embodiment of the present application, the speech synthesis interface may further include a text acquisition area 805, and the text acquisition area 805 is used to acquire the target text input by the user so that the electronic device acquires the target text. The electronic device may update each quantitative feature corresponding to the first speech based on the target text to obtain each updated quantitative feature. The updated quantitative feature may represent information such as the timbre and frequency of the first speech, and may also represent the semantics of the target text.

[0164] Step 604: synthesize the second speech based on each updated quantized feature through the audio generation model.

[0165] The electronic device can synthesize the second speech based on each updated quantization feature through a quantizer and a decoder. The synthesis method of the second speech can be seen from the generation method of the reconstructed audio. The implementation principles of the two are similar and will not be repeated here.

[0166] In actual application, the electronic device can obtain the first voice, voice recognition results and target text through the voice synthesis interface. The voice synthesis interface includes a first synthesis control 808. When the user triggers the first synthesis control 808, the electronic device immediately executes steps 602 to 604 to immediately synthesize the second voice. Alternatively, the voice synthesis interface includes a second synthesis control 809. When the user triggers the second synthesis control 809, the electronic device executes steps 602 to 604 after a set time to synthesize the second voice. The voice synthesis interface may also include other information. For example, the voice synthesis interface also includes a first prompt message 801, a second prompt message 806 and a third prompt message 807. Among them, the first prompt message 801 is used to prompt the process of voice synthesis, the second prompt message 806 is used to prompt the optimization method of synthesized voice, and the third prompt message 807 is used to prompt the use of synthesized voice. The content of each prompt message is not limited here.

[0167] Fig. 9FIG. 1 is a schematic diagram of a structure of a training device for an audio generation model provided in an embodiment of the present application. Fig. 9 As shown, the device includes the following contents.

[0168] The acquisition module 901 is used to acquire sample audio and a neural network model to be trained.

[0169] The encoding module 902 is used to encode the sample audio through a neural network model to obtain at least two first sample features; the at least two first sample features correspond to different time scales, and the i-th first sample feature of the at least two first sample features is used to characterize the feature information of each audio segment obtained after the sample audio is segmented according to the i-th time scale, where i is a positive integer.

[0170] The generating module 903 is used to generate reconstructed audio based on at least two first sample features through a neural network model.

[0171] The training module 904 is used to train the neural network model through the sample audio and the reconstructed audio to obtain an audio generation model, and the audio generation model is used to generate the second audio based on the first audio.

[0172] In one possible implementation, the neural network model includes a quantizer and a decoder.

[0173] The generation module 903 is used to quantize each first sample feature of at least two first sample features through a quantizer to obtain each first quantized feature; fuse each first quantized feature through a quantizer to obtain a reconstructed audio feature; and decode the reconstructed audio feature through a decoder to obtain a reconstructed audio.

[0174] In a possible implementation, the generating module 903 is configured to quantize the first input information through a quantizer to obtain an i-th first quantized feature.

[0175] Among them, when i is 1, the first input information includes the first first sample feature of at least two first sample features; when i is greater than 1, the first input information includes the i-th first sample feature and the i-1-th residual feature, and the i-1-th residual feature represents the difference between the i-1-th first quantitative feature and the i-1-th first sample feature.

[0176] In a possible implementation manner, the number of first quantitative features is N, where N is a positive integer.

[0177] The generating module 903 is used to upsample the second input information through a quantizer to obtain a Ni-th upsampling result, and fuse the Ni-th upsampling result and the i-th first quantized feature through a quantizer to obtain a Ni-th fusion result.

[0178] Among them, when i is N-1, the second input information includes the Nth first quantized feature, and when i is any positive integer from 1 to N-2, the second input information includes the Ni-1th fusion result, and the N-1th fusion result is the reconstructed audio feature.

[0179] In one possible implementation, the training module 904 is used to determine the i-th first loss through the i-th first sample feature and the i-th first quantization feature, where the i-th first loss is used to characterize the difference between the i-th first sample feature and the i-th first quantization feature; based on the sample audio, the reconstructed audio and each first loss, the neural network model is trained to obtain an audio generation model.

[0180] In one possible implementation, the training module 904 is used to determine a second loss through at least two first sample features, where the second loss characterizes the correlation between every two first sample features; and train a neural network model based on the sample audio, reconstructed audio, and the second loss to obtain an audio generation model.

[0181] In one possible implementation, the neural network model includes a first encoder and a second encoder.

[0182] The encoding module 902 is used to encode the sample audio through the first encoder to obtain reference audio features; perform convolution processing on the third input information through the second encoder to obtain the i-th convolution result, and determine the i-th first sample feature based on the i-th convolution result.

[0183] Among them, when i is 1, the third input information includes the reference audio feature, and when i is greater than 1, the third input information includes the i-1th convolution result.

[0184] In one possible implementation, the encoding module 902 is used to perform feature extraction on the i-th convolution result to obtain local features and global features, where the local features represent local information of each audio segment and the global features represent global information of each audio segment; and fuse the local features and the global features to obtain the i-th first sample feature.

[0185] In a possible implementation, the encoding module 902 is configured to downsample the fourth input information through a first encoder to obtain a j-th downsampling result.

[0186] Among them, j is a positive integer. When j is 1, the fourth input information includes sample audio. When j is greater than 1, the fourth input information includes the j-1th downsampling result, and the last downsampling result is used to determine the reference audio feature.

[0187] In one possible implementation, generation module 903 is used to generate a second sample feature based on the j-th downsampling result through a neural network model, where the second sample feature represents time-independent information in the sample audio; and generate reconstructed audio based on the second sample feature and each first sample feature through a neural network model.

[0188] In one possible implementation, the generation module 903 is used to divide the second sample feature to obtain at least two second sample sub-features; quantize each of the at least two second sample sub-features to obtain each second quantized feature; and generate reconstructed audio based on each second quantized feature and each first sample feature through a neural network model.

[0189] In one possible implementation, the training module 904 is used to determine at least one of a third loss or a fourth loss based on the sample audio and the reconstructed audio, the third loss characterizing the difference between the sample audio and the reconstructed audio in the time domain, and the fourth loss characterizing the difference between the sample audio and the reconstructed audio in the frequency domain; based on at least one of the third loss or the fourth loss, train a neural network model to obtain an audio generation model.

[0190] In a possible implementation, the acquisition module 901 is further configured to acquire the first audio.

[0191] The encoding module 902 is also used to encode the first audio through the audio generation model to obtain at least two first audio features, where the at least two first audio features correspond to different time scales, and the i-th first audio feature among the at least two first audio features is used to characterize the feature information of each audio segment obtained after the first audio is segmented according to the i-th time scale, where i is a positive integer.

[0192] The generating module 903 is further configured to generate a second audio based on at least two first audio features by using an audio generation model.

[0193] The above-mentioned device encodes the sample audio through a neural network model to obtain at least two first sample features. Since the i-th first sample feature represents the information of each audio segment obtained after the sample audio is segmented according to the i-th time scale, different first sample features can reflect the information of the sample audio at different time scales, so that the representation ability of each first sample feature is stronger and can accurately reflect the sample audio. Based on this, a reconstructed audio with higher accuracy can be generated according to each first sample feature, so that when the neural network model is trained based on the reconstructed audio and the sample audio, the training effect can be improved, and the accuracy of the audio generation model can be improved, so that high-quality audio can be generated based on the audio generation model.

[0194] It should be understood that the above Fig. 9When the device provided realizes its functions, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0195] Fig.10 The structure block diagram of a terminal device 1000 provided by an exemplary embodiment of the present application is shown. The terminal device 1000 includes: a processor 1001 and a memory 1002 .

[0196] The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0197] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one computer program, which is used to be executed by the processor 1001 to implement the training method of the audio generation model provided in the method embodiment of the present application.

[0198] In some embodiments, the terminal device 1000 may further optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1003 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007 and a power supply 1008.

[0199] The peripheral device interface 1003 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0200] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0201] The display screen 1005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on the surface or above the surface of the display screen 1005. The touch signal can be input to the processor 1001 as a control signal for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, which is set on the front panel of the terminal device 1000; in other embodiments, the display screen 1005 can be at least two, which are respectively set on different surfaces of the terminal device 1000 or are folded; in other embodiments, the display screen 1005 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal device 1000. Even, the display screen 1005 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0202] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash may be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0203] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1001 for processing, or input them into the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal device 1000. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0204] The power supply 1008 is used to power various components in the terminal device 1000. The power supply 1008 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0205] In some embodiments, the terminal device 1000 further includes one or more sensors 1009 . The one or more sensors 1009 include, but are not limited to: an acceleration sensor 1011 , a gyroscope sensor 1012 , a pressure sensor 1013 , an optical sensor 1014 , and a proximity sensor 1015 .

[0206] The acceleration sensor 1011 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal device 1000. For example, the acceleration sensor 1011 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1001 can control the display screen 1005 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1011. The acceleration sensor 1011 can also be used to collect game or user motion data.

[0207] The gyro sensor 1012 can detect the body direction and rotation angle of the terminal device 1000, and the gyro sensor 1012 can cooperate with the acceleration sensor 1011 to collect the user's 3D actions on the terminal device 1000. The processor 1001 can implement the following functions based on the data collected by the gyro sensor 1012: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0208] The pressure sensor 1013 can be set on the side frame of the terminal device 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1013 is set on the side frame of the terminal device 1000, the user's holding signal of the terminal device 1000 can be detected, and the processor 1001 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is set on the lower layer of the display screen 1005, the processor 1001 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1005. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0209] The optical sensor 1014 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1014. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is reduced. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1014.

[0210] The proximity sensor 1015, also called a distance sensor, is usually arranged on the front panel of the terminal device 1000. The proximity sensor 1015 is used to collect the distance between the user and the front of the terminal device 1000. In one embodiment, when the proximity sensor 1015 detects that the distance between the user and the front of the terminal device 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from the screen-on state to the screen-off state; when the proximity sensor 1015 detects that the distance between the user and the front of the terminal device 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from the screen-off state to the screen-on state.

[0211] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal device 1000, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0212] Fig.11This is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more processors 1101 and one or more memories 1102, wherein the one or more memories 1102 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1101 to implement the training method of the audio generation model provided in the above-mentioned various method embodiments. Exemplarily, the processor 1101 is a CPU. Of course, the server 1100 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1100 may also include other components for implementing device functions, which will not be described in detail here.

[0213] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned training methods for an audio generation model.

[0214] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0215] In an exemplary embodiment, a computer program is also provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned training methods for an audio generation model.

[0216] In an exemplary embodiment, a computer program product is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned training methods for an audio generation model.

[0217] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0218] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0219] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for an audio generation model, characterized in that: The method comprises: Obtaining sample audio and a neural network model to be trained, wherein the sample audio is at least two; The sample audio is encoded by the neural network model to obtain at least two first sample features; the at least two first sample features correspond to different time scales, and the i-th first sample feature of the at least two first sample features is used to characterize feature information of each audio segment obtained after the sample audio is segmented according to the i-th time scale, where i is a positive integer; Generate reconstructed audio based on the at least two first sample features through the neural network model; For any two time scales, according to each positive sample pair and each negative sample pair, calculate the mutual information between the first sample features corresponding to the any two time scales, the positive sample pair includes two first sample features corresponding to the any two time scales and belonging to the same sample audio, and the negative sample pair includes two first sample features corresponding to the any two time scales and belonging to different sample audios; Determine a second loss based on the mutual information between the first sample features corresponding to each two time scales; Based on the sample audio, the reconstructed audio and the second loss, the neural network model is trained to obtain an audio generation model, and the audio generation model is used to generate the second audio based on the first audio.

2. The method according to claim 1, characterized in that The neural network model includes a quantizer and a decoder; the generating the reconstructed audio based on the at least two first sample features by the neural network model includes: quantizing each of the at least two first sample features by the quantizer to obtain each first quantized feature; The quantizer is used to fuse the first quantized features to obtain a reconstructed audio feature; The reconstructed audio features are decoded by the decoder to obtain reconstructed audio.

3. The method according to claim 2, characterized in that The step of quantizing each of the at least two first sample features by the quantizer to obtain each first quantized feature includes: quantizing the first input information by the quantizer to obtain an i-th first quantized feature; Among them, when the i is 1, the first input information includes the first first sample feature of the at least two first sample features; when the i is greater than 1, the first input information includes the i-th first sample feature and the i-1-th residual feature, and the i-1-th residual feature represents the difference between the i-1-th first quantized feature and the i-1-th first sample feature.

4. The method according to claim 2, characterized in that: The number of the first quantized features is N, where N is a positive integer; and the step of fusing the first quantized features through the quantizer to obtain the reconstructed audio features includes: Upsampling the second input information through the quantizer to obtain a Ni-th upsampling result, and fusing the Ni-th upsampling result and the i-th first quantized feature through the quantizer to obtain the Ni-th fusion result; Among them, when i is N-1, the second input information includes the Nth first quantized feature, and when i is any positive integer from 1 to N-2, the second input information includes the Ni-1th fusion result, and the N-1th fusion result is the reconstructed audio feature.

5. The method according to claim 2, characterized in that: The step of training the neural network model based on the sample audio, the reconstructed audio and the second loss to obtain an audio generation model comprises: Determine an i-th first loss by using the i-th first sample feature and the i-th first quantitative feature, where the i-th first loss is used to characterize the difference between the i-th first sample feature and the i-th first quantitative feature; Based on the sample audio, the reconstructed audio, the first losses and the second losses, the neural network model is trained to obtain an audio generation model.

6. The method according to any one of claims 1 to 5, characterized in that: The neural network model includes a first encoder and a second encoder; encoding the sample audio through the neural network model to obtain at least two first sample features includes: Encoding the sample audio by the first encoder to obtain reference audio features; Performing convolution processing on the third input information through the second encoder to obtain the i-th convolution result, and determining the i-th first sample feature based on the i-th convolution result; When i is 1, the third input information includes the reference audio feature; when i is greater than 1, the third input information includes the i-1th convolution result.

7. The method according to claim 6, characterized in that The determining the i-th first sample feature based on the i-th convolution result includes: Performing feature extraction on the i-th convolution result to obtain local features and global features, wherein the local features represent local information of each audio segment, and the global features represent global information of each audio segment; The local feature and the global feature are fused to obtain the i-th first sample feature.

8. The method according to claim 6, characterized in that The step of encoding the sample audio by the first encoder to obtain a reference audio feature includes: Down-sampling the fourth input information by the first encoder to obtain a j-th down-sampling result; Wherein, j is a positive integer. When j is 1, the fourth input information includes the sample audio. When j is greater than 1, the fourth input information includes the j-1th downsampling result, and the last downsampling result is used to determine the reference audio feature.

9. The method according to claim 8, characterized in that The step of generating the reconstructed audio based on the at least two first sample features by the neural network model comprises: Generate a second sample feature based on the j-th downsampling result by the neural network model, wherein the second sample feature represents time-independent information in the sample audio; Generate reconstructed audio based on the second sample features and each first sample feature through the neural network model.

10. The method according to claim 9, characterized in that The step of generating the reconstructed audio based on the second sample feature and each first sample feature by the neural network model comprises: dividing the second sample feature to obtain at least two second sample sub-features; quantizing each second sample sub-feature of the at least two second sample sub-features to obtain each second quantized feature; The reconstructed audio is generated by the neural network model based on the second quantized features and the first sample features.

11. The method according to any one of claims 1 to 5, characterized in that: The step of training the neural network model based on the sample audio, the reconstructed audio and the second loss to obtain an audio generation model comprises: Based on the sample audio and the reconstructed audio, determine at least one of a third loss or a fourth loss, wherein the third loss represents a difference between the sample audio and the reconstructed audio in the time domain, and the fourth loss represents a difference between the sample audio and the reconstructed audio in the frequency domain; Based on at least one of the third loss or the fourth loss and the second loss, the neural network model is trained to obtain the audio generation model.

12. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Get the first audio; encoding the first audio through the audio generation model to obtain at least two first audio features, wherein the at least two first audio features correspond to different time scales, and an i-th first audio feature among the at least two first audio features is used to represent feature information of each audio segment obtained after segmenting the first audio according to the i-th time scale, where i is a positive integer; Generate second audio based on the at least two first audio features through the audio generation model.

13. A training device for an audio generation model, characterized in that: The device comprises: An acquisition module, used to acquire sample audio and a neural network model to be trained, wherein the sample audio is at least two; an encoding module, configured to encode the sample audio through the neural network model to obtain at least two first sample features; the at least two first sample features correspond to different time scales, and the i-th first sample feature of the at least two first sample features is used to characterize feature information of each audio segment obtained after segmenting the sample audio according to the i-th time scale, where i is a positive integer; A generating module, configured to generate reconstructed audio based on the at least two first sample features through the neural network model; A training module is used to calculate, for any two time scales, the mutual information between the first sample features corresponding to the any two time scales according to each positive sample pair and each negative sample pair, wherein the positive sample pair includes two first sample features corresponding to the same sample audio at the any two time scales, and the negative sample pair includes two first sample features corresponding to different sample audios at the any two time scales; determine the second loss based on the mutual information between the first sample features corresponding to every two time scales; train the neural network model based on the sample audio, the reconstructed audio and the second loss to obtain an audio generation model, wherein the audio generation model is used to generate the second audio based on the first audio.

14. The device according to claim 13, characterized in that The neural network model includes a quantizer and a decoder; The generating module is configured to quantize each of the at least two first sample features by the quantizer to obtain each first quantized feature; The first quantized features are fused by the quantizer to obtain a reconstructed audio feature; and the reconstructed audio feature is decoded by the decoder to obtain a reconstructed audio.

15. The device according to claim 14, characterized in that The generating module is used to quantize the first input information through the quantizer to obtain an i-th first quantized feature; Among them, when the i is 1, the first input information includes the first first sample feature of the at least two first sample features; when the i is greater than 1, the first input information includes the i-th first sample feature and the i-1-th residual feature, and the i-1-th residual feature represents the difference between the i-1-th first quantized feature and the i-1-th first sample feature.

16. The device according to claim 14, characterized in that The number of the first quantitative features is N, where N is a positive integer; The generating module is configured to upsample the second input information through the quantizer to obtain a Ni-th upsampling result, and fuse the Ni-th upsampling result and the i-th first quantized feature through the quantizer to obtain the Ni-th fusion result; Among them, when i is N-1, the second input information includes the Nth first quantized feature, and when i is any positive integer from 1 to N-2, the second input information includes the Ni-1th fusion result, and the N-1th fusion result is the reconstructed audio feature.

17. The device according to claim 14, characterized in that The training module is used to determine an i-th first loss through the i-th first sample feature and the i-th first quantitative feature, wherein the i-th first loss is used to characterize the difference between the i-th first sample feature and the i-th first quantitative feature; Based on the sample audio, the reconstructed audio, the first losses and the second losses, the neural network model is trained to obtain an audio generation model.

18. The device according to any one of claims 13 to 17, characterized in that The neural network model includes a first encoder and a second encoder; The encoding module is configured to encode the sample audio through the first encoder to obtain a reference audio feature; perform convolution processing on the third input information through the second encoder to obtain the i-th convolution result, and determine the i-th first sample feature based on the i-th convolution result; When i is 1, the third input information includes the reference audio feature; when i is greater than 1, the third input information includes the i-1th convolution result.

19. The device according to claim 18, characterized in that The encoding module is used to perform feature extraction on the i-th convolution result to obtain local features and global features, wherein the local features represent local information of each audio segment, and the global features represent global information of each audio segment; and fuse the local features and the global features to obtain the i-th first sample feature.

20. The device according to claim 18, characterized in that The encoding module is used to downsample the fourth input information through the first encoder to obtain a j-th downsampling result; Wherein, j is a positive integer. When j is 1, the fourth input information includes the sample audio. When j is greater than 1, the fourth input information includes the j-1th downsampling result, and the last downsampling result is used to determine the reference audio feature.

21. The device according to claim 20, characterized in that The generation module is used to generate a second sample feature based on the j-th downsampling result through the neural network model, wherein the second sample feature represents time-independent information in the sample audio; and generate reconstructed audio based on the second sample feature and each first sample feature through the neural network model.

22. The device according to claim 21, characterized in that The generating module is used to divide the second sample feature to obtain at least two second sample sub-features; Quantify each second sample sub-feature of the at least two second sample sub-features to obtain each second quantized feature; and generate the reconstructed audio based on the each second quantized feature and each first sample feature through the neural network model.

23. The device according to any one of claims 13 to 17, characterized in that The training module is used to determine at least one of a third loss or a fourth loss based on the sample audio and the reconstructed audio, wherein the third loss characterizes the difference between the sample audio and the reconstructed audio in the time domain, and the fourth loss characterizes the difference between the sample audio and the reconstructed audio in the frequency domain; based on at least one of the third loss or the fourth loss and the second loss, the neural network model is trained to obtain the audio generation model.

24. The device according to any one of claims 13 to 17, characterized in that The acquisition module is further used to acquire the first audio; The encoding module is further used to encode the first audio through the audio generation model to obtain at least two first audio features, where the at least two first audio features correspond to different time scales, and the i-th first audio feature of the at least two first audio features is used to represent feature information of each audio segment obtained after the first audio is segmented according to the i-th time scale, where i is a positive integer; The generation module is further used to generate a second audio based on the at least two first audio features through the audio generation model.

25. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the training method of the audio generation model as described in any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: At least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the training method of the audio generation model as described in any one of claims 1 to 12.

27. A computer program product, characterized in that The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that the electronic device implements the training method of the audio generation model as described in any one of claims 1 to 12.

Images