Audio reconstruction method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of research and development design and audio processing, is applied to pre-processing scenes of audio reconstruction and audio synthesis, and relates to an audio reconstruction method, device and equipment and a storage medium thereof. A potential feature representation is obtained through encoding by an audio encoder; inputting the potential feature representation into a single quantizer of which the codebook space size is expanded, and performing quantization processing to obtain discretized audio feature representation; carrying out decoding processing on the discretized audio feature representation by adopting an improved decoder after adversarial training to obtain a real value signal; converting the real value signal to obtain a signal spectrogram; and inverse Fourier transform is carried out according to the phase and the amplitude value of the signal spectrogram, and reconstruction of the original audio stream is completed. The codebook space of the quantizer is expanded, the decoder is improved, and the adversarial training mode is adopted to carry out audio coding and decoding training, so that the high availability of the reconstructed audio is ensured when the audio is actually reconstructed.

Description

Technical Field

[0001] The present application relates to the field of R&D design and audio processing technology, and is applied to pre-processing scenarios of audio reconstruction and audio synthesis, and in particular to an audio reconstruction method, device, equipment and storage medium thereof. Background Art

[0002] In recent years, large-scale language models have achieved remarkable success in generative tasks involving multi-speaker speech synthesis, music generation, and audio generation. In addition, integrating speech modalities into multimodal unified large models has also attracted widespread attention. These successes can be largely attributed to the discrete acoustic codec representations produced by neural codec models.

[0003] These discrete acoustic codec models bridge the gap between continuous speech and tag-based language models, discretizing high-rate audio signals into a limited set of tags, allowing audio data to be integrated into large multimodal models. Most end-to-end discrete codec models typically adopt a three-level structure consisting of an encoder, a residual vector quantization (RVQ) module, and a decoder. However, at present, the above three-level structure still has a lot of room for improvement in audio reconstruction to achieve high availability of reconstructed audio. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an audio reconstruction method, apparatus, device and storage medium thereof to improve audio reconstruction and achieve high availability of reconstructed audio.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an audio reconstruction method, which adopts the following technical solution:

[0006] An audio reconstruction method comprises the following steps:

[0007] Get the original audio stream;

[0008] Inputting the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream;

[0009] The latent feature representation is input into a single quantizer with an expanded codebook space size, and the latent feature representation is quantized to obtain a discretized audio feature representation, wherein the codebook space size of the single quantizer is increased by 2 10 Expand to 2 12 ;

[0010] The discretized audio feature representation is decoded by using an improved decoder after adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation, wherein the basic backbone network of the improved decoder is composed of a one-dimensional convolutional layer, an attention module and a convolutional block;

[0011] The real-valued signal is converted using a single-sideband spectrum to obtain a converted signal spectrum diagram;

[0012] An inverse Fourier transform is performed according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream.

[0013] Furthermore, the step of inputting the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream specifically includes:

[0014] Performing an initial convolution operation on the input original audio stream by using the 1D convolution layer in the audio encoder to extract initial audio features in the original audio stream;

[0015] The convolution block in the audio encoder is used to perform downsampling and filtering processing on the initial audio features, and according to the configuration of the sampling step, the key audio features corresponding to the target frequency are obtained;

[0016] The key audio features are represented as potential features corresponding to the original audio stream.

[0017] Furthermore, before executing the step of obtaining the original audio stream, the method further includes:

[0018] Acquire an input audio stream for batch encoding and decoding training, wherein the input audio stream includes an audio stream of a real object;

[0019] The input audio stream is processed by the audio encoder, the single quantizer and the improved decoder in sequence to obtain a reconstructed audio corresponding to the input audio stream;

[0020] The authenticity of the reconstructed audio is estimated using a preset discriminator until the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold, and the training of the audio encoder, the single quantizer and the improved decoder is completed.

[0021] Furthermore, the preset discriminator includes a multi-period discriminator, a multi-resolution discriminator and a multi-scale discriminator, and the step of using the preset discriminator to perform authenticity estimation on the reconstructed audio specifically includes:

[0022] Step 401, using the multi-period discriminator, the multi-resolution discriminator and the multi-scale discriminator to jointly obtain the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value and the feature matching loss value corresponding to the reconstructed audio in the reconstruction process;

[0023] Step 402: The quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value are used as authenticity estimation factors, and combined with a preset comprehensive calculation method, the authenticity estimation is calculated.

[0024] Furthermore, before executing the step of using the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value as authenticity estimation factors and combining a preset comprehensive calculation method to calculate the authenticity estimation, the method further includes:

[0025] Obtain loss requirement conditions pre-set for the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value, respectively, wherein the loss requirement condition includes that the loss value does not exceed a set loss threshold;

[0026] After step 401, the following steps are further included:

[0027] Compare the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value with corresponding loss requirement conditions;

[0028] If any of the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value does not meet the corresponding loss requirement conditions, the audio encoder, the single quantizer, and the improved decoder are subjected to model parameter tuning processing until the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions;

[0029] If the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions, step 402 is directly executed.

[0030] Furthermore, the step of using the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value and the feature matching loss value as authenticity estimation factors, and combining with a preset comprehensive calculation method to calculate the authenticity estimation, specifically includes:

[0031] Obtaining dynamic floating weights pre-set for the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value;

[0032] According to the dynamic floating weights, current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point are obtained respectively;

[0033] Based on the current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point, a weighted sum algorithm is used to calculate the total loss value corresponding to the current training time point;

[0034] The total loss value is used to obtain the corresponding authenticity estimate at the current training time point, wherein the total loss value is inversely proportional to the authenticity estimate.

[0035] Further, after executing the step of obtaining the corresponding truthfulness estimation at the current training time point through the total loss value, the method further includes:

[0036] Determining whether the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold;

[0037] If the authenticity estimate of the reconstructed audio does not reach the preset authenticity threshold, the training is continued, and the latest weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the latest training time point are obtained in an iterative calculation manner, and

[0038] Based on the latest weight, a weighted sum algorithm is used to calculate the total loss value corresponding to the latest training time point, and the corresponding truthfulness estimation value at the latest training time point is obtained through the total loss value;

[0039] Until the latest authenticity estimate corresponding to the reconstructed audio reaches a preset authenticity threshold, the training of the audio encoder, the single quantizer and the improved decoder is completed.

[0040] In order to solve the above technical problems, the embodiment of the present application also provides an audio reconstruction device, which adopts the following technical solution:

[0041] An audio reconstruction device, comprising:

[0042] The original audio stream acquisition module is used to acquire the original audio stream;

[0043] An audio encoding processing module, used for inputting the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream;

[0044] A quantization processing module is used to input the potential feature representation into a single quantizer with an expanded codebook space size, and quantize the potential feature representation to obtain a discretized audio feature representation, wherein the codebook space size of the single quantizer is increased by 2 10 Expand to 2 12 ;

[0045] A decoding processing and real-valued signal acquisition module, used to decode the discretized audio feature representation using an improved decoder after adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation, wherein the basic backbone network of the improved decoder is composed of a one-dimensional convolutional layer, an attention module and a convolutional block;

[0046] A signal spectrum conversion module, used to convert the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum;

[0047] The inverse Fourier transform module is used to perform inverse Fourier transform according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0049] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned audio reconstruction method when executing the computer-readable instructions.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0051] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio reconstruction method described above are implemented.

[0052] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0053] The audio reconstruction method described in the embodiment of the present application is to obtain the original audio stream; input it into a preset audio encoder to obtain the corresponding potential feature representation; input the potential feature representation into a single quantizer with an expanded codebook space size, quantize the potential feature representation to obtain a discretized audio feature representation; use an improved decoder after adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; use a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; perform an inverse Fourier transform based on the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for codec training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0056] Figure 2 is a flowchart of an embodiment of an audio reconstruction method according to the present application;

[0057] Figure 3 is a flowchart of a specific embodiment of adversarial generation training in the audio reconstruction method described in the present application;

[0058] Figure 4 is a flowchart of a specific embodiment of performing authenticity estimation on the reconstructed audio in the audio reconstruction method described in the present application;

[0059] Figure 5 is a flowchart of a specific embodiment of judging the model parameter tuning process in the audio reconstruction method described in the present application;

[0060] Figure 6 yes Figure 4 A flowchart of a specific embodiment of step 402 is shown;

[0061] Figure 7 is a flowchart of a specific embodiment of iteratively calculating the total loss value in the audio reconstruction method described in the present application;

[0062] Figure 8is a structural schematic diagram of an embodiment of an audio reconstruction device according to the present application;

[0063] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0067] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0068] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0069] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0070] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0071] It should be noted that the audio reconstruction method provided in the embodiment of the present application is generally executed by a server, and accordingly, the audio reconstruction device is generally arranged in the server.

[0072] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0073] Continue to refer Figure 2 , shows a flow chart of an embodiment of an audio reconstruction method according to the present application. The audio reconstruction method comprises the following steps:

[0074] Step 201: Obtain the original audio stream.

[0075] In this embodiment, the original audio stream includes an audio stream to be reconstructed, for example, an audio stream of a real speaker in a speech synthesis scenario. The audio stream of the real speaker is reconstructed to facilitate the subsequent synthesis of the imitation speech of the target object.

[0076] Step 202: input the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream.

[0077] In this embodiment, the audio encoder selects an encoder in an end-to-end transformer speech processing architecture model, for example, an encoder in SoundStream and an encoder in Encodec, wherein SoundStream and Encodec are both audio codecs of end-to-end architecture.

[0078] Specifically, the original audio stream is input into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream, that is, the audio signal is downsampled in the time domain by the audio encoder to obtain a compressed audio frame.

[0079] Step 203: input the latent feature representation into a single quantizer whose codebook space size is expanded, and perform quantization processing on the latent feature representation to obtain a discretized audio feature representation.

[0080] The codebook space size of the single quantizer is 2 10 Expand to 2 12 .

[0081] In this embodiment, each compressed audio frame is quantized by the single quantizer to obtain a discrete audio feature representation. Compared with the existing processing method, the present application no longer uses a series of quantizers, but a single quantizer. In order to ensure that the codebook space of a single quantizer meets the quantization processing requirements and compresses all speech representations into the codebook space of the single quantizer, the codebook space size of the single quantizer is first expanded to avoid the use of a series of quantizers and perform a complex cross-channel dimensional hierarchical design on the quantizer.

[0082] Step 204, using an improved decoder that has undergone adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation, wherein the basic backbone network of the improved decoder is composed of a one-dimensional convolutional layer, an attention module and a convolutional block.

[0083] In this embodiment, a standard decoder that is a mirror image of the audio encoder is no longer used for decoding. Since the standard approach is used for decoding, a series of dilated convolutions are required to expand the receptive field, and transposed convolutions are used to sequentially upsample the feature sequence to a waveform, which is susceptible to aliasing artifacts. Here, it is considered that only one-dimensional convolution layers, attention modules, and convolution blocks are used to form the improved decoder.

[0084] Specifically, during the decoding process, the discretized audio feature representation is input into the one-dimensional convolution layer, the attention module and the convolution block. The purpose of the convolution block is to embed the input discretized audio feature representation features into the hidden dimension. A series of convolution blocks can be used, and the kernel sizes of the series of convolution blocks gradually increase to form a deep convolution structure in which the convolution kernel size increases layer by layer. Layer normalization is used between the series of convolution blocks, and the GELU activation function is used to perform activation processing after the last convolution block of the deep convolution structure to obtain a real-valued signal.

[0085] Step 205: convert the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum diagram.

[0086] Step 206: Perform inverse Fourier transform according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream.

[0087] In this embodiment, the original audio stream is obtained; input into a preset audio encoder to obtain the corresponding potential feature representation; the potential feature representation is input into a single quantizer with an expanded codebook space size, and the potential feature representation is quantized to obtain a discretized audio feature representation; the discretized audio feature representation is decoded using an improved decoder after adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation; the real-valued signal is converted using a single sideband spectrum to obtain a converted signal spectrum diagram; an inverse Fourier transform is performed according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for encoding and decoding training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.

[0088] In this embodiment, the audio encoder is composed of a 1D convolutional layer and 4 convolutional blocks, wherein the 1D convolutional layer has 32 convolutional channels and a convolution kernel size of 7×7, each convolutional block is composed of a residual unit and a downsampling layer, the residual unit includes 2 convolutional layers with a convolution kernel size of 3×3 and a jump connection layer, the downsampling layer includes a convolutional layer with a stride of S, and the convolution kernel size of the convolution layer with a stride of S is twice the step size, and the configuration of the step size S includes two configuration modes, one configuration mode is (2,4,5,8), and the other configuration mode is (4,5,5,6).

[0089] In addition, after the convolution block of the audio encoder, a serialization processing structure is also included, for example, a two-layer LSTM timing network is set after each convolution block for sequence modeling. By setting a two-layer LSTM timing network after each convolution block, the timing control during audio reconstruction is guaranteed.

[0090] In this embodiment, the step of inputting the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream specifically includes: using a 1D convolution layer in the audio encoder to perform an initial convolution operation on the input original audio stream to extract initial audio features in the original audio stream; using a convolution block in the audio encoder to downsample and filter the initial audio features to obtain key audio features corresponding to the target frequency based on the configuration of the sampling step; and using the key audio features as the potential feature representation corresponding to the original audio stream.

[0091] Specifically, the 1D convolution layer in the audio encoder is used to perform an initial convolution operation on the input original audio stream to extract the initial audio features in the original audio stream, wherein the initial convolution operation represents sampling the input original audio stream 24,000 times per second, and the initial audio features are audio features obtained by sampling 24,000 times per second; the convolution block in the audio encoder is used to perform downsampling and filtering on the initial audio features, and according to the configuration of the sampling step, the key audio features corresponding to the target frequency are obtained, wherein when the sampling step is configured as When the sampling step size is (2,4,5,8), that is, after the four convolution blocks are down-sampled and filtered, the key audio features with a target frequency of 75 times per second are obtained, and the filtering multiple is 2×4×5×8; furthermore, when the sampling step size is configured as (4,5,5,6), that is, after the four convolution blocks are down-sampled and filtered, the key audio features with a target frequency of 40 times per second are obtained, and the filtering multiple is 4×5×5×6; accordingly, the key audio features with a target frequency of 75 times per second or the key audio features with a target frequency of 40 times per second are represented as the potential features corresponding to the original audio stream.

[0092] It should be understood that, in this embodiment, the audio encoder, the single quantizer and the improved decoder essentially constitute an audio sample generator. In order to ensure that the reconstructed audio of the entire audio sample generator is closer to the real speech, before the actual audio reconstruction is performed, an adversarial generation training method is used in advance to train the entire audio sample generator.

[0093] Continue to refer Figure 3 In some optional implementations, a step of performing adversarial generation training is also included before step 201. Figure 3 is a flowchart of a specific embodiment of adversarial generation training in the audio reconstruction method described in the present application, comprising the following steps:

[0094] Step 301, obtaining an input audio stream for batch encoding and decoding training, wherein the input audio stream includes an audio stream of a real object;

[0095] Specifically, the real object includes a real person.

[0096] Step 302, the input audio stream is processed by the audio encoder, the single quantizer and the improved decoder in sequence to obtain a reconstructed audio corresponding to the input audio stream;

[0097] Step 303: Use a preset discriminator to estimate the authenticity of the reconstructed audio until the estimated authenticity of the reconstructed audio reaches a preset authenticity threshold, and the training of the audio encoder, the single quantizer and the improved decoder is completed.

[0098] Specifically, the entire architecture of the audio encoder, the single quantizer and the improved decoder is regarded as an audio sample generator, and a preset discriminator is used to identify the authenticity estimation of the reconstructed audio generated by the generator compared with the actual audio, thereby ensuring that the training of the audio encoder, the single quantizer and the improved decoder is completed, so as to finally realize the entire audio sample generator to reconstruct the audio stream in the actual speech synthesis pre-task.

[0099] In this embodiment, the preset discriminators include a multi-period discriminator, a multi-resolution discriminator and a multi-scale discriminator. The multi-period discriminator, the multi-resolution discriminator and the multi-scale discriminator are used to jointly perform authenticity identification processing on the reconstructed audio. The above three types of discriminators are used to support multi-period, multi-resolution and multi-scale identification during audio reconstruction training, so as to ensure the high availability of the audio sample generator that is finally trained in adversarial generation.

[0100] Continue to refer Figure 4 , Figure 4 is a flowchart of a specific embodiment of performing authenticity estimation on the reconstructed audio in the audio reconstruction method described in the present application, comprising the following steps:

[0101] Step 401, using the multi-period discriminator, the multi-resolution discriminator and the multi-scale discriminator to jointly obtain the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value and the feature matching loss value corresponding to the reconstructed audio in the reconstruction process;

[0102] It should be understood that the multi-cycle discriminator, multi-resolution discriminator and multi-scale discriminator are three types of discriminators, which refer to the identification that supports multi-cycle, multi-resolution and multi-scale, and do not represent the specific number of discriminators. The specific number of discriminators can be freely set according to the amount of adversarial training data. The discriminator calculates the loss values ​​corresponding to the generator in different processing stages, for example: the feature matching loss value of the encoder stage, the quantization loss value of the quantization processing stage, the feature matching loss value of the decoder stage, the Mel spectrum reconstruction loss value of the signal spectrum graph generation stage and the final adversarial loss value. When the loss value is output, it is combined with the corresponding loss function and output layer output.

[0103] Step 402: The quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value are used as authenticity estimation factors, and combined with a preset comprehensive calculation method, the authenticity estimation is calculated.

[0104] Specifically, when performing realism estimation, the hinge loss form is adopted, and the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value are introduced. The quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value are used as realism estimation factors to jointly calculate the realism estimation.

[0105] In this embodiment, the feature matching loss value includes a feature matching loss value at the encoder stage and a feature matching loss value at the decoder stage.

[0106] In this embodiment, when performing the authenticity valuation calculation, loss values ​​of multiple dimensions are introduced as valuation factors, thereby ensuring the reliability and scientificity of the final authenticity valuation.

[0107] In this embodiment, in some optional implementations, after step 401, it also includes: obtaining loss requirement conditions pre-set for the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value, wherein the loss requirement conditions include that the loss value does not exceed a set loss threshold.

[0108] Continue to refer Figure 5 In some optional implementations, a step of determining the model parameter tuning process is included before step 402. Figure 5 It is a flowchart of a specific embodiment of judging the model parameter tuning process in the audio reconstruction method described in this application, comprising the following steps:

[0109] Step 501, comparing the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value and the feature matching loss value with corresponding loss requirement conditions;

[0110] Step 502: If any of the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value does not meet the corresponding loss requirement conditions, the audio encoder, the single quantizer, and the improved decoder are subjected to model parameter tuning processing until the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions;

[0111] Step 503: If the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions, directly execute step 402.

[0112] Specifically, by limiting that the quantization loss value, Mel-spectrum reconstruction loss value, adversarial loss value and feature matching loss value all meet the corresponding loss requirements, the quantization loss value, Mel-spectrum reconstruction loss value, adversarial loss value and feature matching loss value can be used as valuation factors for authenticity valuation. If any loss value among the quantization loss value, Mel-spectrum reconstruction loss value, adversarial loss value and feature matching loss value does not meet the corresponding loss requirements, it is necessary to perform model parameter tuning processing on the audio encoder, the single quantizer and the improved decoder, thereby ensuring the high availability of the audio sample generator that has completed the final adversarial generation training.

[0113] Continue to refer Figure 6 , Figure 6 yes Figure 4 The flowchart of a specific embodiment of step 402 shown includes the following steps:

[0114] Step 601, obtaining dynamic floating weights pre-set for the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value;

[0115] In this embodiment, a dynamic floating weight is introduced. Specifically, the dynamic floating weight is related to the order and timeliness of audio processing, for example: the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value and the feature matching loss value, wherein the feature matching loss value includes the feature matching loss value of the encoder stage and the feature matching loss value of the decoder stage. In the order of obtaining these loss values, they are the feature matching loss value of the encoder stage, the quantization loss value, the feature matching loss value of the decoder stage, the Mel spectrum reconstruction loss value and the adversarial loss value. Moreover, as the time of adversarial generation training increases, the important loss values ​​concerned in each time period will gradually change. In summary, as the time of adversarial generation training increases, the attention of the loss value gradually focuses on the feature matching loss value of the encoder stage first, then the quantization loss value, the feature matching loss value of the decoder stage, the Mel spectrum reconstruction loss value, and finally the final adversarial loss value. Therefore, the dynamic floating weight is set to combine the order and timeliness of audio processing, so as to dynamically adjust the calculation weight of each loss value, which is more reasonable and scientific.

[0116] Step 602, according to the dynamic floating weights, obtaining the current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value and the feature matching loss value at the current training time point;

[0117] Step 603, based on the current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point, a weighted sum algorithm is used to calculate the total loss value corresponding to the current training time point;

[0118] Step 604, obtaining a corresponding authenticity estimate at the current training time point through the total loss value, wherein the total loss value and the authenticity estimate are in inverse proportion.

[0119] Specifically, the larger the total loss value is, the smaller the authenticity estimate is, and the lower the confidence of the reconstructed audio is; the smaller the total loss value is, the larger the authenticity estimate is, the higher the confidence of the reconstructed audio is, and the higher the availability of the reconstructed audio is.

[0120] Continue to refer Figure 7 In some optional implementations, after step 604, a step of iteratively calculating the total loss value is also included. Figure 7 is a flowchart of a specific embodiment of iteratively calculating the total loss value in the audio reconstruction method described in the present application, comprising the following steps:

[0121] Step 701, determining whether the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold;

[0122] Step 702: If the authenticity estimate of the reconstructed audio does not reach the preset authenticity threshold, continue training, and obtain the latest weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the latest training time point in an iterative calculation manner, and

[0123] Step 703, based on the latest weight, a weighted sum algorithm is used to calculate the total loss value corresponding to the latest training time point, and the corresponding authenticity estimation value at the latest training time point is obtained through the total loss value;

[0124] Step 704: until the latest authenticity estimate corresponding to the reconstructed audio reaches a preset authenticity threshold, the training of the audio encoder, the single quantizer and the improved decoder is completed.

[0125] Specifically, under the condition that the quantization loss value, Mel spectrum reconstruction loss value, adversarial loss value and feature matching loss value all meet the corresponding loss requirements, if the authenticity estimation of the reconstructed audio does not reach the preset authenticity threshold, the training is continued, and the latest authenticity estimation corresponding to the reconstructed audio at the latest time point is obtained by iterative calculation, so as to finally obtain the trained audio encoder, single quantizer and improved decoder, so as to ensure the authenticity of the subsequent actual audio reconstruction.

[0126] The present application obtains the original audio stream; inputs it into a preset audio encoder to obtain the corresponding potential feature representation; inputs the potential feature representation into a single quantizer with an expanded codebook space size, quantizes the potential feature representation, and obtains a discretized audio feature representation; uses an improved decoder after adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; uses a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performs an inverse Fourier transform based on the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for codec training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.

[0127] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0128] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0129] In the embodiment of the present application, the original audio stream is obtained; input into a preset audio encoder to obtain the corresponding potential feature representation; the potential feature representation is input into a single quantizer after the codebook space size is expanded, and the potential feature representation is quantized to obtain a discretized audio feature representation; the discretized audio feature representation is decoded by an improved decoder after adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation; the real-valued signal is converted using a single sideband spectrum to obtain a converted signal spectrum diagram; an inverse Fourier transform is performed according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for codec training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.

[0130] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of an audio reconstruction device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0131] like Figure 8 As shown, the audio reconstruction device 800 described in this embodiment includes: an original audio stream acquisition module 801, an audio encoding processing module 802, a quantization processing module 803, a decoding processing and real-valued signal acquisition module 804, a signal spectrum diagram conversion module 805 and an inverse Fourier transform module 806.

[0132] in:

[0133] The original audio stream acquisition module 801 is used to acquire the original audio stream;

[0134] The audio encoding processing module 802 is used to input the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream;

[0135] The quantization processing module 803 is used to input the potential feature representation into a single quantizer with an expanded codebook space size, and quantize the potential feature representation to obtain a discretized audio feature representation, wherein the codebook space size of the single quantizer is increased by 2 10 Expand to 2 12 ;

[0136] A decoding processing and real-valued signal acquisition module 804 is used to decode the discretized audio feature representation using an improved decoder after adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation, wherein the basic backbone network of the improved decoder is composed of a one-dimensional convolutional layer, an attention module and a convolutional block;

[0137] The signal spectrum conversion module 805 is used to convert the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum;

[0138] The inverse Fourier transform module 806 is used to perform inverse Fourier transform according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream.

[0139] The present application obtains the original audio stream; inputs it into a preset audio encoder to obtain the corresponding potential feature representation; inputs the potential feature representation into a single quantizer with an expanded codebook space size, quantizes the potential feature representation, and obtains a discretized audio feature representation; uses an improved decoder after adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; uses a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performs an inverse Fourier transform based on the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for codec training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0141] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0142] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0143] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c which are interconnected through a system bus. It should be noted that: Fig. 9Only a computer device 9 having a component memory 9a, a processor 9b, and a network interface 9c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0144] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0145] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 9a can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as a computer-readable instruction of an audio reconstruction method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.

[0146] The processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run computer-readable instructions or process data stored in the memory 9a, such as computer-readable instructions for running the audio reconstruction method.

[0147] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0148] The computer device proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to the pre-processing scenarios of audio reconstruction and audio synthesis. The present application obtains the original audio stream; inputs it into a preset audio encoder to obtain the corresponding potential feature representation; inputs the potential feature representation into a single quantizer after the codebook space size is expanded, and quantizes the potential feature representation to obtain a discretized audio feature representation; uses an improved decoder after adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; uses a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performs an inverse Fourier transform according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for encoding and decoding training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.

[0149] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the audio reconstruction method as described above.

[0150] The computer-readable storage medium proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to the pre-processing scenarios of audio reconstruction and audio synthesis. The present application obtains the original audio stream; inputs it into a preset audio encoder to obtain the corresponding potential feature representation; inputs the potential feature representation into a single quantizer after the codebook space size is expanded, and quantizes the potential feature representation to obtain a discretized audio feature representation; uses an improved decoder after adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; uses a single-sideband spectrum to convert the real-valued signal to obtain a converted signal spectrum diagram; performs an inverse Fourier transform according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream. By expanding the codebook space of the quantizer, improving the decoder, and pre-adversarial training for encoding and decoding training, the high availability of the reconstructed audio is guaranteed when the audio reconstruction is actually performed.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0152] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An audio reconstruction method, characterized in that: The steps include: Get the original audio stream; Inputting the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream; Inputting the latent feature representation into a single quantizer with an expanded codebook space size, and quantizing the latent feature representation to obtain a discretized audio feature representation; Using an improved decoder that has undergone adversarial training to decode the discretized audio feature representation to obtain a real-valued signal corresponding to the discretized audio feature representation; The real-valued signal is converted using a single-sideband spectrum to obtain a converted signal spectrum diagram; An inverse Fourier transform is performed according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream.

2. The audio reconstruction method according to claim 1, characterized in that: The step of inputting the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream specifically includes: Performing an initial convolution operation on the input original audio stream by using the 1D convolution layer in the audio encoder to extract initial audio features in the original audio stream; The convolution block in the audio encoder is used to perform downsampling and filtering processing on the initial audio features, and according to the configuration of the sampling step, the key audio features corresponding to the target frequency are obtained; The key audio features are represented as potential features corresponding to the original audio stream.

3. The audio reconstruction method according to claim 1, characterized in that: Before executing the step of obtaining the original audio stream, the method further includes: Acquire an input audio stream for batch encoding and decoding training, wherein the input audio stream includes an audio stream of a real object; The input audio stream is processed by the audio encoder, the single quantizer and the improved decoder in sequence to obtain a reconstructed audio corresponding to the input audio stream; The authenticity of the reconstructed audio is estimated using a preset discriminator until the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold, and the training of the audio encoder, the single quantizer and the improved decoder is completed.

4. The audio reconstruction method according to claim 3, characterized in that: The preset discriminator includes a multi-period discriminator, a multi-resolution discriminator and a multi-scale discriminator. The step of using the preset discriminator to perform authenticity estimation on the reconstructed audio specifically includes: Step 401, using the multi-period discriminator, the multi-resolution discriminator and the multi-scale discriminator to jointly obtain the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value and the feature matching loss value corresponding to the reconstructed audio in the reconstruction process; Step 402: The quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value are used as authenticity estimation factors, and combined with a preset comprehensive calculation method, the authenticity estimation is calculated.

5. The audio reconstruction method according to claim 4, characterized in that: Before executing the step of using the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value as authenticity estimation factors and calculating the authenticity estimation in combination with a preset comprehensive calculation method, the method further includes: Obtain loss requirement conditions pre-set for the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value, respectively, wherein the loss requirement condition includes that the loss value does not exceed a set loss threshold; After step 401, the following steps are further included: Compare the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value with corresponding loss requirement conditions; If any of the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value does not meet the corresponding loss requirement conditions, the audio encoder, the single quantizer, and the improved decoder are subjected to model parameter tuning processing until the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions; If the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value all meet the corresponding loss requirement conditions, step 402 is directly executed.

6. The audio reconstruction method according to claim 4, characterized in that: The step of using the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value and the feature matching loss value as authenticity estimation factors and combining with a preset comprehensive calculation method to calculate the authenticity estimation specifically includes: Obtaining dynamic floating weights pre-set for the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value; According to the dynamic floating weights, current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point are obtained respectively; Based on the current weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the current training time point, a weighted sum algorithm is used to calculate the total loss value corresponding to the current training time point; The total loss value is used to obtain the corresponding authenticity estimate at the current training time point, wherein the total loss value is inversely proportional to the authenticity estimate.

7. The audio reconstruction method according to claim 6, characterized in that: After executing the step of obtaining the corresponding truth value at the current training time point through the total loss value, the method further includes: Determining whether the authenticity estimate of the reconstructed audio reaches a preset authenticity threshold; If the authenticity estimate of the reconstructed audio does not reach the preset authenticity threshold, the training is continued, and the latest weights corresponding to the quantization loss value, the Mel spectrum reconstruction loss value, the adversarial loss value, and the feature matching loss value at the latest training time point are obtained in an iterative calculation manner, and Based on the latest weight, a weighted sum algorithm is used to calculate the total loss value corresponding to the latest training time point, and the corresponding truthfulness estimation value at the latest training time point is obtained through the total loss value; Until the latest authenticity estimate corresponding to the reconstructed audio reaches a preset authenticity threshold, the training of the audio encoder, the single quantizer and the improved decoder is completed.

8. An audio reconstruction device, characterized in that: include: The original audio stream acquisition module is used to acquire the original audio stream; An audio encoding processing module, used for inputting the original audio stream into a preset audio encoder to obtain a potential feature representation corresponding to the original audio stream; A quantization processing module, used for inputting the potential feature representation into a single quantizer with an expanded codebook space size, and performing quantization processing on the potential feature representation to obtain a discretized audio feature representation; A decoding processing and real-valued signal acquisition module, used to decode the discretized audio feature representation using an improved decoder after adversarial training to obtain a real-valued signal corresponding to the discretized audio feature representation; A signal spectrum conversion module, used to convert the real-valued signal using a single-sideband spectrum to obtain a converted signal spectrum; The inverse Fourier transform module is used to perform inverse Fourier transform according to the phase and amplitude values ​​of the signal spectrum diagram to complete the reconstruction of the original audio stream.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio reconstruction method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio reconstruction method according to any one of claims 1 to 7 are implemented.

Images