Audio processing method, device and system and storage medium

Abstract

The invention relates to an audio processing method, device and system and a storage medium, relates to the technical field of data processing, and aims to at least solve the problem of serious distortion of output audio in related technologies. The method comprises the following steps: inputting a to-be-processed audio into an audio conversion model, so as to adopt the audio conversion model to execute the following operations: calling an encoder to encode the to-be-processed audio into a target continuous encoding signal; quantizing the target continuous coded signal vector into a target discrete coded signal based on a preset codebook; calling a decoder to decode the target discrete coded signal into a target audio composed of continuous target output coded signals based on a preset codebook; wherein the preset codebook comprises an association mapping relation between a continuous coding signal and a discrete coding signal, and the audio conversion model is trained by taking reconstruction loss, codebook loss and retention loss of coding and decoding of a coder and a decoder as constraint targets; and outputting a target audio obtained after the operation of the audio conversion model is completed.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an audio processing method, device, system and storage medium. Background Art

[0002] With the development demand of voice intelligence in various service fields such as smart medical and financial fields, audio processing has become an increasingly growing application trend, and audio coding and decoding technology has important application value in audio digital signal processing. In order to ensure that audio data can be efficiently stored, transmitted and played, in the current audio coding and decoding process, the audio signal is usually converted from analog to digital, and the data size is reduced by compression and encoding, and then restored to the analog signal by decoding and reconstruction to achieve the conversion of the audio signal into an analog signal output. Because the above analog signal is obtained based on the signal compression processing technology of reducing the data size by compression and encoding, some frequency bands will be filtered to use less bit signals, resulting in serious distortion of the output audio, especially in the case of extremely low bit numbers (such as 16 kilobits per second or lower), the above audio coding and decoding method will cause the output audio to be significantly distorted, greatly affecting the auditory effect, and has no practical value. Summary of the invention

[0003] The present invention provides an audio processing method, device, system and storage medium to at least solve the problem of serious audio distortion in the related art. The technical solution of the present invention is as follows:

[0004] According to a first aspect of an embodiment of the present invention, there is provided an audio processing method, the method comprising: inputting an audio to be processed into an audio conversion model, so as to perform the following operations using the audio conversion model: calling an encoder to encode the audio to be processed into a target continuous coded signal; based on a preset code book, vector quantizing the target continuous coded signal into a target discrete coded signal; calling a decoder to decode the target discrete coded signal into a target audio composed of a continuous target output coded signal based on the preset code book; wherein the preset code book includes an associated mapping relationship between the continuous coded signal and the discrete coded signal, and the audio conversion model is trained with reconstruction loss, codebook loss and codec retention loss of the encoder and decoder as constraint targets; and outputting the target audio obtained after the audio conversion model completes the operation.

[0005] In one implementation, the reconstruction loss represents the audio signal loss between the input sample audio input to the audio conversion model and the output audio corresponding to the output of the audio conversion model; the codebook loss represents the coding loss between the continuous coded signal output by the encoder and the discrete coded signal after vector quantization processing based on a preset codebook when the first network parameters of the encoder and the second network parameters of the decoder are determined; the retention loss represents the coding loss between the continuous coded signal output by the encoder and the discrete coded signal after vector quantization processing based on a preset codebook when the codeword values ​​corresponding to each codeword representing the association mapping relationship between the continuous coded signal and the discrete coded signal in the preset codebook are determined.

[0006] In another implementation, the constraint target includes a first constraint target and a second constraint target; before the audio to be processed is input into the audio conversion model, the method further includes: alternately performing the following training processes until a preset codebook that satisfies the first constraint target and an encoder under the first network parameters and a decoder under the second network parameters that satisfies the second constraint target are obtained: with the reconstruction loss being less than a first loss threshold and the codebook loss being less than a second loss threshold as the first constraint target, the codeword values ​​corresponding to each codeword in the preset codebook that characterizes the association mapping relationship between the continuous coding signal and the discrete coding signal are trained; and with the reconstruction loss being less than the first loss threshold and the retention loss being less than the third loss threshold as the second constraint target, the first network parameters of the encoder and the second network parameters of the decoder are trained.

[0007] In another implementation, with the reconstruction loss being less than a first loss threshold and the codebook loss being less than a second loss threshold as the first constraint target, the codeword values ​​corresponding to each codeword representing the association mapping relationship between the continuous coding signal and the discrete coding signal in the preset coding book are trained, including: when the codebook loss is greater than or equal to the second loss threshold, determining from each codeword a target codeword whose signal difference between the discrete coding signal and the continuous coding signal is greater than or equal to a first preset difference; determining a first gradient representing a change trend of the codeword value of the target codeword based on the linear relationship represented by the codeword value corresponding to the target codeword and the adjacent codewords; and adjusting the codeword value of the target codeword accordingly according to the first gradient.

[0008] In another implementation, the codeword value of the target codeword is adjusted accordingly according to the first gradient, including: when the first gradient is greater than or equal to 0, the codeword value of the target codeword is reduced; when the first gradient is less than 0, the codeword value of the target codeword is increased.

[0009] In another implementation, the first network parameters of the encoder and the second network parameters of the decoder are trained with the reconstruction loss being less than the first loss threshold and the maintenance loss being less than the third loss threshold as the second constraint target, including: determining the third gradient of the discrete coded signal after vector quantization processing of the continuous coded signal according to the second gradient of the change trend between the continuous coded signals output by the encoder; when the maintenance loss is greater than or equal to the third loss threshold, if the signal loss between the continuous coded signal and the corresponding sample continuous coded signal is greater than the fourth loss threshold, adjusting the first network parameters of the encoder according to the mapping relationship between the second gradient and the first network parameters; and, if the signal loss between the discrete coded signal and the corresponding sample discrete coding is greater than the fifth loss threshold, adjusting the second network parameters of the decoder according to the mapping relationship between the third gradient and the second network parameters.

[0010] In another implementation, with the reconstruction loss being less than a first loss threshold and the codebook loss being less than a second loss threshold as the first constraint target, the codeword values ​​corresponding to each codeword in the preset codebook that characterizes the association mapping relationship between the continuous coding signal and the discrete coding signal are trained, including: when the reconstruction loss is greater than or equal to the first loss threshold and / or the codebook loss is greater than or equal to the second loss threshold, the codeword values ​​corresponding to each codeword in the preset codebook are adjusted; with the reconstruction loss being less than the first loss threshold and the loss being kept less than a third loss threshold as the second constraint target, the first network parameters of the encoder and the second network parameters of the decoder are trained, including: when the reconstruction loss is greater than or equal to the first loss threshold and / or the loss is kept greater than or equal to the third loss threshold, the first network parameters and the second network parameters are adjusted.

[0011] According to a second aspect of an embodiment of the present invention, an audio processing device is provided, which includes: an input unit, which is used to input the audio to be processed into an audio conversion model, so as to use the audio conversion model to perform the following operations: calling the encoder to encode the audio to be processed into a target continuous coding signal; based on a preset coding book, vector quantizing the target continuous coding signal into a target discrete coding signal; calling the decoder to decode the target discrete coding signal into a target audio composed of a continuous target output coding signal based on the preset coding book; wherein the preset coding book includes an association mapping relationship between the continuous coding signal and the discrete coding signal, and the audio conversion model is trained with reconstruction loss, codebook loss and codec retention loss of the encoder and decoder as constraint targets; an output unit, which is used to output the target audio obtained after the audio conversion model completes the operation.

[0012] According to a third aspect of an embodiment of the present invention, an audio processing system is provided, which includes an encoder, a decoder and a preset code book, and the system is provided with an audio conversion model, and the system is configured to execute an audio processing method such as the first aspect and any possible implementation thereof.

[0013] According to a fourth aspect of an embodiment of the present invention, an electronic device is provided, comprising: a processor and a memory for storing processor executable instructions; wherein the processor is configured to execute executable instructions to implement an audio processing method such as the first aspect and any possible implementation thereof.

[0014] According to a fifth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which instructions are stored. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform an audio processing method such as the first aspect and any possible implementation thereof.

[0015] According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided. The computer program product includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the audio processing method of the first aspect and any possible implementation thereof.

[0016] The technical solution provided by the embodiment of the present invention brings at least the following beneficial effects: directly converting the processed audio into the output target audio according to the audio conversion model, the implementation is simple and easy to operate, no additional signal processing module is required, and the hardware cost and operation process are reduced. In the process of directly converting the processed audio into the target audio using the audio conversion model, based on the preset code book, the target continuous coding signal in the encoder is vector quantized into a target discrete coding signal to ensure that the audio signals of each frequency band in the original audio can be converted into corresponding target discrete coding signals, thereby ensuring that the target discrete coding signal input to the decoder can more completely represent the original audio information, so that the decoder can better restore the original audio information according to the preset code book, thereby ensuring that the output audio is more realistic. At the same time, the audio conversion model is trained with three dimensional constraint targets: reconstruction loss, code book loss, and retention loss of the encoding and decoding of the encoder and the decoder, thereby ensuring the accuracy of the encoder, decoder and preset code book, thereby improving the audio conversion accuracy of the audio conversion model.

[0017] The above-mentioned audio signal conversion process is based on the audio conversion model to directly convert the output audio to be processed into the target audio, instead of relying on the signal processing process to filter and compress the signal as in the related technology. It can better restore and retain the audio information of the original audio, thereby reducing the output audio distortion and improving the auditory effect of the output audio.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0020] Figure 1 is a schematic diagram of an audio processing system according to an exemplary embodiment;

[0021] Figure 2 is a flowchart of an audio processing method according to an exemplary embodiment;

[0022] Figure 3 is a schematic diagram of an audio processing process according to an exemplary embodiment;

[0023] Figure 4 is a block diagram of an audio processing device according to an exemplary embodiment;

[0024] Figure 5 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0027] Before introducing the audio processing method provided in the embodiment of the present application in detail, the application scenarios and implementation environment involved in the embodiment of the present application are briefly introduced.

[0028] First, a brief introduction to the application scenarios involved in this application is given.

[0029] With the development demand of voice intelligence in various service fields such as smart medical and financial fields, audio processing has become an increasingly growing application trend, and audio coding and decoding technology has important application value in audio digital signal processing. In order to ensure that audio data can be efficiently stored, transmitted and played, in the current audio coding and decoding process, the audio signal is usually converted from analog to digital, and the data size is reduced by compression and encoding, and then restored to the analog signal by decoding and reconstruction to achieve the conversion of the audio signal into an analog signal output. Because the above analog signal is obtained based on the signal compression processing technology of reducing the data size by compression and encoding, some frequency bands will be filtered to use less bit signals, resulting in serious distortion of the output audio, especially in the case of extremely low bit numbers (such as 16 kilobits per second or lower), the above audio coding and decoding method will cause the output audio to be significantly distorted, greatly affecting the auditory effect, and has no practical value.

[0030] Generally speaking, there is a lot of redundant information in audio signals, and directly transmitting the original audio signals is costly and impractical. Traditional signal processing methods generally reduce the total number of signal bits by modeling the signal based on the human ear and psychoacoustics, encoding the frequency bands that the human ear is not sensitive to using fewer bits.

[0031] Research has found that when the number of bits is extremely low, the current audio coding and decoding methods will cause obvious distortion of the audio, greatly affecting the auditory effect and having no practical value. With the development of deep learning technology, data-driven audio signal processing methods have become possible. The Vector Quantized Variational AutoEncoder (VQ-VAE) can encode data into fixed discrete codes by quantizing the intermediate latent variables of VAE (i.e., the generative model), and restore the discrete codes to data through the decoder.

[0032] In response to the above problems, the present application proposes an audio processing method, which directly converts the audio to be processed into the target audio to be output according to the audio conversion model, which is simple and easy to operate, does not require an additional signal processing module, and reduces hardware costs and operation procedures. In the process of directly converting the audio to be processed into the target audio using the audio conversion model, based on the preset code book, the target continuous coding signal in the encoder is vector-quantized into a target discrete coding signal to ensure that the audio signals of each frequency band in the original audio can be converted into corresponding target discrete coding signals, thereby ensuring that the target discrete coding signal input to the decoder can more completely represent the original audio information, so that the decoder can better restore the original audio information according to the preset code book, thereby ensuring that the output audio is more realistic. At the same time, the audio conversion model is trained with three dimensional constraint targets: reconstruction loss, code book loss, and retention loss of the encoding and decoding of the encoder and the decoder, thereby ensuring the accuracy of the encoder, decoder and preset code book, thereby improving the audio conversion accuracy of the audio conversion model.

[0033] Secondly, the implementation architecture involved in this application is briefly introduced below.

[0034] Figure 1 is a schematic diagram of an audio processing system 10 provided by the present disclosure. Figure 1 As shown, the audio processing system 10 includes an encoder 101 and a decoder 102, and is provided with a preset codebook.

[0035] In some implementations, the encoder 101 and the decoder 102 may be connected via a wired network or a wireless network. The preset codebook may be referred to as a codebook or a codebook.

[0036] In some embodiments, the audio processing system 10 can be set in a terminal device in a service platform such as a medical consultation platform, a financial platform, etc.

[0037] Specifically, in the process of transmitting signals, the sender of the speech signal or audio signal needs a trained encoder and a preset codebook, while the receiver needs a preset codebook and a decoder paired with the encoder. The voice sender can encode each coded signal in the codebook into an integer, that is, the coded signal can be represented by a code, so a code containing 2 kThe codebook of codes (i.e., codewords) can be encoded by k bits per frame. Furthermore, for the input speech audio signal with a sampling rate of 16 kHz, it is encoded by a downsampling encoder, and the resulting frame length is 10 milliseconds per frame, and 4096 discrete codes are used in the codebook. Therefore, the bit rate in the proposed codec model is 1.2 kilobits per second, which is much lower than the bit rate of the related signal processing algorithm. The receiver queries the code corresponding to each frame from the codebook according to the bits of the received discrete coded signal, and inputs it into the decoder to obtain the final restored speech audio signal.

[0038] In other embodiments, the terminal device may be a mobile phone, tablet computer, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR)\virtual reality (VR) device, etc., which can install and use content community applications (such as Kuaishou), and the present disclosure does not impose any special restrictions on the specific form of the terminal device. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device.

[0039] Furthermore, this application does not limit the number and type of terminal devices.

[0040] The audio processing method provided in the embodiment of the present application can be applied to the aforementioned Figure 1 The audio processing system in the implementation architecture shown. For ease of understanding, the audio processing method provided by the present application is specifically introduced below in conjunction with the accompanying drawings.

[0041] Figure 2 is a flowchart of an audio processing method according to an exemplary embodiment. Figure 2 As shown, the audio processing method includes the following steps.

[0042] S21, input the audio to be processed into the audio conversion model to perform the following operations using the audio conversion model: call the encoder to encode the audio to be processed into a target continuous coded signal; based on a preset code book, vector quantize the target continuous coded signal into a target discrete coded signal; call the decoder to decode the target discrete coded signal into a target audio composed of a continuous target output coded signal based on a preset code book.

[0043] Among them, the preset code book includes the correlation mapping relationship between the continuous coding signal and the discrete coding signal, and the audio conversion model is trained with the reconstruction loss, the code book loss and the codec preservation loss of the encoder and the decoder as the constraint targets.

[0044] The reconstruction loss characterizes the audio signal loss between the input sample audio input to the audio conversion model and the output audio corresponding to the output of the audio conversion model.

[0045] The codebook loss represents the coding loss between the continuous coded signal output by the encoder and the discrete coded signal after vector quantization based on the preset codebook when the first network parameters of the encoder and the second network parameters of the decoder are determined.

[0046] The loss characterization maintains the coding loss between the continuous coded signal output by the encoder and the discrete coded signal after vector quantization processing based on the preset code book, when the code word values ​​corresponding to the code words representing the association mapping relationship between the continuous coded signal and the discrete coded signal in the preset code book are determined.

[0047] The audio conversion model can be a VQ-VAE model that is transformed based on vector quantization on the basis of the VAE generation model, that is, a vectorized generation model.

[0048] The above-mentioned vector quantization is a technology for discretizing continuous data. Specifically, it realizes data compression and feature extraction by mapping the input data to a set of discrete code vectors (Codebook). The set of discrete code vectors is the discrete code signal in this application.

[0049] In one embodiment, in order to extract and compress features by reducing the dimensionality of the data while retaining important information, the encoder is a downsampling encoder. The downsampling encoder is used to extract global features of the speech audio, which are then passed to the upsampling layer (Decoder) for pixel-level classification.

[0050] The downsampling encoder is usually composed of multiple layers of convolutional layers and pooling layers. The convolutional layer extracts local features by performing convolution operations on the input audio to be processed by sliding the convolution kernel. With each pooling layer, the size of the feature map decreases, while the number of channels (the number of feature maps) usually increases. This hierarchical structure enables the network to gradually capture higher-level features.

[0051] The downsampling encoder samples the processed audio according to sampling points with Gaussian distribution to mark the discrete coded signal corresponding to the continuous coded signal.

[0052] In one embodiment, if Figure 3 As shown, the audio processing process is described as follows, taking the audio to be processed as speech.

[0053] In the process of model application, the input speech passes through the encoder based on the convolution model, which includes stacked convolution layers and multi-step convolution layers (strided convolution) for downsampling, and obtains the continuous code after downsampling. The discrete code closest to the continuous code is found in the code book, and the discrete code is input into the decoder based on the convolution layer and transposed convolution to obtain the restored speech signal. Among them, the code book includes code words such as code 1, code 2, code 3, code k, etc.

[0054] During the model training process, since the process from continuous coding to discrete coding is not differentiable, the gradient at the continuous coding needs to be copied to the discrete coding. A combination of three loss functions is used during the training process. The first is the reconstruction loss of the input audio and output audio. The sampling points of the speech audio signal are generally Gaussian distributed, so the MSE (Mean Squared Error) loss function is used to determine the reconstruction loss; codebook loss and retention loss are also used.

[0055] Specifically, the codebook loss is the MSE loss between the continuous and discrete codes of the codebook when the fixed codec layer is trained, where there is a linear relationship between the values ​​of the codewords in the codebook. The retention loss is the MSE loss between the continuous and discrete codes of the codec when the fixed codebook is trained. These two losses can maintain the stability of the codebook during training while training the codebook and the codec.

[0056] In the above implementation steps, in the process of using the audio conversion model to directly convert the audio to be processed into the target audio, based on the preset code book, the target continuous coding signal in the encoder is vector-quantized into a target discrete coding signal to ensure that the audio signals of each frequency band in the original audio can be converted into corresponding target discrete coding signals, thereby ensuring that the target discrete coding signals input to the decoder can more completely represent the original audio information, so that the decoder can better restore the original audio information according to the preset code book, thereby ensuring that the output audio is more realistic.

[0057] At the same time, the audio conversion model is trained with three dimensional constraint objectives: reconstruction loss, codebook loss, and codec preservation loss of the encoder and the decoder, thereby ensuring the accuracy of the encoder, decoder and preset codebook, thereby improving the audio conversion accuracy of the audio conversion model.

[0058] S22, output the target audio obtained after the audio conversion model completes the operation.

[0059] The above method directly converts the audio to be processed into the target audio based on the audio conversion model.

[0060] Through the above embodiments, the audio signal conversion process is based on the audio conversion model to directly convert the output audio to be processed into the target audio, instead of relying on the signal processing process to filter and compress the signal as in the related technology. It can better restore and retain the audio information of the original audio, thereby reducing the output audio distortion and improving the auditory effect of the output audio.

[0061] As a refinement and extension of the specific implementation method of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, the embodiment of the present application provides another audio processing method.

[0062] In one embodiment, the constraint target includes a first constraint target and a second constraint target. Based on this, before implementing step S21, the audio conversion model is first trained, and the following training steps are specifically performed alternately until a preset codebook that meets the first constraint target and an encoder under the first network parameters and a decoder under the second network parameters that meet the second constraint target are obtained.

[0063] First, with the reconstruction loss being less than the first loss threshold and the codebook loss being less than the second loss threshold as the first constraint target, the codeword values ​​corresponding to the codewords representing the correlation mapping relationship between the continuous coding signal and the discrete coding signal in the preset coding book are trained.

[0064] Secondly, taking the reconstruction loss less than the first loss threshold and the maintenance loss less than the third loss threshold as the second constraint target, the first network parameters of the encoder and the second network parameters of the decoder are trained.

[0065] Through the above implementation steps, the audio conversion model includes the trained encoder and decoder and the trained preset code book to ensure the accuracy of the audio conversion model.

[0066] As an implementation method, the specific steps of training the codeword values ​​corresponding to each codeword in the preset codebook are as follows.

[0067] First, when the codebook loss is greater than or equal to a second loss threshold, a target codeword whose signal difference between a discrete coded signal and a continuous coded signal is greater than or equal to a first preset difference is determined from each codeword.

[0068] Secondly, based on the linear relationship represented by the codeword value corresponding to the target codeword and the adjacent codewords, a first gradient representing the change trend of the codeword value of the target codeword is determined.

[0069] It can be understood that, based on the linear relationship represented by the codeword values ​​corresponding to the codewords, the first gradient representing the change trend between the codeword values ​​of the codewords is determined.

[0070] Third, according to the first gradient, the codeword value of the target codeword is adjusted accordingly.

[0071] In some embodiments, the first gradient is represented by a derivative obtained by differentiating a curve of a linear relationship.

[0072] Specifically, the codeword value of the target codeword is adjusted accordingly according to the first gradient, including the following two cases.

[0073] The first one is to reduce the codeword value of the target codeword when the first gradient is greater than or equal to 0.

[0074] Specifically, when the first gradient is greater than or equal to 0, it indicates that the codeword value is too large, and the step size may be reduced according to a preset value to reduce the codeword value of the target codeword.

[0075] The second method is to increase the code value of the target code word when the first gradient is less than 0.

[0076] Specifically, when the first gradient is less than 0, it indicates that the codeword value is too small, and the step size can be increased according to a preset value to increase the codeword value of the target codeword.

[0077] As an implementation manner, the steps of training the first network parameters of the encoder and the second network parameters of the decoder are specifically as follows.

[0078] First, according to the second gradient of the change trend between the continuous coded signals output by the encoder, a third gradient of the discrete coded signal after the vector quantization processing of the continuous coded signal is determined.

[0079] Because during the training process, since the process from continuous coding signal to discrete coding signal is not differentiable, the gradient mapping of the continuous coding signal is copied to the corresponding discrete coding signal to ensure that during the training process.

[0080] Secondly, while keeping the loss greater than or equal to the third loss threshold, if the signal loss between the continuous coding signal and the corresponding sample continuous coding signal is greater than the fourth loss threshold, the first network parameter of the encoder is adjusted according to the mapping relationship between the second gradient and the first network parameter; and if the signal loss between the discrete coding signal and the corresponding sample discrete coding is greater than the fifth loss threshold, the second network parameter of the decoder is adjusted according to the mapping relationship between the third gradient and the second network parameter.

[0081] In some embodiments, the above-mentioned training of the codeword values ​​corresponding to each codeword representing the association mapping relationship between the continuous coding signal and the discrete coding signal in the preset coding book with the reconstruction loss being less than the first loss threshold and the codebook loss being less than the second loss threshold as the first constraint target specifically includes the following steps: when the reconstruction loss is greater than or equal to the first loss threshold and / or the codebook loss is greater than or equal to the second loss threshold, adjusting the codeword values ​​corresponding to each codeword in the preset coding book.

[0082] Taking the reconstruction loss less than the first loss threshold and the maintenance loss less than the third loss threshold as the second constraint target, the first network parameters of the encoder and the second network parameters of the decoder are trained, specifically including: when the reconstruction loss is greater than or equal to the first loss threshold and / or the maintenance loss is greater than or equal to the third loss threshold, the first network parameters and the second network parameters are adjusted.

[0083] In order to realize the above functions, the audio processing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0084] The present disclosure also provides a Figure 4 The audio processing device shown includes: an input unit 401, an output unit 402 and a training unit 403.

[0085] The input unit 401 is used to input the audio to be processed into the audio conversion model, so as to use the audio conversion model to perform the following operations: calling the encoder to encode the audio to be processed into a target continuous coded signal; based on a preset code book, vector quantizing the target continuous coded signal into a target discrete coded signal; calling the decoder to decode the target discrete coded signal into a target audio composed of a continuous target output coded signal based on the preset code book; wherein the preset code book includes an associated mapping relationship between the continuous coded signal and the discrete coded signal, and the audio conversion model is trained with reconstruction loss, codebook loss and codec retention loss of the encoder and decoder as constraint targets.

[0086] The output unit 402 is used to output the target audio obtained after the audio conversion model completes the operation.

[0087] In one embodiment, the reconstruction loss represents the audio signal loss between the input sample audio input to the audio conversion model and the output audio corresponding to the output of the audio conversion model; the codebook loss represents the coding loss between the continuous coded signal output by the encoder and the discrete coded signal after vector quantization processing based on a preset codebook when the first network parameters of the encoder and the second network parameters of the decoder are determined; the retention loss represents the coding loss between the continuous coded signal output by the encoder and the discrete coded signal after vector quantization processing based on a preset codebook when the codeword values ​​corresponding to each codeword representing the association mapping relationship between the continuous coded signal and the discrete coded signal in the preset codebook are determined.

[0088] In another embodiment, the constraint target includes a first constraint target and a second constraint target; and further includes a training unit 403, which is used to alternately perform the following training processes until a preset coding book that satisfies the first constraint target and an encoder under the first network parameters and a decoder under the second network parameters that satisfy the second constraint target are obtained: with the reconstruction loss being less than a first loss threshold and the codebook loss being less than a second loss threshold as the first constraint target, the codeword values ​​corresponding to each codeword in the preset coding book that represents the association mapping relationship between the continuous coding signal and the discrete coding signal are trained; and with the reconstruction loss being less than the first loss threshold and the retention loss being less than a third loss threshold as the second constraint target, the first network parameters of the encoder and the second network parameters of the decoder are trained.

[0089] In another embodiment, the training unit 403 is specifically used to: when the codebook loss is greater than or equal to a second loss threshold, determine, from each codeword, a target codeword in which the signal difference between the discrete coding signal and the continuous coding signal is greater than or equal to a first preset difference; based on the linear relationship represented by the codeword value corresponding to the target codeword and the adjacent codewords, determine a first gradient representing the change trend of the codeword value of the target codeword; and according to the first gradient, adjust the codeword value of the target codeword accordingly.

[0090] In another implementation, the codeword value of the target codeword is adjusted accordingly according to the first gradient, including: when the first gradient is greater than or equal to 0, the codeword value of the target codeword is reduced; when the first gradient is less than 0, the codeword value of the target codeword is increased.

[0091] In another embodiment, the training unit 403 is specifically used to: determine the third gradient of the discrete coded signal after vector quantization processing of the continuous coded signal according to the second gradient of the change trend between the continuous coded signals output by the encoder; while keeping the loss greater than or equal to the third loss threshold, if the signal loss between the continuous coded signal and the corresponding sample continuous coded signal is greater than the fourth loss threshold, adjust the first network parameter of the encoder according to the mapping relationship between the second gradient and the first network parameter; and if the signal loss between the discrete coded signal and the corresponding sample discrete coding is greater than the fifth loss threshold, adjust the second network parameter of the decoder according to the mapping relationship between the third gradient and the second network parameter.

[0092] In another embodiment, the training unit 403 is specifically used to: when the reconstruction loss is greater than or equal to the first loss threshold and / or the codebook loss is greater than or equal to the second loss threshold, adjust the codeword values ​​corresponding to each codeword in the preset coding book; with the reconstruction loss being less than the first loss threshold and the loss being kept less than the third loss threshold as the second constraint target, train the first network parameters of the encoder and the second network parameters of the decoder, including: when the reconstruction loss is greater than or equal to the first loss threshold and / or the loss is kept greater than or equal to the third loss threshold, adjust the first network parameters and the second network parameters.

[0093] Regarding the device in the above embodiment, the specific manner in which each unit module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0094] Figure 5 is a schematic diagram of an electronic device provided by the present application. Figure 5 The electronic device 50 may include at least one processor 501 and a memory 503 for storing processor executable instructions. The processor 501 is configured to execute instructions in the memory 503 to implement the audio processing method in the following embodiments.

[0095] In addition, the electronic device 50 may further include a communication bus 502 , at least one communication interface 504 , an input device 506 , and an output device 505 .

[0096] The processor 501 may be a central processing unit (CPU), a microprocessing unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0097] Communication bus 502 may include a pathway for transmitting information between the above-mentioned components.

[0098] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0099] The input device 506 is used to receive input signals and the output device 505 is used to output signals.

[0100] The memory 503 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory may be independent and connected to the processing unit via a bus. The memory may also be integrated with the processing unit.

[0101] The memory 503 is used to store instructions for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the instructions stored in the memory 503, so as to realize the functions in the method of the present application.

[0102] In a specific implementation, as an embodiment, the processor 501 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 in.

[0103] In a specific implementation, as an embodiment, the electronic device 50 may include multiple processors, such as Figure 5 501 and processor 507 in the embodiment of the present invention. Each of these processors may be a single-CPU processor or a multi-CPU processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0104] The electronic equipment Figure 5The system includes: a processor 501 and a memory 503 for storing executable instructions of the processor 501; wherein the processor 501 is configured to execute the executable instructions to implement the audio processing method of any possible implementation as described above. And the same technical effect can be achieved, so it will not be described here to avoid repetition.

[0105] The embodiment of the present application also provides a computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an audio processing device or an electronic device, the audio processing device or the electronic device can execute the audio processing method of any possible implementation as described above. And the same technical effect can be achieved, so it will not be repeated here to avoid repetition.

[0106] The embodiment of the present application also provides a computer program product, including a computer program or an instruction, which is executed by a processor as an audio processing method of any possible implementation method described above, and can achieve the same technical effect, which will not be described here to avoid repetition.

[0107] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0108] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that: The method comprises: Input the audio to be processed into the audio conversion model, so as to perform the following operations using the audio conversion model: call the encoder to encode the audio to be processed into a target continuous coding signal; based on a preset code book, vector quantize the target continuous coding signal into a target discrete coding signal; call the decoder to decode the target discrete coding signal into a target audio composed of a continuous target output coding signal based on the preset code book; wherein the preset code book includes an associated mapping relationship between continuous coding signals and discrete coding signals, and the audio conversion model is trained with reconstruction loss, code book loss and codec retention loss of the encoder and the decoder as constraint targets; The target audio obtained after the audio conversion model completes the operation is output.

2. The audio processing method according to claim 1, characterized in that: The reconstruction loss represents the audio signal loss between the input sample audio input by the audio conversion model and the output audio corresponding to the output of the audio conversion model; the codebook loss represents the coding loss between the continuous coding signal output by the encoder and the discrete coding signal after vector quantization processing based on the preset coding book when the first network parameters of the encoder and the second network parameters of the decoder are determined; the retention loss represents the coding loss between the continuous coding signal output by the encoder and the discrete coding signal after vector quantization processing based on the preset coding book when the codeword values ​​corresponding to each codeword representing the association mapping relationship between the continuous coding signal and the discrete coding signal in the preset codebook are determined.

3. The audio processing method according to claim 2, characterized in that: The constraint target includes a first constraint target and a second constraint target; before inputting the to-be-processed audio into the audio conversion model, the method further includes: The following training processes are performed alternately until the preset codebook that satisfies the first constraint target and the encoder under the first network parameters and the decoder under the second network parameters that satisfy the second constraint target are obtained: Taking the reconstruction loss being less than a first loss threshold and the codebook loss being less than a second loss threshold as the first constraint target, training the codeword values ​​corresponding to the codewords representing the association mapping relationship between the continuous coding signal and the discrete coding signal in the preset codebook; And, taking the reconstruction loss being less than the first loss threshold and the keeping loss being less than the third loss threshold as the second constraint target, the first network parameters of the encoder and the second network parameters of the decoder are trained.

4. The audio processing method according to claim 3, characterized in that: The step of training the codeword values ​​corresponding to the codewords representing the association mapping relationship between the continuous coding signal and the discrete coding signal in the preset codebook by taking the reconstruction loss less than the first loss threshold and the codebook loss less than the second loss threshold as the first constraint target includes: When the codebook loss is greater than or equal to the second loss threshold, determining, from the codewords, a target codeword whose signal difference between the discrete coded signal and the continuous coded signal is greater than or equal to a first preset difference; Determining a first gradient representing a change trend of the codeword value of the target codeword based on a linear relationship represented by a codeword value corresponding to the target codeword and an adjacent codeword; According to the first gradient, the codeword value of the target codeword is correspondingly adjusted.

5. The audio processing method according to claim 4, characterized in that: The step of correspondingly adjusting the codeword value of the target codeword according to the first gradient includes: When the first gradient is greater than or equal to 0, reducing the codeword value of the target codeword; When the first gradient is less than 0, the codeword value of the target codeword is increased.

6. The audio processing method according to claim 3, characterized in that: The step of training the first network parameters of the encoder and the second network parameters of the decoder with the reconstruction loss being less than the first loss threshold and the maintenance loss being less than the third loss threshold as the second constraint target comprises: Determining a third gradient of a discrete coded signal after vector quantization processing of the continuous coded signal according to a second gradient of a change trend between continuous coded signals output by the encoder; When the retention loss is greater than or equal to the third loss threshold, if the signal loss between the continuous coding signal and the corresponding sample continuous coding signal is greater than the fourth loss threshold, the first network parameter of the encoder is adjusted according to the mapping relationship between the second gradient and the first network parameter; and if the signal loss between the discrete coding signal and the corresponding sample discrete coding is greater than the fifth loss threshold, the second network parameter of the decoder is adjusted according to the mapping relationship between the third gradient and the second network parameter.

7. The audio processing method according to any one of claims 3 to 6, characterized in that: The training of codeword values ​​corresponding to each codeword representing the association mapping relationship between the continuous coding signal and the discrete coding signal in the preset coding book with the reconstruction loss being less than the first loss threshold and the codebook loss being less than the second loss threshold as the first constraint target includes: when the reconstruction loss is greater than or equal to the first loss threshold and / or the codebook loss is greater than or equal to the second loss threshold, adjusting the codeword values ​​corresponding to each codeword in the preset coding book; The step of training the first network parameters of the encoder and the second network parameters of the decoder with the reconstruction loss being less than the first loss threshold and the maintenance loss being less than the third loss threshold as the second constraint target comprises: When the reconstruction loss is greater than or equal to the first loss threshold and / or the retention loss is greater than or equal to the third loss threshold, the first network parameter and the second network parameter are adjusted.

8. An audio processing device, characterized in that: The device comprises: An input unit is used to input the audio to be processed into an audio conversion model, so as to perform the following operations using the audio conversion model: calling an encoder to encode the audio to be processed into a target continuous coded signal; based on a preset codebook, vector quantizing the target continuous coded signal into a target discrete coded signal; calling a decoder to decode the target discrete coded signal into a target audio composed of a continuous target output coded signal based on the preset codebook; wherein the preset codebook includes an associated mapping relationship between a continuous coded signal and a discrete coded signal, and the audio conversion model is trained with reconstruction loss, codebook loss, and codec retention loss of the encoder and the decoder as constraint targets; An output unit is used to output the target audio obtained after the audio conversion model completes the operation.

9. An audio processing system, characterized in that: The system comprises an encoder, a decoder and a preset codebook, and the system is provided with an audio conversion model. The system is configured to execute the audio processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the audio processing method according to any one of claims 1 to 7.

Images