Processing method of audio codec model, audio codec method and related device

Abstract

Embodiments of the present application disclose a processing method of an audio coding and decoding model, an audio coding and decoding method and related devices, which are used for accurately matching the codebook layer number of layer-by-layer residual coding according to different sample audios, and fully utilizing the hierarchical quantization capability of cascaded codebooks. The method comprises: encoding a sample audio by an encoder of an initial audio coding and decoding model to obtain audio features; determining a target codebook number required for quantization coding of the audio features based on the audio features; controlling a quantizer in the initial audio coding and decoding model to perform quantization processing on the audio features based on the target codebook number to obtain quantized features; decoding the quantized features by a decoder of the initial audio coding and decoding model to obtain reconstructed audio; and performing parameter optimization on the initial audio coding and decoding model based on the difference between the sample audio and the reconstructed audio to obtain a target audio coding and decoding model.

Description

Technical Field

[0001] This invention relates to the field of audio data processing technology, and in particular to a method for processing an audio encoding and decoding model, an audio encoding and decoding method, and related apparatus. Background Technology

[0002] As the foundational technology of music platforms, audio encoding and decoding efficiency directly impacts user experience and computing costs. Especially in weak network environments such as subways, high-speed trains, and basements, efficient encoders can reduce user waiting time caused by network latency, save bandwidth, and thus save on machine costs.

[0003] As a high-bitrate scene, music audio requires much more information to be encoded than speech signals, thus having greater compression potential. The best AI codec model for music audio scene compression in the current technology comes from Descript Audio Codec (DAC). This model realizes the compression, encoding and decoding of audio signals through four main modules: convolutional encoder, quantizer, deconvolutional decoder and discriminator.

[0004] Although existing DAC models can achieve good compression results, they use a uniform encoding mode for both simple and complex signals. In this uniform encoding mode, the quantizer encodes both simple and complex signals with the same number of codebooks, resulting in a low utilization rate of the codebook number in simple signals. In other words, existing DAC models can be further compressed in terms of the number of encoded bits. Summary of the Invention

[0005] This invention provides an audio encoding and decoding model processing method, an audio encoding and decoding method, and related apparatus, which are used to accurately match the number of codebook layers of layer-by-layer residual encoding according to different sample audio, and make full use of the layered quantization capability of the concatenated codebook.

[0006] The first aspect of this application provides a method for processing an audio codec model, the method comprising:

[0007] The sample audio is encoded by the encoder of the initial audio codec model to obtain audio features;

[0008] Based on the audio features, determine the number of target codebooks required to quantize and encode the audio features;

[0009] Based on the target codebook quantity, the quantizer in the initial audio codec model is controlled to quantize the audio features to obtain quantized features;

[0010] The quantization features are decoded by the decoder of the initial audio codec model to obtain the reconstructed audio;

[0011] Based on the differences between the sample audio and the reconstructed audio, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

[0012] As an optional embodiment, the method further includes:

[0013] Using the target codebook size, the expected bit loss is calculated, and the expected bit loss is used to characterize the compression ratio of the encoder and the quantizer;

[0014] The step of optimizing the parameters of the initial audio codec model based on the difference between the sample audio and the reconstructed audio to obtain the target audio codec model includes:

[0015] Based on the difference between the sample audio and the reconstructed audio, and the expected bit loss, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

[0016] As an optional embodiment, determining the number of target codebooks required for quantization encoding of the audio features based on the audio features includes:

[0017] The audio features are input into a convolutional neural network to obtain the probability value of i concatenated codebooks required when the convolutional neural network encodes the audio features, where i is less than or equal to the total number N of concatenated codebooks built into the quantizer;

[0018] Based on the probability values ​​of the number of i concatenated codebooks, the number of target codebooks required to perform quantization encoding on the audio features is calculated.

[0019] As an optional embodiment, the method further includes:

[0020] Using the probability values ​​of the number of i concatenated codebooks and preset sparsity control parameters, a regularization loss is calculated. The regularization loss is used to prevent the audio codec model from randomly selecting the number of layers of the concatenated codebook.

[0021] The step of optimizing the parameters of the initial audio codec model based on the difference between the sample audio and the reconstructed audio to obtain the target audio codec model includes:

[0022] Based on the difference between the sample audio and the reconstructed audio, and the regularization loss, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

[0023] As an optional embodiment, the step of controlling the quantizer in the initial audio codec model to quantize the audio features based on the target codebook quantity to obtain quantized features includes:

[0024] The target codebook quantity is input into the quantizer to obtain the discrete quantization index output by the quantizer after performing residual vector quantization on the audio features using the target codebook quantity;

[0025] The step of decoding the quantization features using the decoder of the initial audio codec model to obtain the reconstructed audio includes:

[0026] The discrete quantization index is input to the decoder, which decodes the discrete quantization index to restore the reconstructed audio features and restores the reconstructed audio features to the reconstructed audio.

[0027] As an optional embodiment, the step of optimizing the parameters of the initial audio codec model based on the difference between the sample audio and the reconstructed audio to obtain the target audio codec model includes:

[0028] The reconstruction loss is calculated using the sample audio and the reconstructed audio.

[0029] The reconstruction residual loss is calculated using the audio features and the reconstructed audio features.

[0030] The initial audio codec model is optimized using the reconstruction loss, the reconstruction residual loss, and the backpropagation algorithm to obtain the target audio codec model.

[0031] As an optional embodiment, the audio codec model includes a DAC audio codec model or an EnCodec audio codec model.

[0032] A second aspect of this application provides an audio encoding / decoding method, applied to a target audio encoding / decoding model provided in the first aspect of this application, the method comprising:

[0033] The target audio is encoded by the encoder of the target audio codec model to obtain audio features;

[0034] Based on the audio features, determine the number of target codebooks required to quantize and encode the audio features;

[0035] Based on the target codebook quantity, the quantizer in the target audio codec model is controlled to quantize the audio features to obtain quantized features;

[0036] The quantization features are decoded by the decoder of the target audio codec model to obtain the reconstructed audio corresponding to the target audio.

[0037] As an optional embodiment, determining the number of target codebooks required for quantization encoding of the audio features based on the audio features includes:

[0038] The audio features are input into a convolutional neural network to obtain the probability value of i concatenated codebooks required when the convolutional network encodes the audio features, where i is less than or equal to the total number N of concatenated codebooks built into the quantizer.

[0039] Based on the probability values ​​of the number of i concatenated codebooks, the number of target codebooks required to perform quantization encoding on the audio features is calculated.

[0040] As an optional embodiment, the step of controlling the quantizer in the initial audio codec model to quantize the audio features based on the target codebook quantity to obtain quantized features includes:

[0041] The target codebook quantity is input into the quantizer to obtain the discrete quantization index output by the quantizer after performing residual vector quantization on the audio features using the target codebook quantity;

[0042] The step of decoding the quantization features using the decoder of the initial audio codec model to obtain the reconstructed audio includes:

[0043] The discrete quantization index is input to the decoder, which decodes the discrete quantization index to restore the reconstructed audio features and restores the reconstructed audio features to the reconstructed audio.

[0044] A third aspect of this application provides a computer device including a processor, characterized in that, when the processor executes a computer program stored in a memory, it is used to implement the audio codec processing method provided in the first aspect of this application, or the audio codec method provided in the second aspect of this application.

[0045] A fourth aspect of this application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is used to implement the audio codec processing method provided in the first aspect of this application, or the audio codec method provided in the second aspect of this application.

[0046] The fifth aspect of this application provides a computer program product having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it is used to implement the audio codec processing method provided in the first aspect of this application, or the audio codec method provided in the second aspect of this application.

[0047] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages:

[0048] After obtaining audio features using the encoder, this embodiment further determines the number of target codebooks required to quantize the audio features based on these features. This allows the quantizer to encode the audio features using the number of target codebooks matching the sample audio. Compared to a fixed number of codebook layers, this approach can accurately match the layer requirements of layer-by-layer residual coding based on different sample audio, fully utilizing the layered quantization capability of the concatenated codebook, effectively reducing residual errors in the audio feature quantization process, and improving audio coding fidelity. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the architecture of the audio encoding / decoding model processing system in the embodiments of this application;

[0050] Figure 2 This is a schematic diagram of an embodiment of the audio encoding / decoding model processing method in this application.

[0051] Figure 3 This is a schematic diagram of the audio encoding / decoding model in the embodiments of this application;

[0052] Figure 4 For this application Figure 2 Detailed steps of steps 202 and 205 in the embodiment;

[0053] Figure 5 This is a schematic diagram of another embodiment of the audio encoding / decoding model processing method in this application.

[0054] Figure 6 For this application Figure 2 Another detailed step of steps 203, 204 and 205 in the embodiment;

[0055] Figure 7 This is a schematic diagram of one embodiment of the audio encoding / decoding method in this application;

[0056] Figure 8 For this application Figure 7 Detailed steps of step 702 in the embodiment;

[0057] Figure 9 For this application Figure 7Detailed steps of steps 703 and 704 in the embodiment. Detailed Implementation

[0058] This invention provides a method for processing an audio codec model, an audio codec method, and related apparatus for further compressing the number of encoded bits and improving the compression effect of audio.

[0059] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0060] The terms "first," "second," "third," "fourth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] This application provides a method for processing an audio codec model. The general principle of this method is as follows: The encoder of an initial audio codec model encodes sample audio to obtain audio features; based on the audio features, the target codebook quantity required for quantization encoding of the audio features is determined; based on the target codebook quantity, the quantizer in the initial audio codec model is controlled to quantize the audio features to obtain quantized features; the decoder of the initial audio codec model decodes the quantized features to obtain reconstructed audio; based on the difference between the sample audio and the reconstructed audio, the parameters of the initial audio codec model are optimized to obtain a target audio codec model.

[0062] After obtaining audio features using the encoder, this embodiment further determines the number of target codebooks required to quantize the audio features based on these features. This allows the quantizer to encode the audio features using the number of target codebooks matching the sample audio. Compared to a fixed number of codebook layers, this approach can accurately match the layer requirements of layer-by-layer residual coding based on different sample audio, fully utilizing the layered quantization capability of the concatenated codebook, effectively reducing residual errors in the audio feature quantization process, and improving audio coding fidelity.

[0063] To better implement the above-mentioned audio codec model processing scheme, this application provides an audio codec model processing system. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the architecture of an audio codec processing system provided in an embodiment of this application. The audio codec processing system may include at least one terminal device 101 and a server 102. The terminal device 101 may have different types of applications installed, such as search engines, audio players, instant messaging applications, live streaming applications, conferencing applications, etc. The terminal device 101 may be a smartphone, tablet, laptop, desktop computer, smart vehicle, etc. The server 102 may be used to store audio and image data generated by different types of applications on the terminal device 101. The server 102 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc.

[0064] The audio codec model processing method is executed by either terminal device 101 or server 102. When terminal device 101 executes the audio codec model processing method, the sample audio generated by terminal device 101 in different types of applications can be included in the server. When terminal device 101 needs to process the audio codec model, it can obtain a processing sample composed of sample audio from server 102. After obtaining the sample audio from the processing sample from server 102, terminal device 101 inputs the sample audio from the processing sample to the encoder to obtain the audio features output by the encoder. Based on the audio features, the target codebook number required for quantization encoding of the audio features is determined. Based on the target codebook number, the quantizer in the initial audio codec model is controlled to quantize the audio features to obtain quantized features. The quantized features are decoded by the decoder of the initial audio codec model to obtain reconstructed audio. Based on the difference between the sample audio and the reconstructed audio, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

[0065] For ease of understanding, the processing method of the audio encoding / decoding model in this application is further described below. Please refer to [link / reference]. Figure 2 One embodiment of the audio codec model processing method in this application includes:

[0066] 201. The sample audio is encoded by the encoder of the initial audio codec model to obtain audio features;

[0067] Unlike existing audio codec models, the audio codec model in this embodiment incorporates a convolutional neural network between the encoder and quantizer. That is, the audio codec model in this embodiment includes an encoder, a convolutional neural network, a quantizer, and a decoder. For ease of understanding, Figure 3 A schematic diagram of the audio codec model in the embodiments of this application is provided, wherein the structure of the encoder, quantizer and decoder in the embodiments of this application is consistent with that in the prior art.

[0068] Specifically, in this embodiment of the application, after obtaining the sample audio in the processing sample, preprocessing can be performed on the original audio. The preprocessing includes framing, resampling and normalization, which can unify the audio input format and ensure stable input of the encoder.

[0069] After preprocessing the sample audio, the preprocessed audio can be input into the encoder, which will then perform core feature extraction and downsampling compression on the preprocessed audio to obtain the audio features output by the encoder. In other words, the encoder is used to compress high-sampling-rate audio into low-frame-rate, low-dimensional and semantically compact audio features, so as to discard redundancy and retain core information such as timbre, pitch and speech.

[0070] 202. Based on audio features, determine the number of target codebooks required for quantization encoding of the audio features;

[0071] After obtaining the audio features, this application embodiment can determine the number of target codebooks required for quantization encoding of the audio features. Specifically, when determining the number of target codebooks, prediction can be based on empirical values ​​or on neural network models. No specific restrictions are placed on the process of determining the number of target codebooks here.

[0072] 203. Based on the target codebook quantity, control the quantizer in the initial audio codec model to quantize the audio features to obtain quantized features;

[0073] The pre-calculated target codebook number is input into the quantizer of the initial audio codec model. It is assumed that the quantizer has a fixed total of N cascaded codebooks, each codebook corresponding to an independent residual quantization operation level, and each level of codebook performs residual fitting and quantization correction sequentially according to a preset encoding order. In this embodiment, the total number of quantizer codebooks N can be set to 20 levels, and the target codebook number is less than or equal to the total number of codebooks, with an adaptive dynamic value.

[0074] After obtaining the target codebook number, the quantizer, in response to the control command for the target codebook number, calls the corresponding number of preceding valid codebook levels from all concatenated codebooks, discarding the remaining redundant codebook levels to avoid invalid quantization operations. Specifically, if the currently calculated target codebook number is X, the quantizer only starts the first X concatenated codebook levels to perform layer-by-layer residual vector quantization processing on the input audio features.

[0075] During the quantization process, the first-layer codebook first performs initial quantization fitting on the original input audio features, calculates the initial quantization residual, and outputs the first-layer quantized features. Each subsequent layer of the codebook iteratively quantizes and corrects the quantization residual output from the previous layer, compensating for the detailed information of the audio features layer by layer and continuously reducing the feature quantization error. After X layers of target codebooks undergo successive residual quantization iterations, the acoustic distribution characteristics of the current audio features can be adequately fitted.

[0076] After completing the residual quantization operation for all levels corresponding to the target codebook quantity, the quantization iteration process for the remaining codebook levels is terminated, and the feature data of the final level output is extracted to obtain the quantized features after adaptive quantization processing.

[0077] This quantization feature integrates the fine feature information of multi-level residual quantization, while avoiding problems such as feature overquantization and computational redundancy caused by redundant codebook levels.

[0078] 204. The quantization features are decoded using the decoder of the initial audio codec model to obtain the reconstructed audio;

[0079] After the quantizer completes the audio feature quantization based on the adaptive target codebook number and outputs accurate quantized features, the obtained quantized features are input into the decoder that matches the initial audio codec model. The decoder then performs a complete decoding and restoration process on the quantized features and finally outputs the reconstructed audio.

[0080] The decoder in this application corresponds one-to-one with the front-end adaptive residual quantizer and feature encoding structure, and can perform adaptive decoding on the quantization features obtained by quantization for different target codebook quantities, ensuring decoding compatibility under different audio complexities.

[0081] The decoder first receives the input quantization features. Based on the hierarchical residual quantization logic of the encoding stage, it reverses the process layer by layer to recover the audio feature information that has been hierarchically quantized, supplementing the audio details preserved by the residual quantization of the X-layer target codebook, and gradually restoring the complete deep audio feature data. During this process, the decoder can adapt to the quantization characteristics corresponding to the number of dynamic codebook layers. For the quantization features output by different codebook layers, it adaptively matches the corresponding decoding and restoration strategy, avoiding feature loss and audio distortion problems caused by the mismatch between fixed decoding methods and dynamic quantization.

[0082] Finally, the decoder performs dimensionality restoration and waveform reconstruction on the restored deep features of the audio, converting the discrete and compressed quantized feature data into continuous audio time-domain waveform data, and finally generating complete reconstructed audio.

[0083] 205. Based on the differences between the sample audio and the reconstructed audio, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

[0084] Once the reconstructed audio is obtained, the parameters of the initial audio codec model can be optimized using the backpropagation algorithm based on the differences between the sample audio and the reconstructed audio, thereby obtaining the target audio codec model.

[0085] After obtaining audio features using the encoder, this embodiment further determines the number of target codebooks required to quantize the audio features based on these features. This allows the quantizer to encode the audio features using the number of target codebooks matching the sample audio. Compared to a fixed number of codebook layers, this approach can accurately match the layer requirements of layer-by-layer residual coding based on different sample audio, fully utilizing the layered quantization capability of the concatenated codebook, effectively reducing residual errors in the audio feature quantization process, and improving audio coding fidelity.

[0086] based on Figure 2 The embodiment described above will now be described in detail below, focusing on step 202. Please refer to [link / reference]. Figure 3 , Figure 4 Detailed steps for steps 202 and 205:

[0087] 401. Input the audio features into the convolutional neural network to obtain the probability value of i concatenated codebooks required when the convolutional network encodes the audio features, where i is less than or equal to the total number N of concatenated codebooks built into the quantizer.

[0088] As an optional embodiment, when determining the target number of codebooks required by the quantizer to quantize the audio features, the audio features can be input into a convolutional neural network to obtain the probability value of i concatenated codebooks required when the convolutional network encodes the audio features.

[0089] Specifically, this application may input audio features into a convolutional neural network. The convolutional neural network in this embodiment may consist of three ordinary convolutional layers, and the output is processed by a softmax function to output the probability that i concatenated codebooks are needed when encoding the audio features, where i is less than or equal to the total number N of concatenated codebooks built into the quantizer. For ease of description, it is assumed that the probability of needing i concatenated codebooks output by the softmax function is prob_i.

[0090] It should be noted that the quantizer in this application embodiment includes multiple different numbers of layers, and each layer corresponds to a codebook. Assuming that the quantizer includes 20 layers, there are 20 codebooks (that is, the total number of codebooks is 20). Each layer corresponds to one codebook to perform residual encoding on the audio features layer by layer.

[0091] 402. Based on the probability values ​​of the number of i concatenated codebooks, calculate the number of target codebooks required to perform quantization encoding on the audio features.

[0092] Once the probability prob_i of the number of concatenated codebooks required to encode the audio features from the output of the convolutional neural network is obtained, the number of target codebooks required to encode the audio features can be calculated using this probability value.

[0093] Specifically, the following formula can be used for calculation:

[0094]

[0095] Where X is the target codebook number and N is the total number of codebooks in the quantizer. Assuming N is 20, then the total number of codebooks is 20.

[0096] In other words, the embodiments of this application calculate the number of target codebooks required to quantize the audio feature based on the audio feature, so that the quantizer encodes the audio feature using the number of target codebooks matching the sample audio. Thus, compared with a fixed number of codebook layers, it can accurately match the layer requirements of layer-by-layer residual coding according to different sample audio.

[0097] 403. Using the probability values ​​of the number of i concatenated codebooks and the preset sparsity control parameters, calculate the regularization loss. The regularization loss is used to avoid the audio codec model randomly selecting the number of layers of the concatenated codebook.

[0098] Specifically, after obtaining the probability value of using i concatenated codebooks in step 401, the regularization loss can be calculated using the probability value and the preset sparsity control parameters.

[0099] Specifically, the regularization loss L6 can be calculated using the following formula:

[0100] ;

[0101] As stated in the L6 calculation formula, assuming the quantizer includes 3 layers, and the initial audio codec model only selects the first layer of the quantizer, then the probability of the first layer is 1, and the probabilities of the others are 0. When choosing a three-layer quantizer, assuming the probability of the first layer is 0.2, the probability of the second layer is 0.2, and the probability of the third layer is 0.6, then... ;

[0102] Therefore, we can constrain the number of layers of the codebook that the model randomly selects by introducing regularization loss, and reduce the regularization loss so that the audio codec model selects each layer of the quantizer evenly, that is, selects each codebook in the quantizer evenly.

[0103] 404. Based on the differences between the sample audio and the reconstructed audio, and the regularization loss, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

[0104] After obtaining the regularization loss, the difference between the sample audio and the reconstructed audio, as well as the regularization loss, can be used to optimize the parameters of the initial audio codec model to obtain the target audio codec model.

[0105] In this embodiment of the application, after introducing regularization loss, regularization loss is added to the loss function to constrain the audio codec model to select each layer of the quantizer evenly, that is, to select each codebook evenly, thereby improving the utilization rate of each codebook.

[0106] based on Figure 2In the embodiments described above, to automatically guide the audio codec model to encode continuous latent feature vectors using the minimum total number of codebooks during processing, thereby improving the compression ratio of the original audio, this application embodiment can further add the following loss function and use it to process the audio codec model to automatically improve the model's compression ratio of the original audio. Please refer to [link to relevant documentation]. Figure 5 :

[0107] 501. Calculate the expected bit loss using the target codebook size. The expected bit loss is used to characterize the compression ratio of the encoder and quantizer.

[0108] Figure 2 After calculating the target codebook size in the embodiment, the expected bit loss can be further calculated using the target codebook size, where the expected bit loss is used to characterize the compression degree of the original audio by the encoder and quantizer.

[0109] Specifically, assuming the target codebook size is X, the expected bit loss L5 can be calculated using the following formula:

[0110] ,in, The strength of the compression codebook number. A larger value indicates that the model is more inclined towards codebook compression. The smaller the value, the weaker the compression force. The value can be customized according to actual needs. There are no specific restrictions on the value of .

[0111] As an optional embodiment, the embodiments of this application... It is 0.1.

[0112] 502. Based on the difference between the sample audio and the reconstructed audio, and the expected bit loss, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

[0113] By obtaining the expected bit loss L5, when processing the initial audio codec model, the difference between the sample audio and the reconstructed audio, as well as the expected bit loss, can be used to process the encoder, convolutional neural network, quantizer, and decoder in the initial audio codec model. This guides the audio codec model to encode and compress continuous latent feature vectors with as few codebooks as possible, thereby automatically improving the compression ratio of audio features during the processing.

[0114] based on Figure 2 The embodiments described above will now be described in detail with reference to steps 203 and 204. Please refer to [link to relevant documentation]. Figure 6 , Figure 6Detailed steps for steps 203, 204, and 205:

[0115] 601. Input the target codebook quantity into the quantizer to obtain the discrete quantization index output by the quantizer after performing residual vector quantization on the audio features using the target codebook quantity;

[0116] Specifically, in this embodiment, the quantizer in the initial audio codec model is also called the RVQ residual vector quantizer. Assuming the total number of codebooks is 20, the RVQ residual vector quantizer has 20 layers. Each codebook includes 1024 codewords, and each layer is an independent vector quantizer. The structure of each quantizer layer includes: L2 normalized codewords, nearest neighbor search, and residual update. L2 normalized codewords involve normalizing the L2 modulus of all codewords and audio features in the codebook, so that the similarity between audio features and codewords in the codebook can be determined using cosine distance. Nearest neighbor search utilizes L2... The normalized audio features are iterated through all codewords in each codebook layer. Assuming each codebook layer has 1024 codewords, the process iterates through all 1024 codewords in each layer to select the codeword q1 with the smallest distance and the most similarity. Then, the codeword vector q1 is output as an index (here, the index of q1 is used to encode the continuous latent feature vector). The residual update calculates the unquantized error of the current layer. For example, assuming the L2 normalized audio feature is r1, the residual of the current layer quantizer is: r2 = r1 - q1. The residual r2 is then output to the quantizer of the next layer so that the next layer quantizer can use the codebook of the next layer to further encode the residual r2. This process is repeated to obtain the discrete quantization index output by each layer quantizer using the number of codewords in the current layer. Here, the discrete quantization index refers to the index of the codeword in each codebook layer that is most similar to the residual.

[0117] Therefore, in this embodiment of the application, after the target codebook quantity is input to the quantizer, the quantizer performs residual vector quantization on the audio features using the target codebook quantity matching the type of audio, and outputs a discrete quantization index.

[0118] 602. Input the discrete quantization index to the decoder, so that the decoder performs decoding on the discrete quantization index, restores the reconstructed audio features, and restores the reconstructed audio features to the reconstructed audio.

[0119] After obtaining the discrete quantization index of each codebook layer output by the quantizer, the discrete quantization index of each codebook layer can be input into the decoder. The decoder then uses the concatenated codebook shared with the quantizer to decode the discrete quantization index, thereby restoring the reconstructed audio features layer by layer. The process of restoring the reconstructed audio features involves finding the specific residual value using the codebook of each layer, and then accumulating the residual values ​​of multiple layers to obtain the reconstructed audio features. After obtaining the reconstructed audio features, the upsampling convolution module in the decoder (corresponding to the downsampling convolution module in the encoder) is further used to upsample the reconstructed audio features to restore the reconstructed audio features to the reconstructed audio.

[0120] 603. Using the sample audio and the reconstructed audio, calculate the reconstruction loss;

[0121] Once the reconstructed audio is obtained, the difference between the sample audio and the reconstructed audio can be calculated to determine the reconstruction loss.

[0122] After obtaining the reconstructed audio output from the decoder, the reconstruction loss is calculated by further utilizing the reconstructed audio and the original audio.

[0123] Specifically, when calculating the reconstruction loss, the reconstruction loss generally includes spectral loss L1 and temporal loss L2. Spectral loss L1 is used to measure the spectral similarity between the reconstructed audio and the sample audio, while temporal loss L2 is used to measure the temporal similarity between the reconstructed audio and the sample audio.

[0124] 604. Using the audio features and reconstructed audio features, calculate the reconstruction residual loss;

[0125] Similarly, after obtaining the audio features and the reconstructed audio features, the difference between the audio features and the reconstructed audio features can be calculated to obtain the reconstruction residual loss.

[0126] Specifically, the reconstruction residual loss includes the quantization loss L3 and the entropy coding loss L4 of the quantizer. The quantization loss L3 is used to measure the loss between each layer of residual and the codeword most similar to the residual in each codebook. The entropy coding loss L4 is used to measure the codeword utilization of each layer of codebook. The higher the codeword utilization, the smaller the entropy coding loss and the better the compression.

[0127] 605. Using reconstruction loss, reconstruction residual loss, and backpropagation algorithm, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

[0128] After obtaining the reconstruction loss and reconstruction residual loss, the parameters of the initial audio codec model can be optimized using the reconstruction loss, reconstruction residual loss, and backpropagation algorithm to obtain the target audio codec model.

[0129] In this embodiment, the specific process of quantization features and audio reconstruction is described in detail, as well as the process of calculating reconstruction loss and reconstruction residual loss. When calculating reconstruction loss, not only spectral loss and temporal loss are included, but also when calculating reconstruction residual loss, not only quantization loss and entropy coding loss of the quantizer are included. This allows the initial audio codec model to simultaneously constrain temporal and frequency domain features, thereby improving the overall fidelity of the initial audio codec model for reconstructed audio.

[0130] based on Figures 2 to 6 In one embodiment, the initial audio codec model may further include a discriminator. After the decoder outputs the reconstructed audio, the reconstructed audio is further input to the discriminator, so that the discriminator judges the authenticity of the reconstructed audio based on the sample audio, calculates the first adversarial loss based on the judgment result, and processes the discriminator using the first adversarial loss.

[0131] Specifically, in order to determine the authenticity of the reconstructed audio output by the decoder, embodiments of this application may also set a discriminator in the audio codec model. The discriminator includes a multi-scale discriminator and a multi-period discriminator. The processing goal of the discriminator is to learn to distinguish between real audio and fake audio generated by the discriminator, and to make the discriminator output a score of 1 for real original audio and a score of 0 for fake audio. In the process of processing, the discriminator parameters are continuously optimized to make the discriminator's ability to distinguish between real and fake audio stronger and stronger.

[0132] Specifically, in processing the discriminator, a parallel multi-scale discriminator and a multi-cycle discriminator are used. The sample audio and the reconstructed audio are input into the discriminator respectively to obtain the discriminator's output discrimination score Y1 for real audio and discrimination score Y2 for fake audio. Then, using the discrimination scores Y1 and Y2, the first adversarial loss L7 is calculated using the following formula:

[0133] in, ;

[0134] The multi-scale discriminator and the multi-period discriminator each include multiple branched parallel discriminator structures, where k represents the k-th discriminator branch. This represents the score given by the k-th discriminator to the real audio. This represents the score given by the k-th discriminator to the fake audio. This represents summing up the losses of all k discriminators; E represents the expected value of the score for each discriminator.

[0135] After obtaining the first adversarial loss L7, the discriminator can be processed using the first loss L7, that is, the parameters of the discriminator can be processed to make L7 as small as possible, so that the discriminator can accurately distinguish between real and fake audio.

[0136] In this embodiment, a discriminator is also set in the audio codec model, and the parameters of the discriminator are continuously updated to improve the accuracy of judging the authenticity of the reconstructed audio.

[0137] Because a discriminator is included in the audio codec model, the discriminator's judgment of the authenticity of the audio can be used to calculate the generator's second adversarial loss. In this model, the generator is the decoder. The calculation process for the second adversarial loss is described below:

[0138] Specifically, the second adversarial loss L8 can be calculated using the following formula:

[0139]

[0140] In addition to obtaining the second adversarial loss L8, the second adversarial loss of the decoder can be added during the processing of the encoder, convolutional neural network, quantizer and decoder. That is, the encoder, convolutional neural network and quantizer and decoder are processed by using reconstruction loss, reconstruction residual loss, regularization loss, expected bit loss, second adversarial loss and backpropagation algorithm, so that the reconstructed audio output by the decoder can fool the discriminator and make the discriminator judge the fake audio as real audio, that is, to make L8 as small as possible.

[0141] As an optional embodiment, the initial audio codec model in this application embodiment includes, but is not limited to, a DAC audio codec model or an EnCodec audio codec model. When the initial audio codec model includes a DAC audio codec model, a convolutional neural network is added between the encoder and quantizer of the original DAC audio codec model, while the encoder, quantizer, decoder, and discriminator remain consistent with the original. When the initial audio codec model includes an EnCodec audio codec model, a convolutional neural network is also added between the encoder and quantizer of the original EnCodec audio codec model, while the encoder, quantizer, decoder, and discriminator remain consistent with the original.

[0142] The above embodiments have described in detail the processing method of the initial audio codec model. The following describes the process of performing audio encoding and decoding using the target audio codec model obtained by the above processing method. Please refer to [link to relevant documentation]. Figure 7 An embodiment of an audio encoding / decoding method in this application includes:

[0143] 701. Encode the target audio using the encoder of the target audio codec model to obtain audio features;

[0144] 702. Based on the audio features, determine the number of target codebooks required to quantize and encode the audio features;

[0145] 703. Based on the target codebook quantity, control the quantizer in the target audio codec model to quantize the audio features to obtain quantized features;

[0146] 704. The quantization features are decoded by the decoder of the target audio codec model to obtain the reconstructed audio corresponding to the target audio.

[0147] It should be noted that after processing the initial audio codec model, the target audio codec model can be obtained. The encoding and decoding process of the target audio using the target audio codec model is similar to... Figure 2 The encoding and decoding process of the sample audio using the initial audio encoding and decoding model in the embodiment is similar and will not be described again here.

[0148] After obtaining the audio features, the target audio codec model in this embodiment further calculates the number of target codebooks required to quantize the audio features based on these features. The number of target codebooks and the audio features are then input into the quantizer, which encodes the audio features using the target number of codebooks. This allows the quantizer to encode the audio features using as few codebooks as possible during the optimization of the initial audio codec model parameters, thereby further compressing the number of encoded bits and improving the audio compression effect.

[0149] based on Figure 7 The embodiment described below provides a detailed description of step 702. Please refer to [link / reference]. Figure 8 , Figure 8 The detailed steps of step 702 are as follows:

[0150] 801. Input the audio features into the convolutional neural network to obtain the probability value of i concatenated codebooks required when the convolutional network encodes the audio features, where i is less than or equal to the total number N of concatenated codebooks built into the quantizer.

[0151] 802. Based on the probability values ​​of the number of i concatenated codebooks, calculate the number of target codebooks required to perform quantization encoding on the audio features.

[0152] The description of steps 801-802 is consistent with... Figure 4 The descriptions of steps 401-402 in the embodiments are similar and will not be repeated here.

[0153] In this embodiment of the application, when using a convolutional network to encode audio features, the probability value of the required number of i concatenated codebooks is calculated. This target codebook number obtained based on probability weighting is the optimal solution for the probability requirement of audio feature encoding. Compared with directly fixing the number of codebook layers, it can accurately match the layer requirement of layer-by-layer residual encoding according to different target audio, make full use of the layered quantization capability of concatenated codebooks, effectively reduce residual errors in the audio feature quantization process, and improve audio encoding fidelity.

[0154] based on Figure 7 The embodiments described below provide a detailed description of steps 703 and 704. Please refer to [link / reference]. Figure 9 , Figure 9 Detailed steps for steps 703 and 704:

[0155] 901. Input the target codebook quantity into the quantizer to obtain the discrete quantization index output by the quantizer after performing residual vector quantization on the audio features using the target codebook quantity;

[0156] 902. Input the discrete quantization index into the decoder, so that the decoder performs decoding on the discrete quantization index, restores the reconstructed audio features, and restores the reconstructed audio features to the reconstructed audio.

[0157] The descriptions of steps 901-902 are similar to those of steps 601 and 602, and will not be repeated here.

[0158] This application provides a detailed description of the process of quantizing audio features using the target codebook quantity. This process of using the target codebook quantity to drive the quantizer to perform residual vector quantization can dynamically adjust the number of cascaded quantization layers according to the complexity of the audio features. This allows simple audio to achieve efficient compression with fewer codebooks, while complex audio can use a sufficient number of codebooks to ensure detail restoration, avoiding the problems of redundant calculations or loss of details caused by fixed-layer quantization.

[0159] Furthermore, after obtaining the reconstructed audio output by the decoder, in order to improve the accuracy of judging the authenticity of the reconstructed audio, this embodiment of the application can also input the reconstructed audio to the discriminator, so that the discriminator judges the authenticity of the reconstructed audio. That is, when the reconstructed audio is similar to the target audio, the output judgment result is true, and when the reconstructed audio is not similar to the original audio, the output judgment result is false, which is equivalent to further improving the similarity between the reconstructed audio and the original audio.

[0160] It is easy to understand that after the discriminator has finished processing the audio, it has the ability to judge the authenticity of the reconstructed audio. Therefore, after the reconstructed audio is input into the discriminator, the discriminator can judge the authenticity of the reconstructed audio.

[0161] It is understood that, in various embodiments of the present invention, the order of the steps does not imply the order of execution. The execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0162] The computer device in the embodiments of the present invention will now be described from the perspective of hardware processing:

[0163] One embodiment of the computer device in this invention includes:

[0164] Processor and memory;

[0165] The memory is used to store computer programs. When the processor executes the computer programs stored in the memory, it can implement the processing method of the audio codec model in the above method embodiments, or the audio codec method in the above method embodiments.

[0166] It is understood that when the processor in the computer device described above executes the computer program, it can also implement the functions of each unit in the corresponding device embodiments described above, which will not be repeated here. For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device. For example, the computer program can be divided into units in the aforementioned computer device, and each unit can implement the specific functions described in the corresponding computer devices above.

[0167] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the processor and memory are merely examples of a computer device and do not constitute a limitation on the computer device. It may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0168] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0169] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0170] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor can be used to implement the processing method of the audio codec model in the above method embodiments, or the audio codec method in the above method embodiments.

[0171] It is understood that if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a corresponding computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing an audio codec model, characterized in that, The method includes: The sample audio is encoded by the encoder of the initial audio codec model to obtain audio features; Based on the audio features, determine the number of target codebooks required to quantize and encode the audio features; Based on the target codebook quantity, the quantizer in the initial audio codec model is controlled to quantize the audio features to obtain quantized features; The quantization features are decoded by the decoder of the initial audio codec model to obtain the reconstructed audio; Based on the differences between the sample audio and the reconstructed audio, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

2. The method according to claim 1, characterized in that, The method further includes: Using the target codebook size, the expected bit loss is calculated, and the expected bit loss is used to characterize the compression ratio of the encoder and the quantizer; The step of optimizing the parameters of the initial audio codec model based on the difference between the sample audio and the reconstructed audio to obtain the target audio codec model includes: Based on the difference between the sample audio and the reconstructed audio, and the expected bit loss, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

3. The method according to claim 1, characterized in that, The step of determining the number of target codebooks required for quantization encoding of the audio features based on the audio features includes: The audio features are input into a convolutional neural network to obtain the probability value of i concatenated codebooks required when the convolutional neural network encodes the audio features, where i is less than or equal to the total number N of concatenated codebooks built into the quantizer; Based on the probability values ​​of the number of i concatenated codebooks, the number of target codebooks required to perform quantization encoding on the audio features is calculated.

4. The method according to claim 3, characterized in that, The method further includes: Using the probability values ​​of the number of i concatenated codebooks and preset sparsity control parameters, a regularization loss is calculated. The regularization loss is used to prevent the audio codec model from randomly selecting the number of layers of the concatenated codebook. The step of optimizing the parameters of the initial audio codec model based on the difference between the sample audio and the reconstructed audio to obtain the target audio codec model includes: Based on the difference between the sample audio and the reconstructed audio, and the regularization loss, the parameters of the initial audio codec model are optimized to obtain the target audio codec model.

5. The method according to claim 1, characterized in that, Based on the target codebook quantity, the quantizer in the initial audio codec model is controlled to quantize the audio features to obtain quantized features, including: The target codebook quantity is input into the quantizer to obtain the discrete quantization index output by the quantizer after performing residual vector quantization on the audio features using the target codebook quantity; The step of decoding the quantization features using the decoder of the initial audio codec model to obtain the reconstructed audio includes: The discrete quantization index is input to the decoder, which decodes the discrete quantization index to restore the reconstructed audio features and restores the reconstructed audio features to the reconstructed audio.

6. The method according to claim 5, characterized in that, The step of optimizing the parameters of the initial audio codec model based on the difference between the sample audio and the reconstructed audio to obtain the target audio codec model includes: The reconstruction loss is calculated using the sample audio and the reconstructed audio. The reconstruction residual loss is calculated using the audio features and the reconstructed audio features. The initial audio codec model is optimized using the reconstruction loss, the reconstruction residual loss, and the backpropagation algorithm to obtain the target audio codec model.

7. The method according to any one of claims 1 to 6, characterized in that, The audio codec model includes either the DAC audio codec model or the EnCodec audio codec model.

8. An audio encoding / decoding method, characterized in that, The method, applied to the target audio codec model according to any one of claims 1 to 7, comprises: The target audio is encoded by the encoder of the target audio codec model to obtain audio features; Based on the audio features, determine the number of target codebooks required to quantize and encode the audio features; Based on the target codebook quantity, the quantizer in the target audio codec model is controlled to quantize the audio features to obtain quantized features; The quantization features are decoded by the decoder of the target audio codec model to obtain the reconstructed audio corresponding to the target audio.

9. The method according to claim 8, characterized in that, The step of determining the number of target codebooks required for quantization encoding of the audio features based on the audio features includes: The audio features are input into a convolutional neural network to obtain the probability value of i concatenated codebooks required when the convolutional neural network encodes the audio features, where i is less than or equal to the total number N of concatenated codebooks built into the quantizer; Based on the probability values ​​of the number of i concatenated codebooks, the number of target codebooks required to perform quantization encoding on the audio features is calculated.

10. The method according to claim 8, characterized in that, Based on the target codebook quantity, the quantizer in the initial audio codec model is controlled to quantize the audio features to obtain quantized features, including: The target codebook quantity is input into the quantizer to obtain the discrete quantization index output by the quantizer after performing residual vector quantization on the audio features using the target codebook quantity; The step of decoding the quantization features using the decoder of the initial audio codec model to obtain the reconstructed audio includes: The discrete quantization index is input to the decoder, which decodes the discrete quantization index to restore the reconstructed audio features and restores the reconstructed audio features to the reconstructed audio.

11. A computer device comprising a processor, characterized in that, When the processor executes a computer program stored in the memory, it is used to implement the processing method of the audio codec model as described in any one of claims 1 to 7, or the audio codec method as described in any one of claims 8 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it is used to implement the processing method of the audio codec model as described in any one of claims 1 to 7, or the audio codec method as described in any one of claims 8 to 10.

13. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it is used to implement the processing method of the audio codec model as described in any one of claims 1 to 7, or the audio codec method as described in any one of claims 8 to 10.

Images