Audio compression method and device

Abstract

The invention provides an audio compression method and device, and the method comprises the steps: carrying out the compression of a to-be-compressed audio clip based on an audio compression model through employing the complexity of the to-be-compressed audio clip, and obtaining a compressed audio; the audio compression model comprises a plurality of quantization layers, and the audio compression model is used for encoding a to-be-compressed audio clip to obtain encoding features, quantizing the encoding features by adopting a target quantization layer, decoding the quantized encoding features to obtain decoding features, and determining a compressed audio based on the decoding features; the target quantization layer is at least one quantization layer selected from a plurality of quantization layers based on the complexity of the audio clip to be compressed. According to the audio compression model, the compression ratio can be flexibly adjusted on the premise of ensuring the tone quality, so that the audio compression model can adapt to audio clips with different complexities, and the optimal balance between the tone quality and the compression ratio is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an audio compression method and device. Background Art

[0002] Audio compression refers to the process of reducing the storage space occupied by audio data through specific encoding technology while trying to keep the sound quality of the original audio without being significantly damaged.

[0003] The goal is to use unsupervised learning methods to build audio compression models and generate low-bitrate compressed audio based on the audio compression model. However, the audio compression model compresses the input audio with a uniform compression rate. For input audio containing complex sound details, some audio information may be lost. Summary of the invention

[0004] The present invention provides an audio compression method and device to solve the defects in the prior art.

[0005] The present invention provides an audio compression method, comprising the following steps: Determining the complexity of the audio segment to be compressed; Based on the audio compression model, the complexity of the audio segment to be compressed is applied to compress the audio segment to obtain compressed audio; The audio compression model includes multiple quantization layers. The audio compression model is used to encode the audio segment to be compressed to obtain encoding features, then use a target quantization layer to quantize the encoding features, and decode the quantized encoding features to obtain decoding features, and determine the compressed audio based on the decoding features; the target quantization layer is at least one quantization layer selected from the multiple quantization layers based on the complexity of the audio segment to be compressed.

[0006] According to an audio compression method provided by the present invention, the audio compression model is trained based on the following steps: Encode the sample original audio segment to obtain the sample encoding feature; Determining a sample quantization layer for quantizing the sample coding feature from the plurality of quantization layers based on the complexity of the sample original audio segment; quantizing the sample coding feature based on the sample quantization layer to obtain a sample quantization feature; Decoding the sample quantization feature to obtain a sample decoding feature, and generating a sample compressed audio corresponding to the sample original audio segment based on the sample decoding feature; Based on the difference between the sample compressed audio and the sample original audio segment, the initial generation model of the audio compression model is trained to obtain the audio compression model.

[0007] According to an audio compression method provided by the present invention, the initial generation model of the audio compression model is trained based on the difference between the sample compressed audio and the sample original audio segment to obtain the audio compression model, including: Based on the difference between the sample decoding feature and the sample encoding feature, and the difference between the sample compressed audio and the sample original audio segment, the initial generation model of the audio compression model is trained to obtain the audio compression model.

[0008] According to an audio compression method provided by the present invention, the audio compression model is obtained by performing generative adversarial training in conjunction with a discriminant model; The step of training the initial generation model of the audio compression model based on the difference between the sample compressed audio and the sample original audio segment to obtain the audio compression model comprises: Inputting the sample compressed audio into an initial discriminant model to obtain a first classification result output by the initial discriminant model; Inputting the sample original audio segment into the initial discriminant model to obtain a second classification result output by the initial discriminant model; Based on the difference between the first classification result and the classification label corresponding to the sample compressed audio, the difference between the second classification result and the classification label corresponding to the sample original audio segment, and the difference between the sample compressed audio and the sample original audio segment, the initial generative model and the initial discriminant model are jointly trained to obtain the audio compression model.

[0009] According to an audio compression method provided by the present invention, the initial discriminant model includes a plurality of branches, each branch is used to extract audio features of the input audio, and determine the classification result based on the audio features of the input audio, and the scales of the audio features extracted by each branch are different; The method of jointly training the initial generative model and the initial discriminant model based on the difference between the first classification result and the classification label corresponding to the sample compressed audio, the difference between the second classification result and the classification label corresponding to the sample original audio segment, and the difference between the sample compressed audio and the sample original audio segment to obtain the audio compression model includes: Based on the differences between the audio features of the sample original audio segment extracted by each branch and the audio features of the sample compressed audio, the differences between the first classification result and the corresponding classification label of the sample compressed audio, the differences between the second classification result and the corresponding classification label of the sample original audio segment, and the differences between the sample compressed audio and the sample original audio segment, the initial generative model and the initial discriminant model are jointly trained to obtain the audio compression model.

[0010] According to an audio compression method provided by the present invention, determining the complexity of the audio segment to be compressed includes: Determining the complexity of the audio segment to be compressed based on an audio complexity classification model; The audio complexity classification model is trained based on a plurality of sample audio segments and complexity labels of each sample audio segment.

[0011] According to an audio compression method provided by the present invention, the complexity label of each sample audio segment is determined based on the following steps: determining a plurality of different quality indicator scores for each sample audio segment; Based on preset weights, weighted addition is performed on multiple different quality indicator scores of each sample audio segment to determine a quality score of each sample audio segment; Based on the quality score of each sample audio segment and the total number of quantization layers, a complexity label of the sample audio segment is determined.

[0012] The present invention also provides an audio compression device, comprising the following modules: A determination unit, used to determine the complexity of the audio segment to be compressed; A compression unit, configured to compress the audio segment to be compressed based on an audio compression model and the complexity of the audio segment to be compressed to obtain compressed audio; The audio compression model includes multiple quantization layers. The audio compression model is used to encode the audio segment to be compressed to obtain encoding features, then use a target quantization layer to quantize the encoding features, and decode the quantized encoding features to obtain decoding features, and determine the compressed audio based on the decoding features; the target quantization layer is at least one quantization layer selected from the multiple quantization layers based on the complexity of the audio segment to be compressed.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned audio compression methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the audio compression method described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the audio compression method described above is implemented.

[0016] The audio compression method and device provided by the present invention selects one or more suitable quantization layers from multiple quantization layers for quantization processing according to the complexity of the audio segment to be compressed, which not only ensures that the audio segment with high complexity can retain more sound quality details during the quantization process and avoids excessive sound quality loss, but also enables the audio segment with low complexity to achieve a higher compression rate by selecting a higher quantization layer, thereby greatly reducing the storage space and transmission time while ensuring acceptable sound quality, and improving the overall efficiency and practicality of audio compression. In other words, the audio compression model in the present invention can flexibly adjust the compression rate while ensuring sound quality, so that the audio compression model can adapt to audio segments of different complexities and achieve the best balance between sound quality and compression rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the audio compression method provided by the present invention.

[0019] Figure 2 It is a flowchart of the audio compression model training method provided by the present invention.

[0020] Figure 3 It is a flowchart diagram of another audio compression method provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the audio compression device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] At present, most audio compression models use codec models, which usually include an encoder, a residual vector quantizer (RVQ) and a decoder. Both the encoder and the decoder are fully convolutional structures. The encoder is used to map the input original audio to an embedded sequence with a lower sampling rate, the residual vector quantizer is used to quantize the embedded sequence, and the decoder is used to reconstruct the original audio based on the quantized embedded sequence to obtain compressed audio. The training goal of the codec model is to minimize the difference between the compressed audio and the original audio. However, the codec model uses a uniform compression rate to compress the original audio. If a high compression rate is used uniformly, the compressed audio may lose the high-frequency details in the original audio, and the speech part may become blurred due to over-compression. On the contrary, if a low compression rate is used uniformly, although more audio information can be retained, the storage and transmission costs will be significantly increased.

[0025] To this end, the present invention provides an audio compression method. Figure 1 is a flow chart of the audio compression method provided by the present invention, such as Figure 1 As shown, the method includes step 110 and step 120.

[0026] Step 110: Determine the complexity of the audio segment to be compressed.

[0027] Specifically, the audio segment to be compressed is an audio segment that needs to be compressed. The audio segment to be compressed can be collected by a sound pickup device, input by a user, or obtained on the Internet by crawling, which is not specifically limited in the embodiment of the present invention.

[0028] The complexity of the audio clip to be compressed refers to the degree of change of the audio clip in the time domain and frequency domain, including characteristics such as dynamic range, frequency components, sound details, redundancy, etc. Audio clips with high complexity usually contain more sound details and changes, while audio clips with low complexity are relatively stable and simple.

[0029] For audio clips with high complexity, the compressed audio usually needs to retain more detailed information to maintain the sound quality. Therefore, a lower compression rate can be used for audio clips with high complexity to retain the detailed information in the audio clips. For audio clips with low complexity, the compressed audio usually retains less detailed information. Therefore, a higher compression rate can be used for audio clips with low complexity to reduce storage and transmission costs.

[0030] It can be seen that different compression requirements correspond to audio segments to be compressed with different complexities. Therefore, the embodiment of the present invention needs to first determine the complexity of the audio segment to be compressed, and adapt the corresponding compression strategy to the audio segment to be compressed according to the complexity, so as to achieve efficient and high-quality audio compression. The complexity of the audio segment to be compressed can be obtained by performing time domain analysis and frequency domain analysis on the audio signal of the audio segment to be compressed.

[0031] Step 120: Based on the audio compression model, the complexity of the audio segment to be compressed is applied to compress the audio segment to obtain compressed audio; The audio compression model includes multiple quantization layers. The audio compression model is used to encode the audio segment to be compressed to obtain coding features, then use a target quantization layer to quantize the coding features, and decode the quantized coding features to obtain decoding features, and determine the compressed audio based on the decoding features; the target quantization layer is at least one quantization layer selected from multiple quantization layers based on the complexity of the audio segment to be compressed.

[0032] Specifically, the quantization layer is used to convert continuous sample values ​​in the audio segment to be compressed into discrete values. The quantization layer is a key parameter in the audio compression model, which determines the accuracy and step size of quantizing the coding features. The higher the quantization layer (i.e., the later the quantization layer), the larger the quantization step size is usually, and the lower the quantization accuracy is, so that more information is discarded during the quantization process, achieving a higher compression rate. On the contrary, the lower the quantization layer (i.e., the earlier the quantization layer), the smaller the quantization step size is, the higher the quantization accuracy is, and more sound quality details can be retained, but the compression rate is also reduced accordingly.

[0033] For highly complex audio clips to be compressed (such as music clips with rich details and dynamic range), a lower quantization layer can be selected to retain more sound quality details. Although the compression rate will be slightly lower, the sound quality restoration is guaranteed. For less complex audio clips (such as voice calls or simple background music), a higher quantization layer can be selected to significantly improve the compression rate while maintaining an acceptable sound quality level, reducing storage space or transmission bandwidth usage.

[0034] Based on this, after the audio segment to be compressed and its complexity are input into the audio compression model, the audio compression model selects at least one quantization layer from multiple quantization layers based on the complexity as the target quantization layer. As the complexity increases, the number of corresponding target quantization layers decreases.

[0035] Audio compression models usually select quantization layers in order (from front to back) because lower quantization layers can retain more sound quality details, while higher quantization layers can achieve higher compression rates. By selecting quantization layers in sequence, the audio compression model can gradually adjust the compression rate while ensuring sound quality. If you want to reduce the compression rate (that is, improve the sound quality), you can increase the number of selected quantization layers (that is, select more lower quantization layers); if you want to increase the compression rate (that is, reduce the sound quality), you can reduce the number of selected quantization layers (that is, select fewer higher quantization layers).

[0036] For example, for an audio segment 1 with high complexity, the first m quantization layers can be selected as target quantization layers, and for an audio segment 2 with low complexity, the first n quantization layers can be selected as target quantization layers, where m is the number of quantization layers. <n。

[0037] In summary, when selecting quantization layers, the audio compression model can weigh the relationship between sound quality and compression rate according to the complexity of the audio segment to be compressed. As the complexity increases, the audio compression model will tend to select a lower quantization layer to preserve the sound quality; at the same time, by selecting quantization layers in sequence and adjusting the number, the audio compression model can flexibly adjust the compression rate while ensuring the sound quality, so that the audio compression model can adapt to audio segments of different complexities and achieve the best balance between sound quality and compression rate.

[0038] As an optional embodiment, after determining the target quantization layer corresponding to the audio segment to be compressed, the audio segment to be compressed can be encoded to obtain coding features, and the coding features can be quantized by the target quantization layer to convert continuous coding feature values ​​into discrete, finite number of values, so as to reduce the number of bits required to represent the audio segment to be compressed and reduce the amount of data.

[0039] After quantization is complete, the audio compression model decodes the quantized encoded features. Decoding is the reverse process of encoding, which converts the quantized encoded features back into a format or representation that is closer to the audio segment to be compressed. In this process, the decoder attempts to recover the lost information in the encoded features to get as close as possible to the quality of the audio segment to be compressed. After decoding is complete, based on the decoded features, the audio compression model can determine the final compressed audio.

[0040] The determination of the target quantization layer may be performed synchronously with the encoding of the audio segment to be compressed, and after the target quantization layer is determined, the encoding features may be quantized based on the target quantization layer.

[0041] Before compressing the audio segment to be compressed based on the audio compression model and applying the complexity of the audio segment to be compressed to obtain the compressed audio, the audio compression model can be pre-trained, which can be achieved by performing the following steps: First, a large number of sample original audio segments and the complexity of the sample original audio segments are collected, and their corresponding compressed audio labels are determined through manual annotation. Then, the initial model is trained based on the sample original audio segments, the complexity of the sample original audio segments and their corresponding compressed audio labels, so as to obtain the audio compression model.

[0042] In addition, based on the audio compression model, the complexity of the audio segment to be compressed is applied to compress the audio segment to be compressed. Before obtaining the compressed audio, the audio segment to be compressed can also be resampled. The audio compression method provided in the embodiment of the present invention selects one or more suitable quantization layers from multiple quantization layers for quantization processing according to the complexity of the audio segment to be compressed, which not only ensures that the audio segment with high complexity can retain more sound quality details during the quantization process and avoid excessive sound quality loss, but also enables the audio segment with low complexity to achieve a higher compression rate by selecting a higher quantization layer, thereby greatly reducing the storage space and transmission time while ensuring acceptable sound quality, and improving the overall efficiency and practicality of audio compression. In other words, the audio compression model in the embodiment of the present invention can flexibly adjust the compression rate while ensuring sound quality, so that the audio compression model can adapt to audio segments of different complexities and achieve the best balance between sound quality and compression rate.

[0043] Based on the above embodiment, the audio compression model is trained based on the following steps: Encode the sample original audio segment to obtain the sample encoding feature; Determining a sample quantization layer for quantizing coding features of the sample from a plurality of quantization layers based on the complexity of the original audio segment of the sample; Quantize the sample coding features based on the sample quantization layer to obtain the sample quantization features; Decoding the sample quantization feature to obtain the sample decoding feature, and generating the sample compressed audio corresponding to the sample original audio segment based on the sample decoding feature; Based on the difference between the sample compressed audio and the sample original audio segment, an initial generation model of the audio compression model is trained to obtain the audio compression model.

[0044] Specifically, the sample original audio segment refers to the original audio data used to train the audio compression model, which may be a representative segment selected from a large amount of audio data. The sample encoding feature is a feature obtained after encoding the sample original audio segment.

[0045] In addition, the complexity of the sample original audio clip refers to the degree of change of the sample original audio clip in the time domain and frequency domain, including characteristics in terms of dynamic range, frequency components, sound details, redundancy, etc. Audio clips with high complexity usually contain more sound details and changes, while audio clips with low complexity are relatively stable and simple. The sample quantization layer refers to the quantization layer selected according to the complexity of the sample original audio clip, which is used to quantize the sample coding features to obtain the sample quantization features.

[0046] After obtaining the sample quantization features, the sample quantization features are decoded to obtain the sample decoding features, and based on the sample decoding features, a sample compressed audio corresponding to the sample original audio segment is generated. The size of the sample compressed audio is smaller than the sample original audio segment, but the training goal is to require the sample compressed audio to retain the main information of the sample original audio segment as little as possible.

[0047] In this regard, considering that the difference between the sample compressed audio and the sample original audio segment is used to characterize the integrity and fidelity of the sample compressed audio with the sample original audio segment, the smaller the difference between the two, the more important information of the sample compressed audio has been successfully retained in the sample original audio segment, thereby achieving efficient compression and good sound quality retention. Optionally, the difference between the sample compressed audio and the sample original audio segment can be measured by quantitative sound quality loss indicators (such as signal-to-noise ratio, distortion, etc.).

[0048] In this regard, the embodiment of the present invention can determine the training loss value based on the difference between the sample compressed audio and the sample original audio segment, and update the parameters of the initial generation model based on the training loss value to obtain the audio compression model. The initial generation model can be understood as the training starting point of the audio compression model, which is an untrained model whose parameters are randomly generated. The process of training the initial generation model is the process of optimizing its parameters.

[0049] Based on any of the above embodiments, based on the difference between the sample compressed audio and the sample original audio segment, the initial generation model of the audio compression model is trained to obtain the audio compression model, including: Based on the difference between the sample decoding features and the sample encoding features, and the difference between the sample compressed audio and the sample original audio fragment, the initial generation model of the audio compression model is trained to obtain the audio compression model.

[0050] Specifically, considering that information loss is inevitable during the audio compression process, information loss can occur in the encoding stage (i.e., the process of converting the original audio into a compressed format) and the decoding stage (i.e., the process of restoring the compressed format to the original audio). If only the difference between the sample compressed audio and the sample original audio segment is considered, it can only reflect the overall difference between the decoded audio and the original audio, but cannot accurately capture the specific situation of information loss during the encoding process.

[0051] For example, during the encoding process, the sample encoding features may have information loss, while during the decoding process, the features are reconstructed based on the sample quantization features. If the decoded sample compressed audio is slightly different from the sample original audio segment, it means that the decoding process retains the important features of the original audio segment, so that the sample compressed audio with a small difference from the sample original audio segment can be reconstructed based on the sample decoding features. At this time, if the difference between the sample encoding features and the sample decoding features is large, since the decoding process retains the important features of the sample original audio segment, it means that there is information loss in the encoding process, that is, the sample encoding features have lost some important features of the sample original audio segment.

[0052] In this regard, the embodiment of the present invention trains the initial generation model of the audio compression model based on the difference between the sample decoding features and the sample encoding features, and the difference between the sample compressed audio and the sample original audio fragment. That is to say, during the training process, the embodiment of the present invention not only considers the overall difference between the decoded audio and the original audio, but also considers the information loss in the encoding and decoding process, so as to more accurately optimize the compression algorithm, so that when the trained audio compression model is used for audio compression, the compressed audio obtained has a smaller data volume while maintaining better sound quality.

[0053] The sample coding feature is obtained by encoding the original audio clip of the sample, and thus the sample coding feature retains the main features of the original audio clip of the sample. The sample decoding feature is a feature reconstructed based on the sample quantization feature. If the difference between the sample decoding feature and the sample coding feature is small, it means that in the process of encoding and decoding the original audio clip of the sample, the main features of the original audio clip of the sample are retained as much as possible.

[0054] It can be seen that the embodiments of the present invention combine the differences between sample decoding features and sample encoding features, as well as the differences between sample compressed audio and sample original audio segments for training, which can comprehensively improve the performance of the audio compression model and make it more efficient, accurate and reliable in practical applications.

[0055] Based on any of the above embodiments, the audio compression model is combined with the discriminant model to perform generative adversarial training; Based on the difference between the sample compressed audio and the sample original audio segment, the initial generation model of the audio compression model is trained to obtain the audio compression model, including: Inputting the sample compressed audio into the initial discriminant model to obtain a first classification result output by the initial discriminant model; Inputting the sample original audio clip into the initial discriminant model to obtain a second classification result output by the initial discriminant model; Based on the difference between the first classification result and the corresponding classification label of the sample compressed audio, the difference between the second classification result and the corresponding classification label of the sample original audio segment, and the difference between the sample compressed audio and the sample original audio segment, the initial generation model and the initial discriminant model are jointly trained to obtain an audio compression model.

[0056] Specifically, the audio compression model is jointly obtained with the discriminant model through generative adversarial training, that is, the audio compression model can be regarded as a generative model, and its training goal is to make the generated sample compressed audio close to the original audio clip of the sample, that is, to retain the main features of the original audio clip of the sample as much as possible while reducing the amount of data.

[0057] In generative adversarial training, the discriminative model will constantly challenge the audio compression model, forcing it to generate more diverse and difficult to distinguish audio signals, which will help enhance the audio compression model's adaptability to different types of audio data and improve its robustness and generalization capabilities.

[0058] If the discriminant model is not introduced, the audio compression model may easily fall into overfitting, that is, it may rely too much on the training data and fail to process untrained audio clips well, resulting in the weakening of the generalization ability of the audio compression model and the inability to adapt to the diverse needs in practical applications.

[0059] In this regard, the embodiment of the present invention introduces a discriminant model to perform generative adversarial training to improve the quality, robustness and generalization ability of the audio compression model and promote the innovation of the audio compression model. The specific training method is as follows: The sample compressed audio is input into the initial discriminant model to obtain the first classification result output by the initial discriminant model, and the sample original audio segment is input into the initial discriminant model to obtain the second classification result output by the initial discriminant model. The initial discriminant model can be understood as the starting point of the discriminant model training, and its parameters are random.

[0060] Based on the difference between the first classification result and the corresponding classification label of the sample compressed audio, the difference between the second classification result and the corresponding classification label of the sample original audio segment, and the difference between the sample compressed audio and the sample original audio segment, the initial generation model and the initial discriminant model are jointly trained to obtain an audio compression model.

[0061] Among them, the first classification result is used to characterize the predicted classification of the sample compressed audio by the initial discriminant model. The smaller the difference between the first classification result and the corresponding classification label of the sample compressed audio, the more it indicates that the initial discriminant model can accurately identify the sample compressed audio. The second classification result is used to characterize the predicted classification of the sample original audio segment by the initial discriminant model. The smaller the difference between the second classification result and the corresponding classification label of the sample original audio segment, the more it indicates that the initial discriminant model can accurately identify the sample original audio segment. The difference between the sample compressed audio and the sample original audio segment is used to characterize whether the initial generation model is capable of generating sample compressed audio that is as close to the sample original audio segment as possible, that is, the training goal of the initial generation model is to generate sample compressed audio that is as close to the sample original audio segment as possible to deceive the discriminant model. The training goal of the initial discriminant model is to accurately distinguish between the sample original audio segment and the sample compressed audio. By alternately optimizing the initial generation model and the initial discriminant model, an audio compression model and a discriminant model are obtained.

[0062] Optionally, based on the difference between the first classification result and the classification label corresponding to the sample compressed audio and the difference between the second classification result and the classification label corresponding to the sample original audio segment, a discriminator loss value can be determined; based on the difference between the sample compressed audio and the sample original audio segment, a generator loss value can be determined; based on the discriminator loss value and the generator loss value, an adversarial training loss value can be determined; and based on the adversarial training loss value, the initial generation model and the initial discriminant model parameters are updated to obtain an audio compression model and a discriminant model.

[0063] Based on any of the above embodiments, the initial discriminant model includes multiple branches, each branch is used to extract audio features of the input audio, and determine the classification result based on the audio features of the input audio, and the scale of the audio features extracted by each branch is different; Based on the difference between the first classification result and the classification label corresponding to the sample compressed audio, the difference between the second classification result and the classification label corresponding to the sample original audio segment, and the difference between the sample decoding feature and the sample encoding feature, the initial generation model and the initial discrimination model are jointly trained to obtain an audio compression model, including: Based on the differences between the audio features of the sample original audio clips extracted by each branch and the audio features of the sample compressed audio, the differences between the first classification result and the corresponding classification label of the sample compressed audio, the differences between the second classification result and the corresponding classification label of the sample original audio clip, and the differences between the sample compressed audio and the sample original audio clip, the initial generation model and the initial discriminant model are jointly trained to obtain an audio compression model.

[0064] Specifically, considering that audio usually contains multiple audio features, such as frequency, amplitude, timbre, etc., these audio features may be different at different scales, that is, for the same audio, audio features of different scales contain different audio detail information. Since the audio detail information contains key acoustic features and slight changes in sound events, capturing these audio detail information can help the initial discriminant model more accurately identify the sample original audio clips and the sample compressed audio.

[0065] When the initial discriminant model can keenly identify the subtle differences between the sample original audio segment and the sample compressed audio, the initial discriminant model can provide more valuable feedback to the initial generative model. The initial generative model can use this feedback to fine-tune the sample compressed audio it generates, making the sample compressed audio closer to the quality of the sample original audio segment and reducing the distortion or loss introduced by compression.

[0066] In addition, the initial discriminant model includes multiple branches, and the scales of the audio features extracted by each branch are different, that is, the frequency range of the input audio that each branch focuses on is different. In other words, some branches can be set to focus on high-frequency components, and some branches can be set to focus on low-frequency or mid-frequency components, which helps to retain the details of the high-frequency part while reducing unnecessary smoothing processing.

[0067] Based on this, the initial discriminant model introduced in the embodiment of the present invention includes multiple branches, each branch is used to extract the audio features of the input audio, and determine the classification results based on the audio features of the input audio, and the scales of the audio features extracted by each branch are different. For example, the initial discriminant model may include m identical network branches, each network branch is used to perform short-time Fourier transform (STFT) on the input audio features at different scales to obtain corresponding audio features. Optionally, m can be equal to 5, and the window length scales of the STFT corresponding to each network branch include 2048, 1024, 512, 256 and 128. The input audio of each network branch can first pass through a 2D convolution layer (the convolution kernel is 3*8 and the number of channels is 32), followed by a 2D convolution with increasing expansion rate in the time dimension (the expansion rates are 1, 2, and 4, respectively), and finally two 2D convolution layers provide the final prediction results.

[0068] On the basis of introducing multiple branches, based on the differences between the audio features of the sample original audio clips extracted by each branch and the audio features of the sample compressed audio, the difference between the first classification result and the corresponding classification label of the sample compressed audio, the difference between the second classification result and the corresponding classification label of the sample original audio clip, and the difference between the sample compressed audio and the sample original audio clip, the initial generative model and the initial discriminant model are jointly trained to obtain an audio compression model.

[0069] Among them, the difference between the audio features of the sample original audio clip extracted by each branch and the audio features of the sample compressed audio is used to evaluate whether the sample compressed audio generated by the initial generation model is close to the sample original audio clip in terms of audio details. The smaller the difference between the two, it indicates that the initial generation model can generate sample compressed audio that is close to the sample original audio clip.

[0070] Based on any of the foregoing embodiments, determining the complexity of the audio segment to be compressed includes: Determine the complexity of the audio segment to be compressed based on the audio complexity classification model; The audio complexity classification model is trained based on multiple sample audio clips and complexity labels of each sample audio clip.

[0071] Specifically, the audio complexity classification model can be a lightweight neural network model (such as mobilenet), so as to reduce the number of parameters and computational complexity. Before determining the complexity of the audio segment to be compressed based on the audio complexity classification model, the audio complexity classification model can be pre-trained, which can be achieved by executing the following steps: First, a large number of sample audio segments are collected, and the complexity labels of each sample audio segment are manually annotated. Then, the initial model is trained based on multiple sample audio segments and the complexity labels of each sample audio segment to obtain an audio complexity classification model. Among them, the sample audio segment here can be the same as the above-mentioned sample original audio segment, or it can be different from the above-mentioned sample original audio segment, and the embodiment of the present invention does not specifically limit this. Among them, the audio complexity classification model can be a gating unit.

[0072] Optionally, the sample audio clips may include voice clips, music clips, environmental sound clips, etc. The voice clips include DAPS dataset, Common Voice dataset, VCTK dataset, the music clips include MUSDB dataset, Jamendo dataset, and the environmental sound clips include AudioSet dataset. In addition, the sample audio clips may also be obtained by segmenting a larger audio file, such as splitting the audio file according to the duration of each audio clip of 15ms to obtain multiple sample audio clips.

[0073] Before training based on the sample audio clips, each sample audio clip can be normalized to the range of -24dbLUFS, so as to ensure that the loudness between different sample audio clips remains consistent, thereby avoiding model training deviations caused by loudness differences. In addition, the normalized sample audio clips can be resampled to 24khz, so as to reduce the dimension and amount of calculation of the data, accelerate the model training process and reduce resource consumption. Among them, the collected multiple sample audio clips can be divided into training sets, validation sets and test sets. The test set can be selected from the above-mentioned DAPS dataset, CommonVoice dataset, VCTK dataset, MUSDB dataset, Jamendo dataset, and AudioSet dataset. A certain proportion of data is mixed.

[0074] Based on any of the above embodiments, the complexity label of each sample audio segment is determined based on the following steps: determining a plurality of different quality indicator scores for each sample audio segment; Based on preset weights, weighted addition is performed on multiple different quality indicator scores of each sample audio segment to determine a quality score of each sample audio segment; Based on the quality score of each sample audio segment and the total number of quantization layers, a complexity label of the sample audio segment is determined.

[0075] Specifically, different quality index scores are obtained by evaluating the quality of the sample audio clips in different dimensions, which may include clarity, purity, dynamic range, timbre, etc. Optionally, the quality index may include PESQ (Perceptual Evaluation of Speech Quality), STOI (Short-Time Objective Intelligibility), MOS (Mean Opinion Score), WER (Word Error Rate), etc.

[0076] After determining the scores of each quality indicator, based on the preset weights, the weighted sum of multiple different quality indicator scores of each sample audio segment is performed to determine the quality score of each sample audio segment. The preset weights can be determined based on the importance of each quality indicator, and the higher the importance, the greater the corresponding weight.

[0077] As an optional embodiment, the quality score distribution of each sample audio segment can be counted, and each sample audio segment can be divided into complexity levels according to the quality score distribution and the total number of quantization layers. For example, if the total number of quantization layers is 8, 8 complexity levels can be divided according to the quality score distribution, and each complexity level corresponds to a corresponding quality score range, so that the complexity label of each sample audio segment can be determined according to the quality score of each sample audio segment.

[0078] Based on any of the above embodiments, Figure 2 is a flow chart of the audio compression model training method provided by the present invention, such as Figure 2 As shown in Figure 1, the initial generation model of the audio compression model includes an encoder, multiple quantization layers (such as Q1, Q2, Q3, Q4, ...) and a decoder. The audio compression model is obtained by generative adversarial training in conjunction with the discriminative model. Among them, the encoder can be composed of 1 one-dimensional convolution layer and 4 convolution blocks; the kernel size of the one-dimensional convolution layer is 7 and the number of channels is 32; each convolution block consists of 1 residual unit and 1 stride convolution downsampling layer, and the kernel size is twice the stride; the residual unit contains 2 convolutions with a kernel size of 3 and 1 jump connection, and finally connected to a one-dimensional convolution layer with a kernel size of 7; the strides of the 4 convolution blocks are 2, 4, 5, and 8 respectively, to achieve effective feature extraction. The decoder adopts a configuration mirrored with the encoder, replacing the convolution layer with a deconvolution layer to achieve audio signal reconstruction. The quantization layer can use RVQ (Residual Vector Quantizer) to achieve effective quantization of audio features. The initial discriminant model of the discriminant model includes multiple branches, each branch is used to extract audio features of the input audio and determine the classification result based on the audio features of the input audio, and the scale of the audio features extracted by each branch is different.

[0079] First, a sample original audio segment is obtained, and the sample original audio segment is input into an encoder for encoding to obtain sample encoding features. At the same time, the sample original audio segment is input into an audio complexity classification model to obtain the complexity of the sample original audio segment.

[0080] Next, based on the complexity of the sample original audio segment, a sample quantization layer for quantizing the sample coding features is determined from multiple quantization layers, and the sample coding features are input into the sample quantization layer for quantization to obtain the sample quantization features. For example, the total number of quantization layers is 8, and the complexity is normalized to 8 threshold ranges. If the complexity of the sample original audio segment is 5, it belongs to the [5,6] threshold range, so the first 5 RVQ layers are used as sample quantization layers.

[0081] After obtaining the sample quantization feature, the sample quantization feature is input into a decoder for decoding to obtain a sample decoding feature, and a sample compressed audio corresponding to the sample original audio segment is generated based on the sample decoding feature.

[0082] In addition, the sample compressed audio is input into the initial discriminant model to obtain the audio features of the sample compressed audio extracted by each branch in the initial discriminant model, and the first classification result output by the initial discriminant model. At the same time, the sample original audio segment is input into the initial discriminant model to obtain the audio features of the sample original audio segment extracted by each branch in the initial discriminant model, and the second classification result output by the initial discriminant model.

[0083] Based on the difference between the audio features of the sample original audio clip extracted by each branch and the audio features of the sample compressed audio, the multi-scale spectrum reconstruction loss value is determined; based on the difference between the first classification result and the corresponding classification label of the sample compressed audio, and the difference between the second classification result and the corresponding classification label of the sample original audio clip, the discrimination loss value is determined; based on the difference between the sample decoding feature and the sample encoding feature, the feature matching loss value is determined; based on the difference between the sample compressed audio and the sample original audio clip, the generation loss value is determined.

[0084] Based on the multi-scale spectrum reconstruction loss value, the discrimination loss value, the feature matching loss value and the generation loss value, the training loss value is determined, and the initial generation model and the initial discrimination model are jointly trained based on the training loss value to obtain the audio compression model and the discrimination model.

[0085] The audio complexity classification model is trained based on multiple sample audio clips and the complexity labels of each sample audio clip. The complexity label of each sample audio clip is determined based on the following steps: determining multiple different quality index scores of each sample audio clip; based on preset weights, weighting and adding multiple different quality index scores of each sample audio clip to determine the quality score of each sample audio clip; based on the quality score of each sample audio clip and the total number of quantization layers, determining the complexity label of the sample audio clip.

[0086] Figure 3 is a flow chart of another audio compression method provided by the present invention, such as Figure 3 As shown, the audio segment to be compressed is resampled, and the resampled audio segment to be compressed is input into the audio compression model, and the audio compression model compresses the audio segment to be compressed based on the complexity of the audio segment to be compressed to obtain compressed audio.

[0087] The audio compression device provided by the present invention is described below. The audio compression device described below and the audio compression method described above can be referenced to each other.

[0088] Based on any of the above embodiments, Figure 4 is a schematic diagram of the structure of the audio compression device provided by the present invention, such as Figure 4 As shown, the device comprises: A determination unit 410, configured to determine the complexity of the audio segment to be compressed; A compression unit 420, configured to compress the audio segment to be compressed based on the audio compression model and the complexity of the audio segment to be compressed to obtain compressed audio; The audio compression model includes multiple quantization layers. The audio compression model is used to encode the audio segment to be compressed to obtain coding features, then use a target quantization layer to quantize the coding features, and decode the quantized coding features to obtain decoding features, and determine the compressed audio based on the decoding features; the target quantization layer is at least one quantization layer selected from multiple quantization layers based on the complexity of the audio segment to be compressed.

[0089] Based on any of the above embodiments, the audio compression model is trained based on the following steps: Encode the sample original audio segment to obtain the sample encoding feature; Determining a sample quantization layer for quantizing coding features of the sample from a plurality of quantization layers based on the complexity of the original audio segment of the sample; Quantize the sample coding features based on the sample quantization layer to obtain the sample quantization features; Decoding the sample quantization feature to obtain the sample decoding feature, and generating the sample compressed audio corresponding to the sample original audio segment based on the sample decoding feature; Based on the difference between the sample compressed audio and the sample original audio segment, an initial generation model of the audio compression model is trained to obtain the audio compression model.

[0090] Based on any of the above embodiments, based on the difference between the sample compressed audio and the sample original audio segment, the initial generation model of the audio compression model is trained to obtain the audio compression model, including: Based on the difference between the sample decoding features and the sample encoding features, and the difference between the sample compressed audio and the sample original audio fragment, the initial generation model of the audio compression model is trained to obtain the audio compression model.

[0091] Based on any of the above embodiments, the audio compression model is combined with the discriminant model to perform generative adversarial training; Based on the difference between the sample compressed audio and the sample original audio segment, the initial generation model of the audio compression model is trained to obtain the audio compression model, including: Inputting the sample compressed audio into the initial discriminant model to obtain a first classification result output by the initial discriminant model; Inputting the sample original audio clip into the initial discriminant model to obtain a second classification result output by the initial discriminant model; Based on the difference between the first classification result and the corresponding classification label of the sample compressed audio, the difference between the second classification result and the corresponding classification label of the sample original audio segment, and the difference between the sample compressed audio and the sample original audio segment, the initial generation model and the initial discriminant model are jointly trained to obtain an audio compression model.

[0092] Based on any of the above embodiments, the initial discriminant model includes multiple branches, each branch is used to extract audio features of the input audio, and determine the classification result based on the audio features of the input audio, and the scale of the audio features extracted by each branch is different; Based on the difference between the first classification result and the classification label corresponding to the sample compressed audio, the difference between the second classification result and the classification label corresponding to the sample original audio segment, and the difference between the sample compressed audio and the sample original audio segment, the initial generative model and the initial discriminant model are jointly trained to obtain an audio compression model, including: Based on the differences between the audio features of the sample original audio clips extracted by each branch and the audio features of the sample compressed audio, the differences between the first classification result and the corresponding classification label of the sample compressed audio, the differences between the second classification result and the corresponding classification label of the sample original audio clip, and the differences between the sample compressed audio and the sample original audio clip, the initial generation model and the initial discriminant model are jointly trained to obtain an audio compression model.

[0093] Based on any of the foregoing embodiments, determining the complexity of the audio segment to be compressed includes: Determine the complexity of the audio segment to be compressed based on the audio complexity classification model; The audio complexity classification model is trained based on multiple sample audio clips and complexity labels of each sample audio clip.

[0094] Based on any of the above embodiments, the complexity label of each sample audio segment is determined based on the following steps: determining a plurality of different quality indicator scores for each sample audio segment; Based on preset weights, weighted addition is performed on multiple different quality indicator scores of each sample audio segment to determine a quality score of each sample audio segment; Based on the quality score of each sample audio segment and the total number of quantization layers, a complexity label of the sample audio segment is determined.

[0095] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the audio compression method, which includes: determining the complexity of the audio segment to be compressed; based on the audio compression model, applying the complexity of the audio segment to be compressed, compressing the audio segment to be compressed to obtain compressed audio; the audio compression model includes multiple quantization layers, the audio compression model is used to encode the audio segment to be compressed to obtain encoding features, quantize the encoding features using a target quantization layer, and decode the quantized encoding features to obtain decoding features, and determine the compressed audio based on the decoding features; the target quantization layer is at least one quantization layer selected from the multiple quantization layers based on the complexity of the audio segment to be compressed.

[0096] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the audio compression method provided by the above methods, which method includes: determining the complexity of an audio segment to be compressed; based on an audio compression model, applying the complexity of the audio segment to be compressed, compressing the audio segment to be compressed to obtain compressed audio; the audio compression model includes multiple quantization layers, and the audio compression model is used to encode the audio segment to be compressed to obtain encoding features, then use a target quantization layer to quantize the encoding features, and decode the quantized encoding features to obtain decoding features, and determine the compressed audio based on the decoding features; the target quantization layer is at least one quantization layer selected from the multiple quantization layers based on the complexity of the audio segment to be compressed.

[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the audio compression method provided by the above-mentioned methods, the method comprising: determining the complexity of an audio segment to be compressed; based on an audio compression model, applying the complexity of the audio segment to be compressed, compressing the audio segment to be compressed to obtain compressed audio; the audio compression model comprises multiple quantization layers, the audio compression model is used to encode the audio segment to be compressed to obtain encoding features, then use a target quantization layer to quantize the encoding features, and decode the quantized encoding features to obtain decoding features, and determine the compressed audio based on the decoding features; the target quantization layer is at least one quantization layer selected from the multiple quantization layers based on the complexity of the audio segment to be compressed.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio compression method, characterized in that: include: Determining the complexity of the audio segment to be compressed; Based on the audio compression model, the complexity of the audio segment to be compressed is applied to compress the audio segment to obtain compressed audio; The audio compression model includes a plurality of quantization layers, and the audio compression model is used to encode the audio segment to be compressed to obtain encoding features, quantize the encoding features using a target quantization layer, decode the quantized encoding features to obtain decoding features, and determine the compressed audio based on the decoding features; The target quantization layer is at least one quantization layer selected from the multiple quantization layers based on the complexity of the audio segment to be compressed.

2. The audio compression method according to claim 1, characterized in that: The audio compression model is trained based on the following steps: Encode the sample original audio segment to obtain the sample encoding feature; Determining a sample quantization layer for quantizing the sample coding feature from the plurality of quantization layers based on the complexity of the sample original audio segment; quantizing the sample coding feature based on the sample quantization layer to obtain a sample quantization feature; Decoding the sample quantization feature to obtain a sample decoding feature, and generating a sample compressed audio corresponding to the sample original audio segment based on the sample decoding feature; Based on the difference between the sample compressed audio and the sample original audio segment, the initial generation model of the audio compression model is trained to obtain the audio compression model.

3. The audio compression method according to claim 2, characterized in that: The step of training the initial generation model of the audio compression model based on the difference between the sample compressed audio and the sample original audio segment to obtain the audio compression model comprises: Based on the difference between the sample decoding feature and the sample encoding feature, and the difference between the sample compressed audio and the sample original audio segment, the initial generation model of the audio compression model is trained to obtain the audio compression model.

4. The audio compression method according to claim 2, characterized in that: The audio compression model is obtained by performing generative adversarial training in conjunction with the discriminant model; The step of training the initial generation model of the audio compression model based on the difference between the sample compressed audio and the sample original audio segment to obtain the audio compression model comprises: Inputting the sample compressed audio into an initial discriminant model to obtain a first classification result output by the initial discriminant model; Inputting the sample original audio segment into the initial discriminant model to obtain a second classification result output by the initial discriminant model; Based on the difference between the first classification result and the classification label corresponding to the sample compressed audio, the difference between the second classification result and the classification label corresponding to the sample original audio segment, and the difference between the sample compressed audio and the sample original audio segment, the initial generative model and the initial discriminant model are jointly trained to obtain the audio compression model.

5. The audio compression method according to claim 4, characterized in that: The initial discriminant model includes a plurality of branches, each branch is used to extract audio features of the input audio, and determine a classification result based on the audio features of the input audio, and the scale of the audio features extracted by each branch is different; The method of jointly training the initial generative model and the initial discriminant model based on the difference between the first classification result and the classification label corresponding to the sample compressed audio, the difference between the second classification result and the classification label corresponding to the sample original audio segment, and the difference between the sample compressed audio and the sample original audio segment to obtain the audio compression model includes: Based on the differences between the audio features of the sample original audio segment extracted by each branch and the audio features of the sample compressed audio, the differences between the first classification result and the corresponding classification label of the sample compressed audio, the differences between the second classification result and the corresponding classification label of the sample original audio segment, and the differences between the sample compressed audio and the sample original audio segment, the initial generative model and the initial discriminant model are jointly trained to obtain the audio compression model.

6. The audio compression method according to any one of claims 1 to 5, characterized in that: The determining the complexity of the audio segment to be compressed comprises: Determining the complexity of the audio segment to be compressed based on an audio complexity classification model; The audio complexity classification model is trained based on a plurality of sample audio segments and complexity labels of each sample audio segment.

7. The audio compression method according to claim 6, characterized in that: The complexity label of each sample audio segment is determined based on the following steps: determining a plurality of different quality indicator scores for each sample audio segment; Based on preset weights, weighted addition is performed on multiple different quality indicator scores of each sample audio segment to determine a quality score of each sample audio segment; Based on the quality score of each sample audio segment and the total number of quantization layers, a complexity label of the sample audio segment is determined.

8. An audio compression device, characterized in that: include: A determination unit, used to determine the complexity of the audio segment to be compressed; A compression unit, configured to compress the audio segment to be compressed based on an audio compression model and the complexity of the audio segment to be compressed to obtain compressed audio; The audio compression model includes a plurality of quantization layers, and the audio compression model is used to encode the audio segment to be compressed to obtain encoding features, quantize the encoding features using a target quantization layer, decode the quantized encoding features to obtain decoding features, and determine the compressed audio based on the decoding features; The target quantization layer is at least one quantization layer selected from the multiple quantization layers based on the complexity of the audio segment to be compressed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the audio compression method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio compression method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the audio compression method according to any one of claims 1 to 7 is implemented.

Images