Audio encoding method, audio decoding method, song audio encoding and decoding method, device, medium and program product

Abstract

The invention relates to an audio encoding method, an audio decoding method, a song audio encoding and decoding method, equipment, a medium and a program product, relates to the technical field of audio encoding and decoding, and can improve the audio encoding and decoding quality. The method comprises the following steps: acquiring an original audio signal to be coded; obtaining a first high-frequency sub-band signal and a first low-frequency sub-band signal corresponding to the original audio signal based on a sub-band segmentation result of the original audio signal in a time domain; respectively inputting the first high-frequency sub-band signal and the first low-frequency sub-band signal into respective corresponding sub-band encoders to obtain sub-band encoding results output by the sub-band encoders; and determining time domain association information of the plurality of sub-band coding results, and obtaining an audio coding result of the original audio signal according to the time domain association information and the plurality of sub-band coding results.

Description

Technical Field

[0001] This application relates to the field of audio encoding and decoding technology, and in particular to an audio encoding method, an audio decoding method, a song audio encoding method, a song audio decoding method, a computer device, a readable storage medium, and a program product. Background Technology

[0002] To improve the transmission efficiency of audio files, the original audio signal of the audio file is often encoded before the encoded compressed file is sent to the receiving device.

[0003] In related technologies, in order to further save bandwidth resources, efficient neural network codecs have gradually become a hot application. For example, related time-domain codec models use a single-channel "encoder-quantizer-decoder" structure to model time-domain waveforms, which can achieve excellent encoding and decoding effects in speech reconstruction scenarios at extremely low bit rates.

[0004] However, in practice, it has been found that the above-mentioned audio encoding and decoding methods are difficult to reconstruct the low-frequency information of audio with high quality, and the reconstruction quality of audio encoding and decoding still needs to be improved. Summary of the Invention

[0005] Therefore, it is necessary to provide an audio encoding method, an audio decoding method, a song audio encoding method, a song audio decoding method, a computer device, a readable storage medium, and a program product to address the above-mentioned technical problems.

[0006] Firstly, this application provides an audio encoding method, including:

[0007] Obtain the raw audio signal to be encoded;

[0008] Based on the subband segmentation result of the original audio signal in the time domain, the first high-frequency subband signal and the first low-frequency subband signal corresponding to the original audio signal are obtained.

[0009] The first high-frequency sub-band signal and the first low-frequency sub-band signal are respectively input to their respective sub-band encoders to obtain the sub-band encoding results output by each sub-band encoder;

[0010] Determine the temporal correlation information of multiple sub-band coding results, and obtain the audio coding result of the original audio signal based on the temporal correlation information and the multiple sub-band coding results.

[0011] Secondly, this application also provides a song audio encoding method, including:

[0012] Obtain the raw audio signal of the song to be encoded;

[0013] Based on the original audio signal, the audio encoding result of the song audio is obtained; the audio encoding result is obtained by encoding the original audio signal using the audio encoding method described in any of the above claims.

[0014] Thirdly, this application also provides an audio decoding method, the method comprising:

[0015] Obtain the audio encoding result; the audio encoding result is obtained by encoding the original audio signal using the audio encoding method described in any of the preceding items;

[0016] Multiple sub-band encoding results are obtained based on the audio encoding results;

[0017] For each subband encoding result, the subband decoder corresponding to each subband encoding result is used for decoding to obtain the third high-frequency subband signal and the third low-frequency subband signal output by each subband decoder;

[0018] The decoded audio signal is obtained based on the third high-frequency sub-band signal and the third low-frequency sub-band signal.

[0019] Fourthly, this application also provides a method for decoding song audio, including:

[0020] In response to a request to obtain the song audio, the audio encoding result of the song audio is obtained;

[0021] The audio encoding result is decoded, and the decoded song audio signal corresponding to the song audio is obtained based on the decoding result; the decoding result is obtained by the audio decoding method described above.

[0022] Fifthly, this application also provides a computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described in any of the preceding claims.

[0023] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0024] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0025] The aforementioned audio encoding method, audio decoding method, song audio encoding method, song audio decoding method, computer device, readable storage medium, and program product, wherein the audio encoding method includes: after acquiring the original audio signal to be encoded, obtaining the first high-frequency sub-band signal and the first low-frequency sub-band signal corresponding to the original audio signal based on the sub-band segmentation result of the original audio signal in the time domain, and inputting the first high-frequency sub-band signal and the first low-frequency sub-band signal into their respective corresponding sub-band encoders to obtain the sub-band encoding results output by each sub-band encoder; furthermore, determining the time domain correlation information of multiple sub-band encoding results; and obtaining the audio encoding result of the original audio signal based on the time domain correlation information and the multiple sub-band encoding results. In this application, on the one hand, by sub-band segmentation of the original audio signal to obtain corresponding first high-frequency sub-band signals and first low-frequency sub-band signals, and processing the sub-band signals using corresponding sub-band encoders, the features of each sub-band signal can be extracted in a differentiated and targeted manner using different sub-band encoders, which helps to improve the accuracy of feature extraction of low-frequency components in the original audio signal. On the other hand, by fully exploring the temporal correlation information between high and low frequency components and obtaining audio coding results based on the temporal correlation information, high-quality signal quantization and coding can be performed using cross-band temporal correlation, improving the coding quality of the audio coding results. Therefore, the method provided in this embodiment can significantly improve the reconstruction quality of low-frequency components and the overall audio signal in the audio signal. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating an audio encoding method in one embodiment;

[0028] Figure 2 This is a schematic diagram of the structure of an audio codec model in one embodiment;

[0029] Figure 3 This is a flowchart illustrating a step of segmenting a sub-band signal in one embodiment;

[0030] Figure 4 This is a flowchart illustrating a song audio encoding method in one embodiment;

[0031] Figure 5 This is a flowchart illustrating an audio decoding method in one embodiment;

[0032] Figure 6 This is a flowchart illustrating a song audio decoding method in one embodiment;

[0033] Figure 7 This is a framework diagram of a song audio encoding and decoding method in one embodiment;

[0034] Figure 8 This is an internal structural diagram of a computer device in one embodiment;

[0035] Figure 9 This is an internal structural diagram of another computer device in one embodiment. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various objects, but these objects are not limited by these terms. These terms are only used to distinguish the first object from the second object. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the solutions, or any combination of multiple solutions.

[0038] In practice, to improve the transmission efficiency of audio files, the original audio signal of the audio file is often encoded before the encoded compressed file is sent to the receiving device.

[0039] Audio codecs in related technologies can be divided into traditional codecs and neural network-based codecs. Traditional codecs, based on linear predictive coding and modified discrete cosine transform (DCT), encode waveforms and parameters, achieving better audio coding quality at high bit rates compared to other methods. Neural network-based codecs, on the other hand, excel in low bit rate ranges. Examples include end-to-end time-domain or frequency-domain codecs based on Generative Adversarial Networks (GANs) at extremely low bit rates. These schemes obtain bitstreams below 24kbps through feature extraction, compression, and quantization using convolutional and recurrent neural networks, followed by signal reconstruction via a decoder.

[0040] For neural network-based encoding and decoding, in terms of network architecture design, the relevant technologies mainly use a single encoder-quantizer-decoder structure to model time-domain waveforms, such as EnCodec, DAC (Descript AudioCodec), SQCodec (Single Quantizer Codec) and other models.

[0041] Currently, to further conserve bandwidth resources, efficient neural network codecs have gradually become a research and application hotspot. However, among related technologies, neural network-based codecs struggle to reconstruct low-frequency information in audio with high quality, failing to meet the demands of various applications. For example, in music scenarios, higher bitrates are generally required for higher sound quality to preserve low-frequency vocal characteristics and other harmonic features. However, neural network-based codecs often fail to meet this requirement. Therefore, the reconstruction quality of their audio encoding and decoding still needs improvement.

[0042] In response, this application provides an audio encoding method, an audio decoding method, a computer device, a readable storage medium, and a program product, which can effectively improve the encoding and decoding quality of low-frequency components in audio signals by neural network-based codecs.

[0043] In one embodiment, such as Figure 1 As shown, an audio encoding method is provided. This embodiment illustrates the application of this method to an audio encoding terminal. Exemplarily, the audio encoding terminal can be a server, a terminal, or a system including both a terminal and a server, and the audio encoding method provided in this embodiment is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0044] S101, acquire the raw audio signal to be encoded.

[0045] The original audio signal to be encoded can refer to an audio signal that carries sound information directly without compression, encoding or other processing. In some examples, the original audio signal can contain one or more original signal features such as sound frequency, amplitude, and phase.

[0046] In some possible embodiments, the original audio signal can be the audio signal corresponding to the music, such as the audio signal recorded in the song file or the audio signal collected in real time when the user sings karaoke; of course, the original audio signal can also be the audio signal obtained in other scenarios, such as the audio signal generated when speaking.

[0047] In some embodiments, the original audio signal to be encoded can be obtained from pre-stored audio data or real-time acquired audio data. For example, the original audio signal to be encoded can be obtained from data in a pre-stored audio file, or real-time audio data can be obtained from a real-time audio stream sent by an audio acquisition device (such as a microphone built into a terminal device) or other servers, and the original audio signal can be obtained from the real-time audio data.

[0048] S102, based on the sub-band segmentation result of the original audio signal in the time domain, obtain the first high-frequency sub-band signal and the first low-frequency sub-band signal corresponding to the original audio signal.

[0049] Subband segmentation in the time domain can be understood as filtering the original audio signal in the time domain, decomposing it into multiple subband signals with different frequency ranges. The multiple subband signals obtained based on the subband segmentation process are the subband segmentation results in the time domain. In some examples, time-domain subband segmentation can directly filter the time-domain signal, using specific filters to ensure that the output signal retains components of specific frequency ranges from the original signal.

[0050] In practice, after acquiring the original audio signal, a preset filter bank can be used to perform sub-band segmentation on the original audio signal in the time domain, thereby obtaining multiple sub-band signals. Based on the multiple sub-band signals obtained from the segmentation, at least one high-frequency sub-band signal and at least one low-frequency sub-band signal can be obtained.

[0051] In this embodiment, the frequency range of each low-frequency sub-band signal can be smaller than the frequency range of each high-frequency sub-band signal. The low-frequency sub-band signal can be understood as the lower frequency range component obtained from the decomposition of the original audio signal, and the high-frequency sub-band signal can be understood as the higher frequency range component obtained from the decomposition of the original audio signal. For ease of distinction, in this embodiment, the high-frequency sub-band signal and the low-frequency sub-band signal used as input to the sub-band encoder are respectively referred to as the first high-frequency sub-band signal and the first low-frequency sub-band signal.

[0052] In some exemplary embodiments, a time-domain filter bank can be used to perform subband segmentation on the original audio signal. The time-domain filter bank may include a low-pass filter and a high-pass filter, wherein the low-pass filter can be used to extract low-frequency components from the original audio signal, and the high-pass filter can be used to extract high-frequency components from the original audio signal. Thus, a first high-frequency subband signal and a first low-frequency subband signal can be obtained based on the signal extraction results of the low-pass filter and the high-pass filter, respectively. For example, the output results of the low-pass filter and the high-pass filter can be used as the first low-frequency subband signal and the first high-frequency subband signal, respectively. Alternatively, the output results of the low-pass filter and the high-pass filter can be further processed to obtain the first low-frequency subband signal and the first high-frequency subband signal based on the processing results of the further processing.

[0053] S103, the first high-frequency sub-band signal and the first low-frequency sub-band signal are respectively input to their respective sub-band encoders to obtain the sub-band encoding results output by each sub-band encoder.

[0054] In related technologies, time-domain codecs based on neural networks all use a single-channel "encoder-quantizer-decoder" structure for time-domain waveform modeling. The modeling schemes of time-domain codecs mainly focus on full-band speech modeling, which fails to effectively meet the high-quality requirements of different audio components in some encoding and decoding scenarios.

[0055] In this embodiment, considering the reconstruction quality requirements of different components (e.g., higher sound quality requirements for low-frequency audio components), corresponding sub-band encoders can be set in advance for different types of sub-band signals. The sub-band encoder is used to extract features from the input low-frequency sub-band signal, and the sub-band encoder result is obtained based on the feature extraction result. Different types of sub-band signals correspond to different sub-band encoders. For example, the sub-band encoders corresponding to the first high-frequency sub-band signal and the first low-frequency sub-band signal are different.

[0056] Furthermore, after obtaining the first high-frequency sub-band signal and the first low-frequency sub-band signal, the first high-frequency sub-band signal can be input to the sub-band encoder corresponding to the first high-frequency sub-band signal to obtain the sub-band encoder corresponding to the first high-frequency sub-band signal output by the sub-band encoder; and the first low-frequency sub-band signal can be input to the sub-band encoder corresponding to the first low-frequency sub-band signal to obtain the sub-band encoder corresponding to the first low-frequency sub-band signal output by the sub-band encoder.

[0057] In some embodiments, if there are multiple first high-frequency sub-band signals, such as N1 (N1≥2), N1 sub-band encoders can be set accordingly, and the N1 first high-frequency sub-band signals can be input to the N1 sub-band encoders respectively; and / or, if there are multiple first low-frequency sub-band signals, such as N2 (N2≥2), N2 sub-band encoders can be set accordingly, and the N2 first low-frequency sub-band signals can be input to the N2 sub-band encoders respectively. For example, Figure 2 An audio codec model is provided as an example, in which multiple high-frequency encoders (i.e., subband encoders for processing first high-frequency subband signals) and a low-frequency encoder (i.e., subband encoders for processing first low-frequency subband signals) can be set up in parallel, and each high-frequency encoder and low-frequency encoder can selectively process each input subband signal in parallel.

[0058] S104, determine the temporal correlation information of multiple sub-band coding results, and obtain the audio coding result of the original audio signal based on the temporal correlation information and multiple sub-band coding results.

[0059] In this step, after obtaining multiple sub-band coding results, temporal correlation analysis can be performed on these results to determine their temporal correlation information. This temporal correlation information reflects the correlation data of multiple sub-band coding results in the time dimension, such as amplitude correlation, phase consistency, energy change trend correlation, or one or more other information related to different sub-band coding results within the same or adjacent time frames. By mining this temporal correlation information, the coordinated change patterns of each sub-band signal in the time domain can be revealed.

[0060] In one exemplary embodiment, a shared quantizer can be provided. The shared quantizer can be understood as a quantization processing module shared by multiple sub-band encoding results, which can adjust quantization parameters based on the overall characteristics of the multiple sub-band encoding results. The shared quantizer can be used to receive the sub-band encoder results output by multiple sub-band encoders. Then, the shared quantizer can determine the temporal correlation information of the multiple sub-band encoding results. Specifically, by inputting the sub-band encoding results corresponding to the first high-frequency sub-band signal and the first low-frequency sub-band signal together into the shared quantizer, the quantizer can fully exploit the temporal correlation between high and low frequency components in the original audio signal. In some embodiments, the frame rates of each sub-band signal branch are consistent after downsampling, and they can be combined and input as a whole into the shared quantizer to achieve cross-band modeling and obtain the transmission bitstream.

[0061] In another optional embodiment, a corresponding sub-band parallel quantizer can be configured for each sub-band signal branch. The sub-band parallel quantizer can perform quantization operations separately for each sub-band encoding result, allowing each sub-band parallel quantizer to select appropriate quantization parameters based on the signal characteristics of the sub-band encoding result itself, thereby improving the accuracy and effectiveness of the quantization result of a single sub-band. After the sub-band parallel quantizer completes the initial quantization of each sub-band encoding result, it can transmit the quantized sub-band data to a shared quantizer. The shared quantizer analyzes the received initially quantized sub-band data to generate time-domain correlation information that reflects the time-domain correlation characteristics of the sub-band encoding result.

[0062] After obtaining the temporal correlation information, quantization processing can be performed based on the temporal correlation information and multiple sub-band coding results. The audio coding result of the original audio signal can then be obtained based on the quantization processing result. In some embodiments, after obtaining the temporal correlation information, multiple sub-band coding results can be combined for fusion processing to obtain an audio coding result that can completely represent the original audio signal. The main processing steps in this step include using the temporal correlation information to optimize the redundancy removal and information complementarity of the sub-band coding results, avoiding the loss of temporal coherence caused by processing the sub-band coding results separately.

[0063] In one exemplary embodiment, the shared quantizer can identify redundant sub-band encoded data in multiple sub-band encoding results based on temporal correlation information. For example, if the temporal correlation information shows that the energy correlation coefficient between sub-bands Z1 and Z2 is greater than a preset threshold, and their energy change trends are consistent across multiple time frames, it can be determined that there is temporal data redundancy between Z1 and Z2. In this case, the more informative complete sub-band encoding result of Z1 and the parts of the Z2 sub-band encoding result that differ significantly from the Z1 sub-band encoding result can be retained to reduce the amount of data. Furthermore, temporal correlation information can also be used to supplement missing information in the sub-band encoding results. Subsequently, the multiple sub-band encoding results adjusted according to the temporal correlation information can be quantized and encapsulated according to a preset encoding format to generate the audio encoding result of the original audio signal.

[0064] In this step, by obtaining the temporal correlation information of multiple sub-band coding results, and obtaining the audio coding result based on the temporal correlation information and multiple sub-band coding results, cross-band modeling can be achieved. By automatically mining and utilizing the temporal correlation of cross-band, the coding quality of the audio coding result can be improved, which helps to ensure the synchronization and coordination of each frequency band component in time when reconstructing the signal, and generate aurally harmonious audio.

[0065] In the aforementioned audio encoding method, after acquiring the original audio signal to be encoded, the first high-frequency sub-band signal and the first low-frequency sub-band signal corresponding to the original audio signal can be obtained based on the sub-band segmentation result of the original audio signal in the time domain. The first high-frequency sub-band signal and the first low-frequency sub-band signal are then input to their respective corresponding sub-band encoders to obtain the sub-band encoding results output by each sub-band encoder. Furthermore, the time-domain correlation information of multiple sub-band encoding results can be determined. Based on the time-domain correlation information and multiple sub-band encoding results, the audio encoding result of the original audio signal is obtained. In this embodiment, on the one hand, by performing sub-band segmentation on the original audio signal to obtain the corresponding first high-frequency sub-band signal and the first low-frequency sub-band signal, and processing the sub-band signals using the corresponding sub-band encoders, the features of each sub-band signal can be extracted differentially and specifically using different sub-band encoders, which helps improve the accuracy of feature extraction of low-frequency components in the original audio signal. On the other hand, by fully exploring the time-domain correlation information between high and low frequency components and obtaining the audio encoding result based on the time-domain correlation information, high-quality signal quantization and encoding can be performed using cross-band time-domain correlation, improving the encoding quality of the audio encoding result. Therefore, the method provided in this embodiment can significantly improve the reconstruction quality of low-frequency components in the audio signal and the overall audio signal.

[0066] In one embodiment, such as Figure 3 As shown, in step S102, based on the sub-band segmentation result of the original audio signal in the time domain, the first high-frequency sub-band signal and the first low-frequency sub-band signal corresponding to the original audio signal are obtained, which may include the following steps:

[0067] S301: The original audio signal is input to the subband analysis filter. Based on the subband segmentation result of the original audio signal in the time domain by the subband analysis filter, the second high-frequency subband signal and multiple second low-frequency subband signals are obtained.

[0068] The subband analysis filter can be used to perform subband analysis on the input signal, decomposing the input signal into multiple subbands, each corresponding to a different frequency range. In some embodiments, the subband analysis filter can specifically be an analysis filter bank. When performing subband segmentation, the time-domain signal (i.e., the original audio signal) can be passed through a set of bandpass filters, with each filter corresponding to a subband, and the output is the downsampled subband signal.

[0069] In a specific implementation, the original audio signal can be input into a subband analysis filter, which segments the original audio signal in the time domain to obtain subband segmentation results. These subband segmentation results can include multiple subband signals corresponding to different frequency ranges, and these frequency ranges can be non-overlapping. Then, based on the frequency ranges of the multiple subband signals, one or more subband signals corresponding to the frequency range with a higher lower frequency limit are identified as high-frequency subband signals, and multiple subband signals corresponding to the frequency range with a lower lower frequency limit are identified as low-frequency subband signals. For ease of distinction, in this embodiment, the high-frequency subband signal directly determined and output by the subband analysis filter is referred to as the second high-frequency subband signal, and the low-frequency subband signal directly determined and output by the subband analysis filter is referred to as the second low-frequency subband signal.

[0070] S302, based on the subband synthesis results of multiple second low-frequency subband signals by the subband synthesis filter, the first low-frequency subband signal corresponding to the original audio signal is obtained; and based on the subband segmentation results of the second high-frequency subband signal by the subband analysis filter, multiple first high-frequency subband signals corresponding to the original audio signal are obtained.

[0071] Among them, the subband synthesis filter can be used to synthesize and filter multiple subband signals to obtain the reconstructed signal.

[0072] To further improve the reconstruction quality of the low-frequency components of the audio signal, in this embodiment, after obtaining the second high-frequency sub-band signal and multiple second low-frequency sub-band signals, on the one hand, the multiple second low-frequency sub-band signals can be synthesized and filtered using a sub-band synthesis filter to obtain a sub-band synthesis result. The sub-band synthesis result can be used as the first low-frequency sub-band signal. Through this processing, the first low-frequency sub-band signal can contain sufficient data, increasing the overall complexity and number of parameters of the low-frequency branch. This allows the sub-band encoder corresponding to the first low-frequency sub-band signal to obtain a more accurate low-frequency signal sub-band encoding result based on more sufficient information.

[0073] On the other hand, for the high-frequency components, i.e., the second high-frequency sub-band signal, a sub-band analysis filter can be used to further segment the second high-frequency sub-band signal in the time domain to obtain the corresponding sub-band segmentation result. Then, the first low-frequency sub-band signal is obtained based on this sub-band segmentation result. In other words, in this embodiment, the high-frequency components in the original audio signal can be subjected to one or more (e.g., two or more) analysis and filtering processes to obtain the first high-frequency sub-band signal. For example, as... Figure 2The model framework diagram shown is illustrated using two-step analysis and filtering as an example. After two-step analysis and filtering, four high-frequency branches with a bandwidth of 1 / 16 can be obtained, namely four first high-frequency sub-band signals. Thus, the high-frequency branches increase the number of channels and reduce the input window length through further filtering, so that the input sequence length of the sub-band encoder of each high-frequency branch is reduced to 1 / 16 of the original, thereby reducing the overall complexity and number of parameters of the sub-band encoder of the high-frequency branches.

[0074] In this embodiment, on the one hand, for the low-frequency components in the original audio signal, the subband synthesis filter is used to obtain the subband synthesis results of multiple second low-frequency subband signals, which can increase the amount of information contained in the first low-frequency subband signal and improve the accuracy of the subband coding results of the low-frequency components. On the other hand, for the high-frequency components in the original audio signal, the subband segmentation results of the second high-frequency subband signal are obtained through the subband analysis filter, resulting in a first high-frequency subband signal with reduced data volume. This can reduce the overall complexity and number of parameters of the high-frequency branch, thereby allowing more attention to the reconstruction quality of the low-frequency components during the audio coding process, while improving the audio encoding and decoding efficiency.

[0075] In one embodiment, step S302, obtaining the first low-frequency sub-band signal corresponding to the original audio signal based on the sub-band synthesis result of multiple second low-frequency sub-band signals using the sub-band synthesis filter, may include the following steps:

[0076] Determine the total number of subbands corresponding to the second high-frequency subband signal and multiple second low-frequency subband signals; upsample the multiple second low-frequency subband signals according to the total number of subbands to obtain the upsampling result; perform synthesis filtering on the upsampling result by the subband synthesis filter, and obtain the first low-frequency subband signal according to the synthesis filtering result.

[0077] The total number of subbands refers to the sum of the number of second high-frequency subband signals and the number of multiple second low-frequency subband signals. The total number of subbands can reflect the granularity of frequency band division after signal decomposition.

[0078] Upsampling is used to increase the sampling rate of a signal and supplement signal details, so that the sampling frequencies of multiple second low-frequency subband signals can meet the information requirements; for example, it can be achieved by inserting new samples between the original sampling points through interpolation (such as linear interpolation or polynomial interpolation).

[0079] In practical applications, subband signals can be decomposed into multiple frequency bands from the original audio signal using filter banks. The total number of subbands can be determined by summing the number of the second high-frequency subband and the second low-frequency subband.

[0080] In some embodiments, after obtaining the total number of subbands, an upsampling factor can be determined based on the total number of subbands. By determining the upsampling factor based on the total number of subbands, the subsequent obtained first low-frequency subband signal can maintain its original window length. Then, multiple second low-frequency subband signals can be upsampled according to the upsampling factor to obtain the upsampling result. The upsampling result is then synthesized and filtered by a subband synthesis filter, and the output of the synthesis and filtering process is used as the first low-frequency subband signal.

[0081] In some embodiments, high-frequency zero-padding can be applied to multiple second low-frequency sub-band signals to maintain the input window length of the low-frequency branch as T. For example, as... Figure 2 As shown, after acquiring the original audio signal, eight sub-band signals (i.e., the number of sub-bands is 8) can be obtained through analysis and filtering. The window length of each sub-band signal is 8 / T. For the six low-frequency sub-band signals, high-frequency zero-padding is performed before synthesis filtering to ensure that the window length of the low-frequency sub-band signal input to the low-frequency encoder remains at T.

[0082] In this embodiment, by upsampling multiple second low-frequency subband signals according to the total number of subbands, an upsampling result is obtained. This enables adaptive upsampling of the number of subbands obtained after subband segmentation, ensuring that the first low-frequency subband signal has a suitable sampling rate and data volume before being input to the subband encoder, which helps to improve the subband coding quality of the low-frequency branch.

[0083] In one embodiment, the subband analysis filter and the subband synthesis filter can be a filter bank within a pseudo-quadrature mirror filter (PQMF) group. The PQMF group is used for audio signal decomposition and reconstruction, achieving efficient subband coding and processing by decomposing the time-domain signal into multiple subband signals.

[0084] To improve the accuracy of subband segmentation results and the high-quality reconstruction of the original audio signal based on subband reconstruction, this embodiment can determine the subband analysis filter and the subband synthesis filter based on a pseudo-orthogonal mirror filter bank.

[0085] Specifically, the pseudo-orthogonal mirror filter bank relaxes the strict constraint of perfect reconstruction by superimposing cosine modulation on a low-pass filter based on finite impulse response (FIR). In the pseudo-orthogonal mirror filter bank, analytical filtering and synthetic filtering are performed. Opposite phases cause the aliasing terms of the upper and lower mirror images to cancel each other out during subband synthesis, and symmetrical zero padding is used to cancel the group delay introduced by the linear phase FIR. In one example, for a pseudo-orthogonal mirror filter bank, for M subband signals, the impulse response of the analysis filter for the k-th subband is... impulse response of the synthesis filter They are respectively:

[0086]

[0087]

[0088] Where k is the sub-band index, This is a prototype filter based on the sinc function, employing a Kaiser window to suppress sidelobes and control the transition band. L is the filter length. Subband filters share the same phase reference point, therefore... By shifting the time axis to the center of the filter, the synthesized filters of the corresponding subbands will have the same frequency shift but opposite phase.

[0089] The pseudo-orthogonal mirror filter bank involves upsampling and downsampling; that is, the analysis filter achieves isochronous decimation through a one-dimensional convolution with a stride of M, while the synthesis filter achieves zero-interpolation upsampling through a deconvolution with a stride of M. In some embodiments, to recover the original signal amplitude, a global gain M can be multiplied to approximate the power loss.

[0090] In some possible implementations, when constructing the pseudo-orthogonal mirror filter, the prototype filter length can be 62, the number of subbands M can be set to 8, and the normalized cutoff frequency, which is strongly negatively correlated with the number of subbands, can be set to 0.0795. For example... Figure 2 As shown, the framework includes 4 high-frequency branches and 1 low-frequency branch. For the low-frequency branch, 8 sub-bands can be obtained through a pseudo-orthogonal mirror filter PQMF(8). For the low-frequency signals of the first 6 extracted sub-bands, the input window length of the low-frequency branch is kept at T by zero-padding at high frequencies before synthesis filtering, and the step size of the two-layer convolution downsampling is set to [16,4]. For the high-frequency branch, the high-frequency signals of the last 2 high-frequency sub-bands of the 8 sub-bands can be downsampled again through a pseudo-orthogonal mirror filter PQMF(2) to obtain 4 parallel high-frequency sub-bands with a length of T / 16. Correspondingly, the downsampling convolution step size of each high-frequency branch at the same frame rate is [2,2], which reduces the overall complexity and number of parameters of the high-frequency branch.

[0091] In this embodiment, by utilizing the filters in the pseudo-orthogonal mirror filter bank, subband analysis filters and subband synthesis filters are obtained, which can effectively improve the accuracy of subband segmentation results and subband synthesis results, achieve accurate subband segmentation and near-perfect reconstruction in the time domain, thereby improving the encoding and decoding quality of each high-frequency branch and low-frequency branch.

[0092] In related technologies, neural network-based codecs primarily utilize convolution to extract local features within a specific receptive field at the feature extraction level. However, limited by the receptive field of ordinary convolution, the obtained feature results cannot comprehensively and accurately reflect the signal characteristics.

[0093] In one embodiment, each sub-band encoder among multiple sub-band encoding results can obtain the sub-band encoding result through the following steps:

[0094] The input subband signal corresponding to the subband encoder is pooled using pooling kernels of multiple sizes. Multi-scale envelope features are obtained based on the pooling results of the pooling kernels of multiple sizes. Subband encoding results are obtained based on the multi-scale envelope features.

[0095] The input subband signal includes either the first high-frequency subband signal or the first low-frequency subband signal input to the subband encoder.

[0096] A pooling kernel is a window used to downsample a signal and extract local features. In some examples, the size of the pooling kernel can be defined by the "time step" or the "number of sampling points". Multiple pooling kernels can be two or more different window sizes.

[0097] This embodiment improves the structure of the sub-band encoder. Specifically, compared to the related technology that simply performs up-channel convolution on the input data to obtain local features within a specific receptive field, this embodiment uses pooling kernels of multiple sizes to pool the input sub-band signal to the sub-band encoder. That is, for each sub-band encoder, the first high-frequency sub-band signal or the first low-frequency sub-band signal input to the sub-band encoder can be pooled at different scales using pooling kernels of multiple sizes contained in the sub-band encoder. Then, based on the pooling results, envelope features corresponding to pooling kernels of different sizes can be obtained, i.e., multi-scale envelope features.

[0098] For example, the subband encoder can be equipped with a pooling-convolution module, which includes multiple pooling kernels with sizes of 1, 5, 11, 21, and 45. For the input subband signal 1, the input subband signal 1 can be subjected to max pooling and average pooling operations using the above five pooling kernels to obtain the pooling processing result corresponding to the input subband signal 1. Then, the pooling processing result can be subjected to convolution processing, and the convolution processing result can be used as a multi-scale envelope feature.

[0099] In this embodiment, by using pooling kernels of multiple sizes to perform pooling processing on the input subband signal corresponding to the subband encoder, and obtaining multi-scale envelope features based on the pooling processing results of each of the multiple pooling kernels, it is possible to use a larger, multi-sized receptive field for modeling, effectively improving the subband encoder's global modeling capability for the input subband signal and enhancing the accuracy of the subband encoding results.

[0100] In some related technologies, to improve the temporal modeling level of the codec, network architectures such as transformer attention architecture or Long Short-Term Memory (LSTM) networks are introduced. However, this temporal modeling scheme increases the computational overhead of the model. Therefore, in one embodiment of this application, obtaining sub-band encoding results based on multi-scale envelope features may include the following steps:

[0101] The multi-scale envelope features are input into the residual convolution module in the sub-band encoder. The residual convolution module performs depthwise separable convolution processing on the multi-scale envelope features. The sub-band encoding result is obtained based on the depthwise separable convolution processing result.

[0102] In practical applications, the subband encoder may include a residual convolutional module based on depthwise separable convolution (also known as the residual convolutional network ConvNeXt). This residual convolutional module may consist of depthwise separable convolution, a preset activation function (such as the snake activation function), and global response normalization.

[0103] After obtaining the multi-scale envelope features, these features can be input into the residual convolution module for feature extraction. The residual convolution module performs feature extraction processing, where depthwise separable convolution consists of channel convolution and point convolution, which can reduce network complexity while maintaining modeling quality. In some embodiments, this feature extraction process includes depthwise separable convolution and residual operations. After obtaining the depthwise separable convolution processing result, the residual processing result can be obtained based on the depthwise separable convolution processing result and the original input multi-scale envelope features. Furthermore, the sub-band encoding result can be determined based on the residual processing result.

[0104] In this embodiment, by introducing a residual convolution module into the subband encoder and using depthwise separable convolution to process multi-scale envelope features, it is possible to reduce network complexity and minimize computational requirements while ensuring efficient feature extraction capabilities and modeling quality.

[0105] Furthermore, compared to the related technologies that mainly use multi-layer convolution with a specific stride for feature extraction during feature compression in the encoding and decoding process, this embodiment introduces a residual convolution module, which allows the sub-band encoder to set fewer downsampling layers to further reduce model complexity and the number of parameters.

[0106] In some related technologies, LSTM modules are mainly used for timing modeling, but this process has a high computational cost. Therefore, in one embodiment of this application, obtaining the sub-band encoding result based on the depthwise separable convolution processing result may further include the following steps:

[0107] The depthwise separable convolution processing result is input into the bidirectional gated recurrent unit network in the sub-band encoder. Based on the output characteristics of the bidirectional gated recurrent unit network, the sub-band encoding result is obtained.

[0108] In this embodiment, when performing timing modeling of subband signals, a bidirectional gated cyclic unit network can be used to replace the LSTM module in the traditional technology. For example, a single bidirectional gated cyclic unit network can be used to replace the commonly used two-layer LSTM module in related technologies.

[0109] Specifically, a bidirectional gated recurrent unit network can be set in the sub-band encoder. After obtaining the depthwise separable convolution processing result, the depthwise separable convolution processing result can be input into the bidirectional gated recurrent unit network. Then, the bidirectional gated recurrent unit network can concatenate the forward and reverse GRU outputs in the feature dimension, and then use a linear layer to perform feature dimensionality reduction.

[0110] In this embodiment, the main structure of the bidirectional gated recurrent unit network includes update gates and reset gates. Compared with the internal structure of LSTM, the bidirectional gated recurrent unit network has a simpler structure, fewer parameters, and less computation. At the same time, the bidirectional operation of the bidirectional gated recurrent unit network can capture past and future information simultaneously, effectively utilize positive and negative context information, and improve sequence modeling and time series data processing capabilities. Thus, the audio reconstruction quality can be improved while reducing the number and complexity of network parameters.

[0111] In practice, it has been found that the encoder-decoder structure combining deep residual convolution, pooling-convolution envelope feature extraction module and bidirectional gated recurrent unit (BiGRU) can improve the scores of indicators such as low-frequency log-spectral distance (LSD) and scale-invariant signal-to-noise ratio (SI-SDR) and subjective listening quality while reducing the number of network parameters and complexity.

[0112] In one embodiment, the subband encoder is a subband encoder in an audio codec model obtained through adversarial training. This audio codec model also includes a corresponding subband decoder for obtaining the subband decoding results. Specifically, a Generative Adversarial Network (GAN) can be constructed. This network is a deep learning framework composed of a generator network and a discriminator network. High-quality data generation and modeling are achieved through adversarial training of the generator network and the discriminator network.

[0113] In this embodiment, the generative network in a GAN can be used as the audio codec model to be trained. The audio codec model can include a codec, namely a sub-band encoder and a sub-band decoder corresponding to the sub-band encoder. There are multiple sub-band encoders, the number of which can be determined based on the number of sub-bands obtained from segmenting the original audio signal. Multiple sub-band encoders can be used to process multiple sub-band signals separately, and different sub-band encoders input different sub-band signals. The sub-band decoder corresponds to the sub-band encoder, and a mirror branch is used at the decoder end to model sub-bands. For example, it can be used to perform the opposite or corresponding processing on the sub-band signals processed by the sub-band encoder.

[0114] by Figure 2 Taking the audio codec model architecture shown as an example, the encoder module includes 4 high-frequency encoders and 1 low-frequency encoder, and the decoder module can include 4 high-frequency decoders and 1 low-frequency decoder. These 5 decoders correspond one-to-one with the 5 encoders in the encoder module and can be used to decode the subband encoding results of the corresponding branches to obtain the corresponding subband decoding results. For example, the subband decoders corresponding to the high and low frequency branches can all adopt the deconvolution operation with the step size corresponding to the subband encoder, and then perform corresponding analysis filtering and synthesis filtering through PQMF.

[0115] In this embodiment, the audio codec model can be trained based on the following steps:

[0116] S1, the sub-band encoding results of each first sample sub-band signal of the sample audio signal are obtained by the sub-band encoder in the audio codec model to be trained, and the sub-band decoding results of each first sample sub-band signal of the sample audio signal are obtained by the sub-band decoder in the audio codec model to be trained.

[0117] In specific implementation, the original audio signal used to train the audio codec model can be obtained. For ease of distinction, the original audio signal used to train the audio codec model is referred to as the sample audio signal. Subsequently, based on the sub-band segmentation results of the sample audio signal in the time domain, multiple sub-band signals, namely multiple first sample sub-band signals, can be obtained. The multiple first sample sub-band signals may include a first high-frequency sub-band signal and a first low-frequency sub-band signal. Then, the multiple first sample sub-band signals are respectively input into the corresponding sub-band encoders to obtain the sample sub-band encoding results. The method of obtaining multiple first sample sub-band signals can refer to the process of obtaining the first high-frequency sub-band signal and the first low-frequency sub-band signal from the original audio signal in the aforementioned embodiment. The process of obtaining multiple first sample sub-band signals can be the same as or similar to this, and will not be elaborated here.

[0118] After obtaining the sample subband encoding results, the temporal correlation information of multiple sample subband encoding results can be determined. Based on the temporal correlation information and multiple sample subband encoding results, the sample audio encoding result of the sample audio signal is obtained. The sample audio encoding results can be segmented to obtain the sample subband encoding results corresponding to different first sample subband signals. Then, the subband decoder in the audio codec model to be trained can process the obtained multiple sample subband encoding results to obtain the sample subband decoding result corresponding to each first sample subband signal. The sample subband decoding result can be the first sample subband signal obtained after decoding.

[0119] S2, for each first sample sub-band signal, determine the first sub-loss value of the first sample sub-band signal based on the sample sub-band decoding result.

[0120] In this step, after obtaining the decoding results of multiple sample subbands, for each first sample subband signal, the loss value corresponding to the first sample subband signal can be determined based on the decoding result of the sample subband corresponding to the first sample subband signal. For ease of distinction, this loss value is called the first sub-loss value.

[0121] In one embodiment, when determining the sub-loss value, the corresponding Mel spectrum can be determined based on the first sample subband signal obtained after decoding, and the first sub-loss value of the first sample subband signal can be determined based on the difference between the Mel spectrum corresponding to the first sample subband signal before encoding and the Mel spectrum corresponding to the first sample subband signal obtained after decoding.

[0122] In another embodiment, the signal features corresponding to the first sample subband signal obtained after decoding and the signal features corresponding to the first sample subband signal before encoding can also be obtained. Then, based on the feature differences between the signal features corresponding to the first sample subband signal obtained after decoding and the signal features corresponding to the first sample subband signal before encoding, the sub-loss value of the first sample subband signal can be determined.

[0123] In some embodiments, a quantization loss can also be introduced, which can be determined based on the difference between the input and output of the shared quantizer. Furthermore, the sub-loss value corresponding to the sample subband signal can be determined based on the Mel spectrum loss, quantization loss, and feature loss.

[0124] S3, determine the generated network loss value based on each first sub-loss value.

[0125] After obtaining the sub-loss values ​​of multiple first sample sub-band signals, the first sub-loss values ​​of the multiple sample sub-band signals can be fused to determine the generated network loss value. For example, the generated network loss value can be determined by direct summation, weighted summation, or other processing methods.

[0126] Furthermore, since the parameters of the audio codec model can be adjusted through adversarial training in this embodiment, the generative network loss value can also be determined by combining the discrimination results of the discriminative network on the sample audio signals (i.e., the audio signals determined based on the decoding results of multiple sample subbands) obtained by the audio codec model.

[0127] S4. Adjust the audio codec model to be trained based on the loss value of the generator network, and obtain the trained audio codec model when the adversarial training termination condition is met.

[0128] In practical applications, the audio codec model to be trained can be adjusted based on the loss value of the generator network, and adversarial training can be performed based on the adjusted audio codec model and the discriminator. For example, after adjusting the model parameters of the audio codec model to be trained based on the generator network loss value, the model parameters can be fixed, and the sample audio signals decoded by the audio codec model can be used as training samples to provide to the discriminator model for signal authenticity discrimination. The parameters of the discriminator network can be adjusted based on the discrimination results, and then the parameters of the discriminator network can be fixed again, and the audio codec model can be adjusted again. After repeated iterations, a trained audio codec model can be obtained.

[0129] In this embodiment, an audio codec model containing sub-band encoders and sub-band decoders is obtained through adversarial training, which helps to improve the codec quality of the audio codec model. At the same time, by obtaining the first sub-loss value corresponding to each sub-band signal branch, the loss value of the generator network is determined based on each first sub-loss value, so that each sub-band encoder can focus on feature encoding in a specific frequency range. During the model training process, the feature extraction and processing methods of each sub-band signal are fully learned, realizing differentiated optimization of sub-bands, which helps to improve the audio codec quality.

[0130] In one embodiment, each first sample sub-band signal includes a low-frequency sample sub-band signal and a high-frequency sample sub-band signal. That is, by segmenting the sample audio signal, multiple first sample sub-band signals, including low-frequency and high-frequency sample sub-band signals, can be obtained. In step S3, determining the generation network loss value based on each first sub-loss value may include the following steps:

[0131] Determine the first loss weight of the low-frequency subband signal and the second loss weight of the high-frequency subband signal; the first loss weight is greater than the second loss weight; determine the generation network loss value based on the first loss weight, the second loss weight, the first sub-loss value of the low-frequency subband signal and the first sub-loss value of the high-frequency subband signal.

[0132] After obtaining multiple first sub-loss values, to further improve the reconstruction quality of low-frequency components in the audio signal, in addition to performing sub-band encoding and decoding modeling, different weights can be configured for different sub-band branches. Specifically, the first loss weight of the sample low-frequency sub-band signal and the second loss weight of the sample high-frequency sub-band signal can be determined, with the first loss weight being greater than the second loss weight. Then, the generation network loss value can be determined based on the weighted fusion result of the first loss weight, the second loss weight, the first sub-loss value of the sample low-frequency sub-band signal, and the first sub-loss value of the sample high-frequency sub-band signal. For example, the ratio of the first loss weight to the second loss weight can be set to 100:1, and then a weighted sum is performed based on the first loss weight, the second loss weight, the first sub-loss value of the sample low-frequency sub-band signal, and the first sub-loss value of the sample high-frequency sub-band signal, yielding the generation network loss value from the weighted summation result.

[0133] In this embodiment, by making the first loss weight greater than the second loss weight, and then determining the generation network loss value based on the first loss weight, the second loss weight, the first sub-loss value of the sample low-frequency sub-band signal, and the first sub-loss value of the sample high-frequency sub-band signal, the model's ability to model low-frequency features can be further improved by increasing the loss weight of the low-frequency sub-band branch in the generation network.

[0134] In one embodiment, after adjusting the audio codec model to be trained based on the generator network loss value in step S4, the following steps are also included:

[0135] Multiple second sample sub-band signals are acquired; for each second sample sub-band signal, the discrimination network determines the second sub-loss value of the second sample sub-band signal based on the discrimination result corresponding to the second sample sub-band signal; the discrimination network loss value is determined based on each second sub-loss value, and the discrimination network is adjusted based on the discrimination network loss value.

[0136] Among them, multiple second sample subband signals are obtained based on the decoding results of multiple sample subbands.

[0137] In one embodiment, since each sample subband decoding result is a sample subband signal, this sample subband decoding result can be used as a second sample subband signal. In another embodiment, after obtaining multiple sample subband decoding results, the decoder segment in the audio codec model can perform further subband synthesis and subband analysis processing based on the sample subband decoding results and filters. The decoded sample audio signal is obtained based on the processing results, and then multiple second sample subband signals are obtained based on the subband signal segmentation results of the decoded sample audio signal. For example, as... Figure 2As shown, for the high-frequency components, after obtaining the four high-frequency sub-band signals decoded and output by the four high-frequency decoders, a synthesis filtering process can be performed to obtain two merged high-frequency sub-band signals. For the low-frequency components, after obtaining the one low-frequency sub-band signal output by the low-frequency decoder, a sub-band analysis filtering process can be performed to obtain two segmented signals, one containing two low-frequency sub-band signals and the other containing six low-frequency sub-band signals. Furthermore, based on the two merged high-frequency sub-band signals, the signal containing two low-frequency sub-band signals, and the signal containing six low-frequency sub-band signals, a synthesis process can be performed to obtain the decoded sample audio signal. By sub-segmenting the decoded sample audio signal, multiple second sample sub-band signals are obtained.

[0138] For each second sample sub-band signal, a discrimination network can obtain a discrimination result corresponding to the second sample sub-band signal. This discrimination result can characterize the authenticity identification result of the second sample sub-band signal; for example, it can be a probability value indicating that the second sample sub-band signal is true. Then, based on the discrimination result determined by the discrimination network, a second sub-loss value for the second sample sub-band signal can be determined. In one embodiment, the second sub-loss value can be composed of multi-scale, multi-resolution, and multi-period waveform discrimination terms.

[0139] Furthermore, the discrimination network loss value can be determined based on the second sub-loss value corresponding to each second sample sub-band signal, and the discrimination network can be adjusted based on the discrimination network loss value. In some embodiments, the discrimination network loss value can be determined based on the weighted summation result of each second sub-loss value.

[0140] In some embodiments, the multiple second sample subband signals include sample low-frequency subband signals and sample high-frequency subband signals. When determining the second sub-loss value, the first loss weight corresponding to the sample low-frequency subband signal can be greater than the second loss weight of the sample high-frequency subband signal (e.g., 100:1). Thus, the model's ability to distinguish and model low-frequency features can be further improved by increasing the loss weight of the low-frequency subband branch in the discriminant network.

[0141] In this embodiment, during the training of the discriminant network, the second sub-loss value corresponding to each sub-band signal branch is obtained separately, and the discriminant network loss value is determined based on each second sub-loss value. This allows the discriminant network to perform specialized and differentiated learning for the unique acoustic characteristics of different sub-bands, enabling the development of a discrimination strategy for specific frequency bands, rather than using a single, coarse-grained discrimination standard. As a result, the discriminant network can be more accurate in identifying defects in the generated audio. By conducting adversarial training with the audio codec model that serves as the generating network, it helps drive the audio codec model to produce more realistic audio output in both the time and frequency domains, thereby improving the quality of subsequent audio codecs.

[0142] In one embodiment, a song audio encoding method is also provided, and this embodiment will illustrate the application of this method to the audio encoding end. For example... Figure 4 As shown, the method may include the following steps:

[0143] S401, acquire the raw audio signal of the song to be encoded.

[0144] S402, based on the original audio signal, obtains the audio encoding result of the song audio.

[0145] The audio encoding result is obtained by encoding the original audio signal of the song audio using the audio encoding method described in any one or more of the above embodiments.

[0146] The aforementioned song audio encoding method encodes the original audio signal of a song using the aforementioned audio encoding method. It leverages different sub-band encoders to extract features from each sub-band signal of the song audio in a differentiated and targeted manner, helping to improve the accuracy of feature extraction of low-frequency components in the original audio signal. Furthermore, by fully exploiting the temporal correlation information between high and low-frequency components through a shared quantizer and obtaining the audio encoding result based on this temporal correlation information, it can utilize cross-band temporal correlation for high-quality signal quantization and encoding, improving the encoding quality of the audio encoding result. Therefore, the method provided in this embodiment can significantly improve the reconstruction quality of low-frequency components and the overall audio during the encoding and decoding process. Practice has proven that encoding and decoding the audio signal of a song using this embodiment can achieve superior sound quality compared to traditional MP3 codecs at a relatively lower bitrate, while significantly improving the reconstruction quality of low-frequency components.

[0147] In one embodiment, an audio decoding method is also provided, which is illustrated in this embodiment by applying the method to a server. It is understood that the method can also be applied to a terminal, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, as... Figure 5 As shown, the method includes the following steps:

[0148] S501, obtain the audio encoding result.

[0149] The audio encoding result can be obtained by encoding the original audio signal using the audio encoding method described in any one or more of the above embodiments.

[0150] S502 obtains multiple sub-band encoding results based on the audio encoding results.

[0151] In this step, the audio encoding result provided by the audio encoding end can be analyzed to obtain multiple sub-band encoding results. In some embodiments, the audio encoding result can be obtained by concatenating multiple sub-band encoding results. Correspondingly, after obtaining the audio encoding result, the audio decoding end can segment the audio encoding result to obtain multiple sub-band encoding results.

[0152] S503, for each sub-band encoding result, decodes it using the sub-band decoder corresponding to each sub-band encoding result to obtain the third high-frequency sub-band signal and the third low-frequency sub-band signal output by each sub-band decoder.

[0153] In one example, the subband decoder can be a network used to decode the subband encoding results.

[0154] In this step, for each subband encoding result, a corresponding subband decoder can be determined. For example, the subband encoding result may contain prompt information indicating the frequency band range corresponding to the subband encoding result. Each frequency band range can correspond to a subband decoder. Therefore, the subband encoder can determine the subband decoder corresponding to the subband encoding result based on the prompt information. Alternatively, multiple subband encoding results can be concatenated sequentially according to their corresponding frequency band ranges. Then, the audio decoder can determine the corresponding subband decoder based on the order of the multiple subband encoding results obtained after segmentation.

[0155] After determining the subband decoders corresponding to each of the multiple subband encoding results, the subband encoding results can be input into the corresponding subband decoders for decoding, obtaining the high-frequency subband signal and the low-frequency subband signal output by the subband decoder. For ease of distinction, the high-frequency subband signal output by the subband decoder after decoding is called the third high-frequency subband signal, and the low-frequency subband signal output by the subband decoder after decoding is called the third low-frequency subband signal.

[0156] S504 obtains the decoded audio signal based on the third high-frequency sub-band signal and the third low-frequency sub-band signal.

[0157] In some embodiments, mirroring can be performed at the audio decoding end based on the processing at the audio encoding end. For example, such as Figure 2As shown, for the high-frequency components, the four third high-frequency sub-band signals decoded and output by the four high-frequency decoders can be synthesized and filtered. For the low-frequency components, after obtaining the one third low-frequency sub-band signal output by the low-frequency decoder, sub-band analysis and filtering can be performed to obtain two split signals, one containing two low-frequency sub-band signals and the other containing six low-frequency sub-band signals. Furthermore, the two high-frequency sub-band signals, the signal containing two low-frequency sub-band signals, and the signal containing six low-frequency sub-band signals obtained after merging can be synthesized to obtain the decoded audio signal.

[0158] In the aforementioned audio decoding method, an audio encoding result can be obtained. This audio encoding result can be obtained by encoding the original audio signal using any of the audio encoding methods described above. Furthermore, multiple sub-band encoding results can be obtained based on the audio encoding result. For each sub-band encoding result, it is decoded using its corresponding sub-band decoder to obtain the third high-frequency sub-band signal or the third low-frequency sub-band signal output by each sub-band decoder. Subsequently, the decoded audio signal can be obtained based on the third high-frequency sub-band signal and the third low-frequency sub-band signal. In this embodiment, by utilizing the sub-band decoders corresponding to each sub-band encoding result during audio decoding, different decoders can be used to decode different audio components, effectively improving the decoding accuracy of each sub-band signal. Therefore, the method provided in this embodiment can significantly improve the reconstruction quality of the low-frequency components and the overall audio signal.

[0159] In one embodiment, a song audio decoding method is also provided, which is illustrated in this embodiment by applying the method to a terminal. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0160] In this embodiment, as Figure 6 As shown, the method includes the following steps:

[0161] S601, in response to a request to obtain the song audio, obtains the audio encoding result of the song audio.

[0162] In practical applications, users can perform preset audio acquisition operations on the terminal to trigger the terminal to acquire the corresponding song audio signal. When the terminal detects the user-triggered audio acquisition operation, it can determine that it has received a request to acquire the song audio, and then obtain the audio encoding result of the song audio from the server. In some embodiments, the terminal can send a song download request carrying an audio identifier to the server. In response to the song download request, the server can acquire the original audio signal of the song, encode the original audio signal, obtain the audio encoding result of the song audio, and return it to the terminal.

[0163] S602 decodes the audio encoding result and obtains the decoded song audio signal corresponding to the song audio based on the decoding result.

[0164] The decoding result can be obtained using the audio decoding method described above.

[0165] After obtaining the audio encoding result, the terminal can decode the audio encoding result using one or more of the above-mentioned audio decoding methods, thereby obtaining the decoded song audio signal corresponding to the song audio based on the decoding result.

[0166] In the above-described song audio decoding method, when decoding the audio encoding results of the song audio, the sub-band decoders corresponding to each sub-band encoding result are used for decoding. This allows for differentiated decoding of different audio components in different song audios using different decoders, effectively improving the decoding accuracy of each sub-band signal, especially the reconstruction quality of low-frequency components in the song audio. Therefore, the method provided in this embodiment can significantly improve the decoding quality of song audio signals.

[0167] Furthermore, this embodiment provides an audio codec framework that utilizes subband prior information to address the problem of poor interpretability of frequency band modeling in the time-domain encoding and decoding process in related technologies. Specifically, subband decomposition provides a physically interpretable spectral perspective, independent codecs enable targeted and differentiated modeling of high- and low-frequency subband signals, and a shared quantizer ensures the preservation of global time-domain correlation. Thus, while improving model performance and efficiency, it also provides excellent interpretability for the model framework, enabling researchers to clearly understand how the model analyzes and reconstructs audio signals from both frequency and time dimensions, facilitating further optimization and adjustment of the model.

[0168] To enable those skilled in the art to better understand this application, the following is combined with Figure 7 The embodiments described herein are provided by way of example. However, it should be understood that the embodiments of this application are not limited thereto.

[0169] like Figure 7The diagram illustrates an application scenario framework for an audio codec model. Specifically, the audio codec model can include a temporal sub-band encoder (i.e., the encoding end in the audio codec model) and a temporal sub-band decoder (i.e., the decoding end in the audio codec model). The temporal sub-band encoder can be deployed on a music platform server, and the temporal sub-band decoder can be deployed on the user's terminal device. Within this framework, this embodiment can achieve superior sound quality compared to traditional MP3 codecs at a relatively lower bitrate, with a greater focus on the reconstruction quality of low-frequency components.

[0170] In practical applications, the first step is to construct an end-to-end molecular band generative adversarial network (GAN) training framework. Specifically, a molecular band encoding / decoding network (i.e., an audio encoding / decoding model) can be built, comprising a temporal molecular band encoder and a temporal molecular band decoder. The molecular band encoding / decoding network can be structured into a subband segmentation part and an encoder / decoder part. The subband segmentation part utilizes a PQMF-based subband analysis-synthesis filter bank to achieve temporal subband segmentation. In the encoder / decoder part, taking the encoder as an example, it employs pooling-convolution operations to extract multi-scale envelope features, thereby improving global modeling capabilities. It also uses depthwise separable convolutions and fewer downsampling layers to further reduce model complexity and parameter count. The decoder uses mirror branches for molecular band modeling. Furthermore, both the encoder and decoder use BiGRU instead of LSTM networks, achieving more efficient global feature extraction through bidirectional temporal modeling and a simplified gating mechanism.

[0171] Then, the discriminant network can be combined with the molecular band encoding / decoding network for adversarial training. The loss functions of the generator and discriminator networks are linear weights of the respective loss functions of the parallel branches. The loss weight ratio between the low-frequency and high-frequency branches can be 100:1. Both the generator and discriminator networks use a sub-band loss weighting scheme. Therefore, this model can further improve its low-frequency feature modeling capability and low-frequency signal reconstruction quality by increasing the complexity of the low-frequency branches or the low-frequency sub-band loss weights, while also improving the stability of GAN training.

[0172] The generator loss consists of Mel spectrum loss, quantization loss and feature loss, while the discriminator loss consists of multi-scale, multi-resolution and multi-period waveform discrimination terms. Gradient pruning is used during training to ensure training stability.

[0173] Furthermore, when the music platform server encodes the collected original high-fidelity audio, it can use a time-domain sub-band encoder deployed on the music platform server to encode the original audio signal, compressing the original audio signal into a low-bitrate stream. When a user selects a song through their terminal, the low-bitrate stream can be transmitted to the user's terminal via the network. After receiving the low-bitrate stream, the terminal can use a pre-deployed time-domain sub-band decoder to decode the stream, obtaining the decoded audio signal corresponding to the song's audio, and then play it.

[0174] In practice, it has been found that, compared with the original encoding and decoding scheme, the audio encoding and decoding method provided in this embodiment can achieve overall objective and subjective performance improvements, including low-frequency and high-frequency components. For example, it improves objective indicators such as scale-invariant signal-to-noise ratio and low-frequency logarithmic spectral distance, effectively reduces audio signal reconstruction artifacts, and significantly improves the overall reconstruction quality of audio signals.

[0175] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0176] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores audio data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements an audio encoding method.

[0177] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an audio decoding method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0178] Those skilled in the art will understand that Figure 8 and Figure 9The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0179] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0180] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0181] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0182] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0183] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0184] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0185] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio encoding method, characterized in that, The method includes: Obtain the raw audio signal to be encoded; Based on the subband segmentation result of the original audio signal in the time domain, the first high-frequency subband signal and the first low-frequency subband signal corresponding to the original audio signal are obtained. The first high-frequency sub-band signal and the first low-frequency sub-band signal are respectively input to their respective sub-band encoders to obtain the sub-band encoding results output by each sub-band encoder; Determine the temporal correlation information of multiple sub-band coding results, and obtain the audio coding result of the original audio signal based on the temporal correlation information and the multiple sub-band coding results.

2. The method according to claim 1, characterized in that, The step of obtaining the first high-frequency sub-band signal and the first low-frequency sub-band signal corresponding to the original audio signal based on the sub-band segmentation result in the time domain includes: The original audio signal is input into a subband analysis filter. Based on the subband segmentation result of the original audio signal in the time domain by the subband analysis filter, a second high-frequency subband signal and multiple second low-frequency subband signals are obtained. Based on the subband synthesis results of the plurality of second low-frequency subband signals by the subband synthesis filter, the first low-frequency subband signal corresponding to the original audio signal is obtained; and based on the subband segmentation results of the second high-frequency subband signal by the subband analysis filter, the plurality of first high-frequency subband signals corresponding to the original audio signal are obtained.

3. The method according to claim 2, characterized in that, The subband synthesis result of the plurality of second low-frequency subband signals based on the subband synthesis filter is used to obtain the first low-frequency subband signal corresponding to the original audio signal, including: Determine the total number of subbands corresponding to the second high-frequency subband signal and the multiple second low-frequency subband signals; The multiple second low-frequency sub-band signals are upsampled according to the total number of sub-bands to obtain the upsampling result; The upsampling result is synthesized and filtered using a subband synthesis filter to obtain the first low-frequency subband signal.

4. The method according to claim 2, characterized in that, The subband analysis filter and the subband synthesis filter are filters in a pseudo-orthogonal mirror filter bank.

5. The method according to claim 1, characterized in that, The sub-band encoder obtains the sub-band encoding result through the following steps: The input subband signal corresponding to the subband encoder is pooled using pooling kernels of multiple sizes respectively. Based on the pooling results of the pooling kernels of multiple sizes, multi-scale envelope features are obtained. The input subband signal includes the first high-frequency subband signal or the first low-frequency subband signal input to the subband encoder. The subband encoding result is obtained based on the multi-scale envelope features.

6. The method according to claim 5, characterized in that, The step of obtaining the sub-band coding result based on the multi-scale envelope features includes: The multi-scale envelope features are input into the residual convolution module in the sub-band encoder. The residual convolution module performs depthwise separable convolution processing on the multi-scale envelope features. The sub-band encoding result is obtained based on the depthwise separable convolution processing result.

7. The method according to claim 6, characterized in that, The step of obtaining the sub-band encoding result based on the depth-separable convolution processing result includes: The depth-separable convolution processing result is input into the bidirectional gated recurrent unit network in the sub-band encoder, and the sub-band encoding result is obtained based on the output characteristics of the bidirectional gated recurrent unit network.

8. The method according to any one of claims 1 to 7, characterized in that, The sub-band encoder is a sub-band encoder in an audio codec model obtained based on adversarial training. The audio codec model also includes a corresponding sub-band decoder for obtaining the sub-band decoding results. The audio codec model is trained based on the following steps: The sub-band encoder in the audio codec model to be trained obtains the sample sub-band encoding results of each first sample sub-band signal of the sample audio signal, and the sub-band decoder in the audio codec model to be trained obtains the sample sub-band decoding results of each first sample sub-band signal of the sample audio signal. For each of the first sample subband signals, a first sub-loss value of the first sample subband signal is determined based on the sample subband decoding result; The generated network loss value is determined based on each of the first sub-loss values; The audio codec model to be trained is adjusted according to the loss value of the generated network, and the trained audio codec model is obtained when the adversarial training termination condition is met.

9. The method according to claim 8, characterized in that, Each first sample sub-band signal includes a low-frequency sample sub-band signal and a high-frequency sample sub-band signal; The step of determining the generated network loss value based on each of the first sub-loss values ​​includes: Determine a first loss weight for the low-frequency subband signal of the sample and a second loss weight for the high-frequency subband signal of the sample; the first loss weight is greater than the second loss weight. The generation network loss value is determined based on the first loss weight, the second loss weight, the first sub-loss value of the sample low-frequency subband signal, and the first sub-loss value of the sample high-frequency subband signal.

10. The method according to claim 8, characterized in that, After adjusting the audio codec model to be trained based on the generated network loss value, the method further includes: Multiple second sample sub-band signals are acquired; the multiple second sample sub-band signals are acquired based on the decoding results of the multiple sample sub-bands. For each of the second sample sub-band signals, the discrimination network determines the second sub-loss value of the second sample sub-band signal based on the discrimination result corresponding to the second sample sub-band signal; The discrimination network loss value is determined based on each of the second sub-loss values, and the discrimination network is adjusted based on the discrimination network loss value.

11. A song audio encoding method, characterized in that, The method includes: Obtain the raw audio signal of the song to be encoded; Based on the original audio signal, the audio encoding result of the song audio is obtained; the audio encoding result is obtained by encoding the original audio signal using the audio encoding method as described in any one of claims 1 to 10.

12. An audio decoding method, characterized in that, The method includes: Obtain the audio encoding result; the audio encoding result is obtained by encoding the original audio signal using the audio encoding method as described in any one of claims 1 to 10; Multiple sub-band encoding results are obtained based on the audio encoding results; For each subband encoding result, the subband decoder corresponding to each subband encoding result is used for decoding to obtain the third high-frequency subband signal and the third low-frequency subband signal output by each subband decoder; The decoded audio signal is obtained based on the third high-frequency sub-band signal and the third low-frequency sub-band signal.

13. A method for decoding song audio, characterized in that, The method includes: In response to a request to obtain the song audio, the audio encoding result of the song audio is obtained; The audio encoding result is decoded, and the decoded song audio signal corresponding to the song audio is obtained based on the decoding result; the decoding result is obtained by the audio decoding method as described in claim 12.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11 or the steps of the method according to any one of claims 12 to 13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11 or the steps of the method according to any one of claims 12 to 13.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11 or the steps of the method according to any one of claims 12 to 13.

Images