Audio encoding method, audio decoding method, apparatus, and computer program

Abstract

This application provides an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, the audio decoding method comprising the steps of: obtaining an audio bitstream package, wherein the audio bitstream package includes an audio bitstream, the audio bitstream is obtained by performing audio encoding on an audio signal using a target encoding mode and a target bitrate mode, the target encoding mode is obtained from a plurality of candidate encoding modes, and the target bitrate mode is obtained from a plurality of candidate bitrate modes; in response to a decoding request for the audio bitstream package, obtain a target encoding mode and a target bitrate mode from a frame header included in the audio bitstream package; perform signal decoding on the audio bitstream using the target encoding mode and the target bitrate mode to obtain an estimated encoding feature corresponding to the audio bitstream; and reconstruct the estimated encoding feature using the target encoding mode to obtain a reconstructed audio signal corresponding to the audio bitstream.

Description

TECHNICAL FIELD

[0001] (Cross-reference to Related Applications) The embodiments of the present application are filed based on a Chinese patent application with application number 202311614893.4 and filing date November 29, 2023, claiming priority to said Chinese patent application, the entire content of which is incorporated into the embodiments of the present application by reference.

[0002] The present application relates to artificial intelligence technology, and in particular to an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

BACKGROUND ART

[0003] Audio encoding and decoding technology is one of the important applications in the field of artificial intelligence, and is a core technology in communication services including remote audio and video calls. In simple terms, audio encoding technology is to transmit as much audio information as possible using a relatively small network bandwidth. From the perspective of Shannon's information theory, audio encoding is a type of source coding, which aims to compress the amount of data to be transmitted on the encoding side as much as possible, remove redundancy in the information, and at the same time enable lossless (or nearly lossless) reconstruction on the decoding side.

[0004] In related technologies, the audio quality generated by decoding on the decoding side is relatively uniform, which cannot meet the needs of users.

SUMMARY OF THE INVENTION

MEANS FOR SOLVING THE PROBLEMS

[0005] The embodiments of the present application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can output reconstructed audio signals of different quality levels while ensuring the efficiency of audio decoding.

[0006] The technical solutions of the embodiments of the present application are implemented as follows.

[0007] The embodiments of the present application provide an audio encoding method applied to an electronic device, and the method comprises: in response to an encoding request for an audio signal, acquiring a target encoding mode for the audio signal from a plurality of encoding modes, and acquiring a target bit rate mode for the audio signal from a plurality of bit rate modes; extracting an encoding feature of the audio signal from the audio signal according to the target encoding mode; performing signal encoding on the encoding feature according to the target bit rate mode to obtain an audio bitstream of the audio signal; determining a frame header based on the target encoding mode and the target bit rate mode; generating an audio bitstream package of the audio signal based on the audio bitstream and the frame header.

[0008] The embodiments of the present application provide an audio decoding method applied to an electronic device, and the method comprises: acquiring an audio bitstream package, wherein the audio bitstream package comprises an audio bitstream, the audio bitstream is obtained by performing audio encoding on an audio signal according to a target encoding mode and a target bit rate mode, the target encoding mode is acquired from a plurality of candidate encoding modes, and the target bit rate mode is acquired from a plurality of candidate bit rate modes; In response to a decoding request for the audio bitstream package, the steps include obtaining the target encoding mode and the target bitrate mode from the frame header included in the audio bitstream package, The steps include: performing signal decoding on the audio bitstream using the target coding mode and the target bitrate mode to obtain an estimated coding feature value corresponding to the audio bitstream; The process includes the step of reconstructing the estimated encoded features using the target encoding mode to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0009] The embodiments of this application provide an audio coding device, and the audio coding device is A second acquisition module is configured to respond to an encoding request for an audio signal by acquiring a target encoding mode for the audio signal from a plurality of encoding modes and a target bitrate mode for the audio signal from a plurality of bitrate modes. An extraction module configured to extract the encoded features of the audio signal from the audio signal using the target coding mode, A signal coding module configured to perform signal coding on the coding features in the target bitrate mode to obtain an audio bitstream of the audio signal, A configuration module configured to determine the frame header based on the target encoding mode and the target bitrate mode, The system includes a generation module configured to generate an audio bitstream package of the audio signal based on the audio bitstream and the frame header.

[0010] Embodiments of this application provide an audio decoding device, the audio decoding device is A step of obtaining an audio bitstream package, wherein the audio bitstream package includes an audio bitstream, the audio bitstream is obtained by performing audio encoding on an audio signal using a target encoding mode and a target bitrate mode, and the target bitrate mode is obtained from a plurality of candidate bitrate modes. A first acquisition module is configured to perform the steps of: obtaining the target encoding mode and the target bitrate mode from the frame header included in the audio bitstream package in response to a decoding request for the audio bitstream package; A signal decoding module configured to perform signal decoding on the audio bitstream using the target coding mode and the target bitrate mode, and to obtain an estimated coding feature value corresponding to the audio bitstream, The system includes a reconstruction module configured to reconstruct the estimated encoded features using the target encoding mode to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0011] The embodiments of this application provide an electronic device, said electronic device, Memory for storing computer-executable instructions, When executing computer-executable instructions stored in the memory, the system includes a processor for implementing the audio encoding method or audio decoding method provided in the embodiment of this application.

[0012] The embodiments of this application provide a computer-readable storage medium that stores computer-executable instructions for implementing an audio encoding method or an audio decoding method provided in the embodiments of this application, when executed by a processor.

[0013] The embodiments of this application provide a computer program product that includes computer-executable instructions for implementing an audio encoding method or an audio decoding method provided in the embodiments of this application, when executed by a processor.

[0014] The embodiments of this application provide the following beneficial effects.

[0015] By decoding the audio bitstream using different encoding modes and bitrate modes, encoding feature estimates of different precisions are obtained. Subsequently, by reconstructing using these different precision encoding feature estimates, reconstructed audio signals of different quality levels are obtained, improving the diversification of the quality of the reconstructed audio signals and meeting the actual application needs of users. [Brief explanation of the drawing]

[0016] [Figure 1] This is a schematic diagram comparing spectra at different bit rates according to an embodiment of this application. [Figure 2] This is a schematic diagram of the architecture of an audio coding and decoding system according to an embodiment of this application. [Figure 3A] This is a schematic diagram of the first structure of an electronic device according to an embodiment of this application. [Figure 3B] This is a schematic diagram of the second structure of an electronic device according to an embodiment of this application. [Figure 4A] This is a schematic diagram of the first flowchart of the audio encoding method according to an embodiment of this application. [Figure 4B] This is a schematic diagram of a second flowchart of an audio encoding method according to an embodiment of this application. [Figure 4C] This is a schematic diagram of the third flowchart of the audio encoding method according to an embodiment of this application. [Figure 4D] This is a schematic diagram of the fourth flowchart of the audio encoding method according to an embodiment of this application. [Figure 4E] This is a schematic diagram of the fifth flowchart of the audio encoding method according to an embodiment of this application. [Figure 5A] This is a schematic diagram of the first flowchart of the audio decoding method according to an embodiment of this application. [Figure 5B] This is a schematic diagram of a second flowchart of an audio decoding method according to an embodiment of this application. [Figure 5C] This is a schematic diagram of the third flowchart of the audio decoding method according to an embodiment of this application. [Figure 6A] This is a schematic diagram of a channel in which the grouped convolution described in the embodiment of this application is not employed. [Figure 6B] This is a schematic diagram of a channel employing grouped convolution according to the embodiment of this application. [Figure 6C] This is a schematic diagram of an audio communication link according to an embodiment of the present application. [Figure 7] This is a schematic flowchart of a multimode multirate speech coding method according to an embodiment of this application. [Figure 8] This is a schematic diagram of a filter bank according to an embodiment of the present application. [Figure 9A] This is a schematic diagram of a typical convolutional network according to an embodiment of this application. [Figure 9B] This is a schematic diagram of an extended convolutional network according to an embodiment of this application. [Figure 10] This is a schematic diagram of frequency band extension according to the embodiment of this application. [Figure 11] This is a schematic diagram of the third neural network according to the embodiment of this application. [Figure 12A] This is a schematic diagram of the residual block structure used in the coding block according to the embodiment of this application. [Figure 12B] This is a schematic diagram of the structure of the residual unit according to the embodiment of this application. [Figure 13A] This is a schematic diagram of the structure of a bitstream package in a broadband coding mode according to an embodiment of this application. [Figure 13B] This is a schematic diagram of the structure of a bitstream package in an ultra-wideband coding mode that does not include residual and flatness side information according to the embodiment of this application. [Figure 13C]This is a schematic diagram of the structure of a bitstream package in an ultra-wideband coding mode that includes residuals but does not include flatness side information, according to an embodiment of this application. [Figure 13D] This is a schematic diagram of the structure of a bitstream package in an ultra-wideband coding mode that does not include residuals but includes flatness side information, according to an embodiment of this application. [Figure 13E] This is a schematic diagram of the structure of a bitstream package in an ultra-wideband coding mode including residual and flatness side information according to the embodiment of this application. [Figure 14] This is a schematic diagram of the first neural network according to the embodiment of this application. [Figure 15A] This is a schematic diagram of the structure of a bitstream package in a broadband stereo coding mode according to an embodiment of this application. [Figure 15B] This is a schematic diagram of the structure of a bitstream package in an ultra-wideband stereo coding mode that does not include residual and flatness side information according to the embodiment of this application. [Figure 15C] This is a schematic diagram of the structure of a bitstream package in an ultra-wideband stereo coding mode that includes residuals but does not include flatness side information, according to an embodiment of this application. [Figure 15D] This is a schematic diagram of the structure of a bitstream package in an ultra-wideband stereo coding mode that does not include residuals but includes flatness side information, according to an embodiment of this application. [Figure 15E] This is a schematic diagram of the structure of a bitstream package in an ultra-wideband stereo coding mode, including residual and flatness side information, according to an embodiment of this application. [Modes for carrying out the invention]

[0017] To further clarify the purpose, technical proposal and merits of this application, the application will be described in more detail below with reference to the drawings, and the embodiments described therein should not be considered as limitations to the embodiments of this application. All other embodiments that can be obtained by a person skilled in the art without creative effort are all within the scope of protection of this application.

[0018] The terms “First / Second” as used below are merely used to distinguish similar objects and do not represent a particular order of objects. Understandably, “First / Second” may be interchanged in any order or sequence so that, where possible, the embodiments of this application described herein may be carried out in an order other than those illustrated or described herein.

[0019] The “Several Examples” described below represent a subset of all possible examples, but it is understandable that “Several Examples” may be the same subset or different subsets of all possible examples, and can be combined with each other as long as they do not contradict each other.

[0020] In the embodiments of this application, the terms “module” or “unit” refer to a computer program or part of a computer program having a predetermined function, which works together with other relevant parts to achieve a predetermined goal, and which can be realized in whole or in part by software, hardware (such as processing circuits and memory), or a combination thereof. Similarly, one processor (or more processors and memory) may be used to realize one or more modules or units. Furthermore, each module or unit may be part of an overall module or unit that includes the function of the module or unit.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as those generally understood by those skilled in the art. Terms used herein are not intended to limit this application, but merely to describe the embodiments of this application.

[0022] Before describing the embodiments of this application in further detail, the nouns and terms relating to the embodiments of this application will be explained below, and the nouns and terms relating to the embodiments of this application will be interpreted as follows.

[0023] 1) Neural Network (NN): An algorithmic mathematical model that mimics the behavior and characteristics of animal neural networks and performs distributed parallel information processing. Such networks achieve their information processing objectives by adjusting the interconnection relationships between numerous nodes within the system, depending on the complexity of the system.

[0024] 2) Deep Learning (DL): This represents a new research direction in the field of Machine Learning (ML). Deep learning involves learning the intrinsic rules and representational hierarchies of sample data, and the information acquired through these learning processes is extremely useful in interpreting data such as text, images, and audio. Its ultimate goal is for machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images, and audio.

[0025] 3) Quantization: This refers to the process of approximating a continuous value (or a large number of discrete values) of a signal to a finite number (or a small number) of discrete values. Here, quantization includes vector quantization (VQ) and scalar quantization.

[0026] Vector quantization is an effective lossy compression technique based on Shannon's rate-distortion theory. The basic principle of vector quantization is to replace the input vector with the index of the codeword in the codebook that best matches it (also called the quantized value) for transmission and storage, requiring only simple table lookup operations during decoding. For example, several scalar data can be combined to form a vector space, which can then be divided into several smaller regions. During quantization, the input vectors corresponding to these smaller regions are replaced with their corresponding indices.

[0027] Scalar quantization is quantization applied to scalars, i.e., one-dimensional vector quantization. The dynamic range is divided into several small intervals, and each small interval has one representative value (i.e., an index). If the input signal falls within a certain interval, the input signal is quantized to that representative value.

[0028] 4) Entropy coding: In the coding process, this is a reversible coding method that follows the principle of entropy and does not lose any information. It is also an important module in reversible coding and is located at the final stage of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exp-Golomb coding, and arithmetic coding.

[0029] 5) Quadrature Mirror Filter Bank (QMF): This is a filter pair that includes analysis and synthesis. Here, the QMF analysis filter is used to decompose the subband signals to reduce the signal bandwidth and allow each subband signal to pass smoothly through its respective channel for processing. The QMF synthesis filter is used to synthesize each subband signal recovered on the decoding side, reconstructing the original audio signal through methods such as zero-padding and bandpass filtering.

[0030] 6) Bitstream Packaging: This refers to packaging an encoded bitstream into a specific format to facilitate storage or network transmission. In the packaging process, the already encoded and compressed bitstream is stored in a file according to a certain format, forming a container. This container may contain not only the bitstream but also metadata such as encoding type, bitrate, and frame rate. The package format (also called a container format) can be considered the outer shell of the bitstream (audio bitstream, video bitstream, etc.), and the bitstream is stored within the file of this package format. The choice of package format depends on the specific application scenario and requirements, and different package formats support different encoding formats. For example, MP4 and Flash video format (FLV: Adobe Flash Video) are common package formats.

[0031] 7) Broadband: This refers to the resolution or sampling rate of an audio signal. According to the definitions of standardization bodies such as the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) and the 3rd Generation Partnership Project (3GPP), a broadband sampling rate is 16,000 Hz, and the effective bandwidth is up to 8,000 Hz. The transmission speed of a broadband signal is typically 1.54 Mbps (megabits per second) or higher.

[0032] 8) Ultra-wideband: This refers to the resolution or sampling rate of an audio signal. According to the definitions of standardization bodies such as the ITU-T and 3GPP, the sampling rate of ultra-wideband is 32,000 Hz, and the effective bandwidth is up to 16,000 Hz. Ultra-wideband is a type of wireless communication technology, and generally speaking, it refers to an audio signal whose bandwidth is at least 500 MHz, or whose ratio of the audio signal bandwidth to the center frequency exceeds 20%.

[0033] Before specifically describing the audio coding and decoding methods provided in the embodiments of this application, we will first introduce the QMF filter bank, the extended convolutional network, and the frequency bandwidth extension.

[0034] A QMF filter bank is a pair of filters that include analysis and synthesis. For a QMF analysis filter, an input signal with sampling rate Fs can be decomposed into two signals with sampling rates Fs / 2, which are represented as a QMF low-pass signal and a QMF high-pass signal, respectively. Figure 8 shows the spectral responses of the low-pass portion H_Low(z) and the high-pass portion H_High(z) of the QMF filter. Based on the relevant theoretical knowledge of the QMF analysis filter bank, the correlation between the coefficients of the low-pass and high-pass filters can be easily described as shown in equation (1).

[0035]

number

number

number

[0036] Similarly, based on QMF-related theory, the QMF synthesis filter bank can be described based on H_Low(z) and H_High(z) of the QMF analysis filter bank, as shown in equation (2).

number

[0037]

number

number

number

[0038] The low-pass and high-pass signals reconstructed on the decoding side are combined by the QMF composite filter bank, thereby reconstructing a reconstructed signal (i.e., a composite signal) with a sampling rate Fs corresponding to the input signal.

[0039] Referring to Figures 9A and 9B, Figure 9A is a schematic diagram of a normal convolutional network (e.g., causal convolution) according to an embodiment of this application, and Figure 9B is a schematic diagram of an extended convolutional network according to an embodiment of this application. Compared to a normal convolutional network, extended convolution can expand the receptive field while maintaining the size of the feature map, and can also avoid errors caused by upsampling and downsampling. The convolutional kernel size shown in Figures 9A and 9B is 3x3 in both cases, but the receptive field 901 of the normal convolution shown in Figure 9A is only 3, while the receptive field 902 of the extended convolution shown in Figure 9B reaches 5. That is, for a convolutional kernel of size 3x3, the receptive field of the normal convolution shown in Figure 9A is 3, and the dilation rate (number of intervals between points in the convolutional kernel) is 1. The receptive field of the extended convolution shown in Figure 9B is 5, and the dilation rate is 2.

[0040] The convolutional kernel can also move on a plane similar to that shown in Figure 9A or Figure 9B, where the concept of stride rate (stride) comes into play. For example, for every one-square movement of the convolutional kernel, the corresponding stride rate is 1.

[0041] There is also the concept of the number of convolution channels. This refers to the number of parameters corresponding to the convolution kernels used in the convolution analysis. Theoretically, the more channels there are, the more comprehensive and accurate the analysis of the signal becomes. However, the more channels there are, the higher the complexity becomes. For example, a 1x320 tensor can be subjected to a 24-channel convolution operation, and the output is a 24x320 tensor.

[0042] It should be noted that the size of the extended convolutional kernel (for example, for an audio signal, the convolutional kernel size can be set to 1 × 3), the extension ratio, the stride rate, and the number of channels can be defined according to the needs of the actual application. The embodiments of this application are not specifically limited thereto.

[0043] Figure 10 shows a schematic diagram of frequency band extension (or frequency band copying). First, a broadband signal is reconstructed, then the broadband signal is copied to an ultra-broadband signal, and finally it is shaped based on an ultra-broadband envelope. The frequency domain realization solution shown in Figure 10 specifically includes the following 1) to 3): 1) Achieve core layer coding at a low sampling rate. 2) Select the low-frequency portion of the spectrum and copy it to a high-frequency portion. 3) Apply gain to the copied high-frequency spectrum based on pre-recorded boundary information (describes the energy correlation between high and low frequencies, etc.). This can achieve the effect of doubling the sampling rate with a bitrate of only 1-2 kbps.

[0044] Speech coding technology aims to transmit as much audio information as possible with relatively little network bandwidth resources. Speech codecs can achieve compression ratios of 10x or more; that is, 10MB of audio data, after being compressed by the encoder, requires only 1MB for transmission, significantly reducing the bandwidth resources consumed to transmit information. For example, for a wideband audio signal with a sampling rate of 16000Hz, using a 16-bit sampling depth (the fineness of audio intensity recorded during sampling), the uncompressed version has a bitrate (amount of data transmitted per unit time) of 256kbps. Using speech coding technology, even with lossy coding, the quality of the reconstructed audio signal can approach that of the uncompressed version within a bitrate range of 10-20kbps, and ultimately be perceived as indistinguishable by sound. For services requiring higher sampling rates, such as ultra-wideband audio of 32000Hz, the bitrate range needs to reach at least 30kbps.

[0045] In communication systems, standard audio codec protocols are introduced within the industry to ensure smooth communication. Examples include standards from international and national standardization organizations such as ITU-T, 3GPP, IETF, AVS, and CCSA, as well as standards such as G.711, G.722, AMR series, EVS, and OPUS. Figure 1 shows a schematic diagram comparing spectra at different bitrates to illustrate the relationship between compression bitrate and quality. Curve 101 is the spectral curve of the original audio, i.e., the uncompressed signal. Curve 102 is the spectral curve of the OPUS encoder at a bitrate of 20kbps. Curve 103 is the spectral curve of OPUS encoding at a bitrate of 6kbps. As can be seen from Figure 1, as the encoding bitrate increases, the compressed signal becomes closer to the original signal.

[0046] The principle of speech coding is roughly as follows: Speech coding can be performed directly on a sample-by-sample basis on a speech waveform sample, or, based on the principles of human speech, relevant low-dimensional features are extracted, the features are coded on the coding side, and the speech signal is reconstructed on the decoding side based on these parameters.

[0047] The encoding principles described above all originate from audio signal modeling, i.e., compression methods based on signal processing, and cannot guarantee the encoding quality of audio. To improve encoding efficiency while guaranteeing audio quality, the embodiments of this application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The following describes exemplary applications of the electronic device provided in the embodiments of this application. The electronic device provided in the embodiments of this application may be implemented as a terminal device, as a server, or in cooperation with a terminal device and a server. The following description will use the implementation of the electronic device as a terminal device as an example.

[0048] For illustrative purposes, referring to Figure 2, which is a schematic diagram of the architecture of an audio coding / decoding system 10 according to an embodiment of the present application, the audio coding / decoding system 10 includes a server 200, a network 300, terminal equipment 400 (i.e., coding side), and terminal equipment 500 (i.e., decoding side). The network 300 may be a local area network, a wide area network, or a combination of the two.

[0049] In some embodiments, client 410 runs on terminal device 400. Client 410 can be various types of clients, such as an instant messaging client, a web conferencing client, a live streaming client, or a browser. In response to an audio capture command triggered by a sender (e.g., a web conference initiator, host, or voice call initiator), client 410 invokes the microphone built into terminal device 400 to capture an audio signal, performs audio encoding on the captured audio signal, and obtains an audio bitstream.

[0050] For example, client 410 invokes the audio coding method provided in the embodiment of this application to code the collected audio signal. That is, it obtains a target coding mode for the audio signal from a plurality of coding modes, obtains a target bitrate mode for the audio signal from a plurality of bitrate modes, extracts the coding features of the audio signal from the audio signal using the target coding mode, performs signal coding on the coding features of the audio signal using the target bitrate mode to obtain an audio bitstream of the audio signal, determines a frame header based on the target coding mode and target bitrate mode, and generates an audio bitstream package of the audio signal based on the audio bitstream and frame header.

[0051] The client 410 can send an audio bitstream package to the server 200 via the network 300, which then sends the audio bitstream package to the terminal device 500 associated with the recipient (e.g., a participant or viewer in a web conference, or a recipient in an audio call).

[0052] A client 510 (such as an instant messaging client, web conferencing client, live streaming client, or browser) operating on terminal device 500 receives an audio bitstream package sent from server 200, performs audio decoding on the audio bitstream package to obtain a reconstructed audio signal, thereby enabling audio communication.

[0053] For example, client 510 invokes the audio decoding method provided in the embodiment of this application to decode the received audio bitstream package, that is, to obtain a target coding mode and a target bitrate mode from the frame header contained in the audio bitstream package, where the audio bitstream contained in the audio bitstream package is obtained by performing audio coding on the audio signal using the target coding mode and the target bitrate mode, the target coding mode is obtained from a plurality of coding modes, and the target bitrate mode is obtained from a plurality of bitrate modes, and the client 510 performs signal decoding on the audio bitstream using the target coding mode and the target bitrate mode to obtain an estimated coding feature value corresponding to the audio bitstream, and then reconstructs the estimated coding feature value corresponding to the audio bitstream using the target coding mode to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0054] For example, the server 200 shown in Figure 2 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms. The terminal devices 400 and 500 shown in Figure 2 may be, but are not limited to, smartphones, tablet computers, notebook computers, desktop computers, smart speakers, smartwatches, or in-vehicle terminals. The terminal devices (e.g., terminal devices 400 and 500) and the server 200 can be connected directly or indirectly by wired or wireless communication, but are not limited to the embodiments of this application.

[0055] In some embodiments, a terminal device or server 200 can implement an audio encoding method or audio decoding method provided by the embodiments of this application by executing a computer program. For example, the computer program may be a native program or software module in an operating system, a native application (APP), i.e., a program that needs to be installed on the operating system to run, such as a live streaming APP, a web conferencing APP, or an instant messaging APP, a mini-program, i.e., a program that can be run simply by downloading it to a browser environment, or even a mini-program that can be embedded in any APP. In short, the above computer program may be any form of application program, module, or plug-in.

[0056] Referring to Figure 3A, which is a schematic diagram of the structure of an electronic device 500 according to an embodiment of the present application. Using the example that the electronic device 500 is a terminal device, the electronic device 500 shown in Figure 3A includes at least one processor 520, memory 550, at least one network interface 530, and a user interface 540. Each component within the electronic device 500 is coupled to one another via a bus system 550. It can be understood that the bus system 550 is used to enable connection and communication between these components. In addition to the data bus, the bus system 550 also includes a power bus, a control bus, and a status signal bus. However, for clarity, each bus is labeled as the bus system 550 in Figure 3A.

[0057] The processor 520 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Here, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0058] Memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 550 optionally includes one or more storage devices located physically separate from the processor 520.

[0059] The memory 550 includes volatile memory or non-volatile memory, and may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random-access memory (RAM). The memory 550 described in the embodiments of this application is intended to include any suitable type of memory.

[0060] In some embodiments, memory 550 can store data to support various operations, and examples of this data include programs, modules, and data structures, or subsets or supersets thereof, which are illustrated below.

[0061] Operating system 551 is configured to implement various basic services and handle hardware-based tasks, including system programs such as a framework layer, core library layer, and driver layer, for handling various basic system services and executing hardware-related tasks.

[0062] The network communication module 552 is configured to reach other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0063] In some embodiments, the audio coding device provided by the embodiments of this application can be implemented by software, and Figure 3A shows the audio coding device 555 stored in memory 550, where the audio coding device 555 may be software in the form of a program or plug-in, and includes a second acquisition module 5551, an extraction module 5552, a signal coding module 5553, a configuration module 5554, and a generation module 5555, where the second acquisition module 5551, the extraction module 5552, the signal coding module 5553, the configuration module 5554, and the generation module 5555 are used to implement the audio coding function, and since these modules are logical, any combination or further division may be made depending on the function to be implemented.

[0064] Referring to Figure 3B, which is a schematic diagram of the structure of an electronic device 600 according to an embodiment of the present application. As an example, the electronic device 600 is described as a terminal device, and the electronic device 600 shown in Figure 3B includes at least one processor 620, memory 650, at least one network interface 630, and a user interface 640. Each component within the electronic device 600 is coupled to one another via a bus system 650. The memory 650 includes an operating system 651 and a network communication module 652. It should be noted that the function of the structure in Figure 3B is similar to that of the structure in Figure 3A. The audio coding device provided in an embodiment of the present application can be implemented by software, and Figure 3B shows an audio decoding device 655 stored in memory 650, where the audio decoding device 655 may be software in the form of a program or plug-in, and includes a first acquisition module 6551, a signal decoding module 6552, and a reconstruction module 6553. The first acquisition module 6551, the signal decoding module 6552, and the reconstruction module 6553 are used to implement the audio decoding function, and since these modules are logical, they may be combined in any way or further divided depending on the function to be implemented.

[0065] As described above, the audio coding method provided in the embodiment of this application can be implemented by various types of electronic devices. Referring to Figure 4A, Figure 4A is a schematic flowchart of the audio coding method according to the embodiment of this application, and the audio coding function is realized by the audio coding method. The following explanation will be given with reference to steps 101 to 105 shown in Figure 4A.

[0066] Before describing the following steps, the audio coding method provided in the embodiments of this application will be described. The audio coding method provided in the embodiments of this application includes two coding methods: a signal coding method in signal processing techniques and feature coding (i.e., feature extraction) in artificial intelligence techniques. For details on feature coding in artificial intelligence techniques, see step 102 below. For details on the signal coding method in signal processing techniques, see step 103 below.

[0067] In step 101, in response to an encoding request for an audio signal, the target encoding mode for the audio signal is obtained from multiple encoding modes, and the target bitrate mode for the audio signal is obtained from multiple bitrate modes.

[0068] Here, the coding request is used to indicate that audio coding will be performed on an audio signal. The coding request includes coding configuration information, such as the sampling rate of the audio signal and the type of audio signal (e.g., broadband signal, ultra-broadband signal). Here, multiple coding modes include broadband coding modes and ultra-broadband coding modes. The broadband coding mode is used to indicate that broadband coding will be performed on an audio signal, meaning that the audio signal is a broadband signal, and because the sampling rate of broadband signals is lower than that of ultra-broadband signals, subband decomposition of the audio signal is not required. The ultra-broadband coding mode is used to indicate that ultra-broadband coding will be performed on an audio signal, meaning that the audio signal is an ultra-broadband signal, and because the sampling rate of ultra-broadband signals is higher than that of broadband signals, subband decomposition of the audio signal can be performed, and then coding can be performed on the subband signals obtained by the decomposition. The bitrate mode is used to indicate that signal coding or decoding will be performed using a specified codebook and corresponding bitrate, and different bitrate modes correspond to different codebooks and bitrates. In the configuration phase, different coding modes and bitrate modes can be configured for different audio signals.

[0069] As an example of acquiring an audio signal, the encoding device responds to an audio collection command triggered by the sender (e.g., the initiator or host of a web conference, or the initiator of a voice call) by calling the microphone built into the encoding device's terminal equipment to collect the audio signal (also called the input signal).

[0070] As an example of obtaining an audio signal encoding request, the encoding side can select a target encoding mode for the audio signal from multiple encoding modes in response to the sender's selection operation for multiple encoding models, and can also select a target bitrate mode for the audio signal from multiple bitrate modes in response to the sender's selection operation for multiple bitrate modes. Based on the target encoding mode and target bitrate mode, it generates an encoding request for the audio signal, thereby enabling the setting of different encoding modes and bitrate modes for different audio signals according to the user's needs.

[0071] Another example of obtaining an audio signal encoding request is that the encoding side can pre-store a configuration table. Here, the configuration table stores the correspondence between different configuration information and different encoding modes, and the correspondence between different configuration information and different bitrate modes. After obtaining the audio signal, the encoding side generates an audio signal encoding request based on the audio signal's configuration information. Subsequently, it obtains the audio signal's configuration information from the encoding request, queries the configuration table based on the configuration information, determines the encoding mode corresponding to the queried audio signal's configuration information as the target encoding mode, and determines the bitrate mode corresponding to the queried audio signal's configuration information as the target bitrate mode.

[0072] In step 102, the target coding mode is used to extract the coded features of the audio signal from the audio signal.

[0073] For example, if the target coding mode is a broadband coding mode, the coding features of the audio signal are extracted from the audio signal based on the broadband coding mode. If the target coding mode is an ultra-broadband coding mode, the coding features of the audio signal are extracted from the audio signal based on the ultra-broadband coding mode.

[0074] As shown in Figure 4B, Figure 4B is a schematic flowchart of an audio coding method according to an embodiment of the present application, and shows that when the target coding mode is a broadband coding mode, step 102 in Figure 4A can be realized by steps 1021A to 1022A. In step 1021A, a first neural network is invoked based on the broadband coding mode. In step 1022A, the first neural network extracts the coded features of the audio signal from the audio signal.

[0075] Here, if the target coding mode is a broadband coding mode, it means that the audio signal is a broadband signal (low frequency signal), and the coding features of the audio signal can be directly extracted using artificial intelligence technology, thereby enabling the generation of a corresponding audio bitstream based on the coding features of the audio signal in subsequent steps. It should be noted that the embodiments of this application do not limit the structure of the first neural network, and the first neural network may be a convolutional neural network, a deep neural network, or the like.

[0076] As shown in Figure 4C, Figure 4C is a schematic flowchart of an audio encoding method according to an embodiment of the present application, and Figure 4C shows that step 1022A in Figure 4B can be realized by steps 10221A to 10222A.

[0077] In step 10221A, feature extraction is performed on the audio signal to obtain the audio features of the audio signal.

[0078] In this embodiment of the present application, a first neural network (NN) is invoked based on an audio signal, and audio features are extracted from the audio signal by the first neural network, thereby enabling subsequent feature extraction based on the audio features. It should be noted that the embodiment of the present application does not limit the structure of the first NN, and the first NN may be a convolutional neural network, a deep neural network, or the like.

[0079] In step 10222A, residual processing is performed on the audio features using at least one residual unit included in the first neural network to obtain the encoded features of the audio signal.

[0080] Here, the first neural network includes four coding blocks, and each coding block includes four or five residual units.

[0081] In neural network models, residual units are a special structure used in the construction of residual networks (ResNets). Residual units are intended to solve the vanishing gradient and gradient explosion problems during the training process of deep neural networks, helping the network learn features better. Residual units introduce skip connections, meaning that inputs are directly added to outputs rather than simply propagating between layers. This skip connection allows the network to learn the residual function—the difference between input and output—rather than learning a direct mapping relationship. This design facilitates network optimization and simultaneously helps mitigate the vanishing gradient problem.

[0082] Here, by performing residual processing on the encoding side for the audio features, we can ensure that comprehensive learning is performed on the audio features based on the characteristics of the residual processing, while at the same time making better use of the shallow feature information of the audio features and avoiding overlooking the shallow feature information of the audio features.

[0083] Based on the characteristics of the residual unit, the residual processing in step 10222A is used to calculate the residuals of the audio features on the encoding side for subsequent signal encoding, and to determine the residuals of the audio features as the encoded features. For example, the residuals of the audio features are obtained by adding the audio features to the output of the residual unit. That is, the audio features are used as input to the residual unit, and after the audio features are processed by the residual unit, the output of the residual unit is obtained. Due to the skip connection characteristics of the residual unit, the input and output of the residual unit are added to obtain the residuals of the audio features.

[0084] As shown in Figure 4D, Figure 4D is a schematic flowchart of an audio coding method according to an embodiment of the present application, and shows that when the target coding mode is an ultra-wideband coding mode, step 102 in Figure 4A can be realized by steps 1021B to 1024B.

[0085] In step 1021B, subband decomposition is performed on the audio signal to obtain the low-frequency subband signal and the high-frequency subband signal of the audio signal.

[0086] Here, if the target coding mode is an ultra-wideband coding mode, it means that the audio signal is an ultra-wideband signal. Subband decomposition can be performed on the ultra-wideband signal to obtain the low-frequency and high-frequency subband signals of the audio signal.

[0087] It should be noted that the embodiments of this application do not limit the frequency bandwidths of the low-frequency subband signal and the high-frequency subband signal. That is, the low-frequency subband signal and the high-frequency subband signal obtained by decomposition may be two subband signals obtained by equally dividing the frequency bandwidth of the audio signal, or they may be two subband signals obtained by unevenly dividing the frequency bandwidth of the audio signal. For example, if the effective bandwidth of the audio signal x(n) is 0 to 16 kHz, the low-frequency subband signal xLB (n) and high-frequency subband signal x HB The effective bandwidths of (n) are 0-8kHz and 8-16kHz, respectively, and the low-frequency subband signal x LB (n) and high-frequency subband signal x HB The effective bandwidths of (n) may further be 0-6kHz and 6-16kHz, respectively. Furthermore, the embodiments of this application do not limit the number of frequency bands to be divided. That is, two subband signals can be obtained by dividing equally or unevenly, and more than two subband signals, such as three, four, or more, can be obtained by dividing the frequency band of the audio signal equally or unevenly.

[0088] Here, an audio signal includes a low-frequency portion and a high-frequency portion. The low-frequency signal (i.e., the low-frequency subband signal) is the low-frequency portion of the audio signal separated from the audio signal at a specific sampling rate by a filter based on the characteristics of the audio signal. The high-frequency signal (i.e., the high-frequency subband signal) is the high-frequency portion of the audio signal separated from the audio signal at a specific sampling rate. For example, if the effective bandwidth of audio signal x(n) is 0 to 16 kHz, then the effective bandwidth of the low-frequency signal is 0 to 8 kHz, and the high-frequency signal x HB The effective bandwidth of (n) can be 6 to 16 kHz. Furthermore, the embodiments of this application do not limit the frequency band division of the audio signal; for example, the audio signal can be divided equally or unevenly to obtain uniform low-frequency and high-frequency signals.

[0089] As an example of subband decomposition, a QMF analysis filter is used to analyze an audio signal into a low-frequency subband signal x LB (n) and high-frequency subband signal x HB Decompose into (n). Since the low-frequency subband signal has a greater impact on audio coding than the high-frequency subband signal, differentiated signal processing can be performed on the low-frequency and high-frequency subband signals in subsequent steps.

[0090] In some embodiments, step 1021B can be implemented in the following manner: an audio signal is sampled to obtain a sampled signal, which includes multiple sample points obtained by sampling; a low-pass filter is applied to the sampled signal to obtain a low-pass filtered signal; a downsampling is applied to the low-pass filtered signal to obtain a low-frequency subband signal of the audio signal; a high-pass filter is applied to the sampled signal to obtain a high-pass filtered signal; and a downsampling is applied to the high-pass filtered signal to obtain a high-frequency subband signal of the audio signal.

[0091] Here, the audio signal is a continuous analog signal, the sampled signal is a discrete digital signal, and the sampling point is a sampled value obtained by sampling from the audio signal.

[0092] In the field of digital signal processing, downsampling is used to reduce the amount of data, simplify the system, or adapt to specific application requirements by lowering the sampling rate of an audio signal. Here, the downsampling factor can be a multiple of 2, such as 2, 4, or 8.

[0093] As an example, assuming the audio signal is an input signal with a sampling rate Fs = 32000 Hz, the audio signal is sampled to obtain a sampled signal x(n) containing 640 sample points. The analysis filter (2 channels) in the QMF filter bank is called and low-pass filtering is performed on the sampled signal to obtain the low-pass filtered signal. High-pass filtering is performed on the sampled signal to obtain the high-pass filtered signal. Downsampling is performed on the low-pass filtered signal to obtain the low-frequency subband signal x of the audio signal. LB(n) is obtained. Downsampling processing is performed on the high-pass filtered signal to obtain the high-frequency subband signal x of the audio signal HB (n). The low-frequency subband signal x LB (n) and the high-frequency subband signal x HB (n) have effective bandwidths of 0~8 kHz and 8~16 kHz respectively, and the low-frequency subband signal x LB (n) and the high-frequency subband signal x HB (n) have 320 sampling points.

[0094] It should be noted that the QMF filter bank is a filter pair including analysis and synthesis. For a QMF analysis filter, an input signal with a sampling rate Fs can be decomposed into two signals with a sampling rate Fs / 2, which are represented as a QMF low-pass signal and a QMF high-pass signal respectively. The low-pass signal and high-pass signal restored on the decoding side are subjected to synthesis processing by the QMF synthesis filter, that is, a reconstructed signal with the sampling rate Fs corresponding to the input signal can be restored.

[0095] In the field of digital signal processing, the embodiments of the present application first filter an audio signal through filters (low-frequency filter, high-pass filter, etc.) to remove high-frequency components and aliasing interference in the audio signal, ensure that the audio signal after downsampling does not lose necessary information, and then retain one sampling point at every fixed number of sampling points in the filtered audio signal through downsampling processing, thereby reducing the sampling rate of the audio signal.

[0096] In step 1022B, a second neural network extracts low-frequency features of the low-frequency subband signal from the low-frequency subband signal.

[0097] It should be explained that step 1022B is similar to step 10221A, the difference being that step 10221A processes audio signals, while step 1022B processes low-frequency subband signals. Here, the structures of the first and second neural networks are similar. For example, the second neural network, like the first neural network, includes four coding blocks, and each coding block includes four or five residual units.

[0098] In some embodiments, step 1022B can be achieved by steps 10221B to 10222B.

[0099] In step 10221B, feature extraction is performed on the low-frequency subband signal to obtain the low-frequency features of the low-frequency subband signal.

[0100] In this embodiment of the present application, a second neural network (NN) is invoked based on a low-frequency subband signal, and low-frequency features are extracted from the low-frequency subband signal by the second neural network, thereby enabling subsequent feature extraction based on important low-frequency features. It should be noted that the embodiment of the present application does not limit the structure of the second NN, which may be a convolutional neural network, a deep neural network, or the like.

[0101] In some embodiments, step 1022B can be implemented in the following manner: A second neural network performs causal convolution on the low-frequency subband signal of the audio signal to obtain causal convolution features, and then performs pooling on the causal convolution features to obtain low-frequency encoded features of the low-frequency subband signal.

[0102] In the field of audio codecs, neural network (NN) operations such as causal convolution and pooling play a crucial role in processing audio signals and extracting features within them. In audio codecs, causal convolution can be used to extract local features within an audio signal. By applying a convolution kernel (a type of learnable filter), convolution operations can be performed in the temporal dimension of the audio signal, capturing modes and resonances within the signal. Causal convolution can extract features in the time and frequency domains of an audio signal and can be used for tasks such as noise reduction, feature extraction, and signal separation. Pooling operations are used to reduce the temporal dimension of an audio signal, thereby reducing data complexity and computational cost. Pooling can sample local regions of the input signal, aggregating information from those regions such as maximum and average values, thereby generating a more compact feature representation. In audio signals, pooling can help improve the robustness and generalization ability of the network and reduce the risk of overfitting. In the field of audio codecs, operations such as convolution and pooling can be used to construct appropriate neural network structures, enabling tasks such as feature extraction, encoding, and decoding of audio signals. These operations improve the efficiency and quality of audio signal processing and help expand the range of applications of audio encoding and decoding technologies in fields such as audio processing, speech recognition, and music generation.

[0103] For example, referring to the network structure diagram of the second NN shown in Figure 11, the second NN includes one causal convolution layer and a preprocessing layer. First, a 16-channel causal convolution layer is invoked to expand the input tensor (i.e., the low-frequency subband signal) into 16 × 320 causal convolution features. Next, the preprocessing layer preprocesses the 16 × 320 causal convolution features. For example, after performing one convolution operation on the 16 × 320 causal convolution features, a pooling operation with a factor of 2 is performed, and the activation function may be a Parametric Rectified Linear Unit (PReLU), generating a 16 × 160 tensor (i.e., a low-frequency encoded feature).

[0104] In step 10222B, residual processing is performed on the low-frequency coding features using at least one residual unit included in the second neural network to obtain the low-frequency features of the low-frequency subband signal.

[0105] Here, by performing residual processing on the low-frequency coding features of the low-frequency subband signal, we can ensure that comprehensive learning of the low-frequency coding features is performed based on the characteristics of residual processing. At the same time, we can better utilize the shallow feature information of the low-frequency coding features and avoid overlooking it.

[0106] In some embodiments, step 10222B can be achieved by steps 102221B to 102222B.

[0107] In step 102221B, feature residual processing is performed on the low-frequency features by at least one residual unit included in the second neural network to obtain the residual features of the low-frequency subband signal.

[0108] Here, the feature residual processing in step 102221B calculates the residuals of the low-frequency features and uses these residuals to determine the residual features of the low-frequency subband signal for subsequent feature coding.

[0109] In some embodiments, if at least one residual unit is a single residual unit, step 102221B can be implemented in the following manner: one residual unit performs one residual processing on the low-frequency features to obtain the residual features of the low-frequency subband signal. Here, one residual processing of one residual unit is used to calculate the residual corresponding to the low-frequency subband signal once on the encoding side.

[0110] In some embodiments, if at least one residual unit is a plurality of cascaded residual units, step 102221B can be implemented in the following manner: the first residual unit of the plurality of cascaded residual units performs one residual process on the low-frequency coding feature, where the one residual process of the first residual unit calculates the residual of the low-frequency coding feature once and is used to determine the residual of the low-frequency coding feature as the residual result of the first residual unit, the residual result output by the first residual unit is output to subsequent cascaded residual units, and the subsequent cascaded residual units continue to perform one residual process and output residual results, where the one residual process of the subsequent cascaded residual units is used to calculate the residual of the residual results input to the subsequent cascaded residual units, and the residual result output by the last residual unit is taken as the residual feature of the low-frequency subband signal.

[0111] For example, the second neural network includes four coding blocks, and each coding block includes four or five residual units. As shown in Figure 12A, if at least one residual unit used for feature residuals has five cascaded residual units, the first residual unit performs one residual process on the low-frequency coded features and outputs the residual result output by the first residual unit to the second residual unit. The second residual unit performs one residual process on the residual result output by the first residual unit and outputs the residual result output by the second residual unit to the third residual unit. The third residual unit performs one residual process on the residual result output by the second residual unit and outputs the residual result output by the third residual unit to the fourth residual unit. The fourth residual unit performs one residual process on the residual result output by the third residual unit and outputs the residual result output by the fourth residual unit to the fifth residual unit. The fifth residual unit performs one residual processing operation on the residual result output by the fourth residual unit to obtain the residual features of the low-frequency subband signal.

[0112] Each embodiment of this application does not limit the number of coding blocks in the neural network, and can be any positive integer such as 2, 3, 4, or 5. Similarly, the embodiments of this application do not limit the number of residual units in a coding block, and can be any positive integer such as 2, 3, 4, 5, or 6. The number of residual units in multiple coding blocks may be the same or different; for example, one coding block may contain 4 residual units, while another coding block may contain 5 residual units.

[0113] In some embodiments, the processing process for residual units is as follows: The k-th residual unit among multiple cascaded residual units performs the following operation: a convolution operation is performed on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. The convolution result of the k-th residual unit and the input of the k-th residual unit are added together to obtain the residual result output by the k-th residual unit. Here, k is a progressively increasing positive integer, 1 ≤ k ≤ J, and J is the number of residual units. If k is 1, the input of the k-th residual unit is a low-frequency coded feature; if k is not 1, the input of the k-th residual unit is a residual feature and the residual result output by the (k-1)-th residual unit.

[0114] Following the above embodiment, each residual unit includes a dilated convolution operator. Performing convolution processing on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit can be achieved by the following method. The k-th residual unit of the plurality of cascaded residual units performs the following processing, that is, performing dilated convolution processing on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. That is, the dilated convolution operator included in the first residual unit performs dilated convolution processing on the low-frequency encoded feature to obtain the dilated convolution result of the first residual unit. The j-th residual unit among the plurality of cascaded residual units performs the following processing, that is, the dilated convolution operator included in the j-th residual unit performs dilated convolution processing on the residual result output by the j-1-th residual unit to obtain the dilated convolution result of the j-th residual unit. Here, j is a positive integer that increases sequentially, 1<j≦J, and J is the number of residual units. It should be noted that each residual unit includes a dilated convolution operator with a specified dilation rate. Using a dilated convolution operator with a progressive dilation rate is equivalent to extracting input features at different resolutions using different receptive fields, which enables a relatively more comprehensive analysis of the data to be better performed. After each residual unit undergoes convolution processing by the dilated convolution operator with the specified dilation rate, it is added to the shallow feature from the skip connection (that is, the input of each residual unit), so that the shallow feature information is directly utilized, and the network can fully utilize the shallow feature information during the learning process.

[0115] Continuing from the above embodiment, each residual unit includes not only an extended convolution operator but also at least one causal convolution operator, performs a convolution operation on the input of the k-th residual unit, and after obtaining the convolution result of the k-th residual unit, performs a causal convolution operation on the obtained extended convolution result using at least one causal convolution operator included in the k-th residual unit, and the obtained causal convolution result is taken as the convolution result of the k-th residual unit. That is, performs a causal convolution operation on the extended convolution result of the first residual unit using at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is taken as the convolution result output from the first residual unit. An extended convolution operation is performed on the residual result output by the (j-1)th residual unit using the extended convolution operator contained in the j-th residual unit to obtain the extended convolution result of the j-th residual unit. Then, a causal convolution operation is performed on the extended convolution result of the j-th residual unit using at least one causal convolution operator contained in the j-th residual unit, and the causal convolution result of the j-th residual unit is taken as the convolution result of the j-th residual unit. It should be noted that each residual unit also contains at least one causal convolution operator, which continues to extract local information of the features input to the causal convolution operator.

[0116] In neural network models, causal convolution is a special type of processing used when dealing with time-series data (audio signals are a type of time-series data). It ensures that the output of the neural network depends only on the current and past time steps, thereby preserving temporal causality. In practical applications, causal convolution can ensure that the kernel does not extend beyond the region prior to the current time step by adjusting the size of the convolution kernel. This allows for effective capture of long-term dependencies in the time series while avoiding gradient vanishing or explosion problems caused by future information. Causal convolution is particularly important in fields such as natural language processing, speech recognition, and time series forecasting because it adheres to the temporal order of the data, avoids confusion of past information, and can effectively process and forecast long-term series data. In tasks such as speech recognition and time series forecasting, causal convolution performs well because it can preserve the characteristics of temporal order.

[0117] In some embodiments, grouped convolution can be applied to residual unit convolution operators (including extended convolution operators and causal convolution operators). Grouped convolution involves dividing the input channels into multiple groups and performing the convolution operation, where input channels and output channels are associated only within each group. It should be noted that after the input channels are divided into multiple groups, the corresponding output channels are also divided into multiple groups. That is, the number of input channel groups is the same as the number of output channel groups, thereby ensuring that after convolution within a group, input channels and output channels are associated only within each group. Here, assuming that a certain convolution operator has four input channels and four output channels, if the number of groupings is 1, each input channel is associated with one of the four output channels. If the number of groupings is 2, the four input channels are first divided into two groups, 0-1 and 2-3, and within these two groups, the input channels are associated with the output channels within that group. For example, input channels 0-1 in the first group are associated with output channels 0-1, and input channels 2-3 in the second group are associated with output channels 2-3. As shown in Figure 6A, if the grouped convolution solution is not adopted, each input channel is associated with all four output channels. As shown in Figure 6B, if the grouped convolution solution is not adopted, the 0th output channel is associated only with input channels 0-1 and not with input channels 2-3, and the 2nd output channel is associated only with input channels 2-3 and not with input channels 0-1. As this comparison shows, by introducing grouped convolution, it is possible to avoid any arbitrary input channel being associated with all output channels, reduce the number of connections, and lower complexity.

[0118] For example, when applying grouped convolution to an extended convolution operator included in a residual unit, extending convolution processing to low-frequency encoded features can be achieved in the following manner: the input channels of the low-frequency encoded features are grouped to obtain multiple groups, where each group contains a first element corresponding to at least two channels in the low-frequency encoded features, and the extended convolution processing is performed on the first element within each group. When applying grouped convolution to a causal convolution operator included in a residual unit, causal convolution processing to the obtained extended convolution result can be achieved in the following manner: the input channels of the extended convolution result are grouped to obtain multiple groups, where each group contains a second element corresponding to at least two channels in the extended convolution result, and the causal convolution processing is performed on the second element within each group.

[0119] For example, grouped convolution can be applied to residual unit convolution operators (including extended convolution operators and causal convolution operators). Grouped convolution involves dividing the input channels into multiple groups and performing the convolution operation, where input channels and output channels are associated only within each group. It should be explained that after the input channels are divided into multiple groups, the corresponding output channels are also divided into multiple groups. That is, the number of input channel groups is the same as the number of output channel groups, thereby ensuring that after convolution within a group, input channels and output channels are associated only within each group. Here, assuming that a certain convolution operator has four input channels and four output channels, if the number of groupings is 1, each input channel is associated with one of the four output channels. If the number of groupings is 2, the four input channels are first divided into two groups, 0-1 and 2-3, and within these two groups, the input channels are associated with the output channels within that group. For example, input channels 0-1 in the first group are associated with output channels 0-1, and input channels 2-3 in the second group are associated with output channels 2-3. As shown in Figure 6A, if the grouped convolution solution is not adopted, each input channel is associated with all four output channels. As shown in Figure 6B, if the grouped convolution solution is not adopted, the 0th output channel is associated only with input channels 0-1 and not with input channels 2-3, and the 2nd output channel is associated only with input channels 2-3 and not with input channels 0-1. As this comparison shows, by introducing grouped convolution, it is possible to avoid any arbitrary input channel being associated with all output channels, reduce the number of connections, and lower complexity.

[0120] Following on from step 102221B, in step 102222B, feature coding is performed on the residual features to obtain the low-frequency features of the low-frequency subband signal.

[0121] Here, feature coding is performed on the residual features to obtain the low-frequency features of the low-frequency subband signal. Subsequently, signal coding is performed based on these low-frequency features to obtain the low-frequency bitstream of the audio signal.

[0122] In some embodiments, step 102222B can be implemented by the following method: convolution is performed on the residual features to obtain convolutional features, where the number of channels in the convolutional features is greater than the number of channels in the residual features, and pooling is performed on the convolutional features to obtain low-frequency features of the low-frequency subband signal.

[0123] For example, a second neural network (NN) is invoked based on a low-frequency subband signal, processed by a residual unit within the second NN, and then residual features are obtained. Next, a convolutional layer within the second NN performs a convolution on the residual features, increasing the number of channels in the residual features. Finally, a pooling layer within the second NN performs a pooling on the convolutional features to obtain the low-frequency features of the low-frequency subband signal. Naturally, the second NN can further include a causal convolutional layer. The causal convolutional layer performs a causal convolution on the low-frequency features to obtain the low-frequency features after causal convolution, and then signal coding is performed on the low-frequency features of the low-frequency subband signal after causal convolution to obtain a low-frequency bitstream of the audio signal.

[0124] For example, as shown in Figure 11, a second neural network (NN) is invoked based on a low-frequency subband signal, and the second NN then obtains low-frequency coded features (a 16 × 160 tensor obtained after preprocessing in Figure 11). The second NN includes coding blocks of four different downsampling factors (Down_factors) that are cascaded together. Here, each coding block includes one residual block (containing at least one residual unit), one convolutional layer, and one pooling layer. Here, each residual block contains five residual units based on extended convolutions (the input and output feature dimensions of the residual units do not change). One convolutional layer is used to double the number of input channels, and the activation function may be PReLU, thereby guaranteeing the amount of data and avoiding data loss. The pooling layer is a pooling operation that includes a Down_factor to perform downsampling and achieve data compression. Here, the Down_factors of the four coding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels for the four coding blocks are set to 32, 64, 128, and 256, respectively. After being processed by the four coding blocks, the input 16×160 tensor is transformed into tensors of 32×80, 64×20, 128×5, and 256×1, respectively.For example, the residual block in the first coding block performs residual processing on the low-frequency coded feature (i.e., a 16x160 tensor), the residual result output by the residual block in the first coding block is output to the feature coding block in the first coding block, and after processing by the feature coding block in the first coding block (which includes one convolutional layer and one pooling layer), the coded result of the feature coding block in the first coding block (i.e., a 32x80 tensor) is obtained, the coded result of the feature coding block in the first coding block (i.e., a 32x80 tensor) is output to the second coding block, and the residual block in the second coding block processes the first Residual processing is performed on the encoding result of the feature encoding block within the first encoding block (i.e., a 32 × 80 tensor), the residual result output by the residual block within the second encoding block is output to the feature encoding block within the second encoding block, processed by the feature encoding block within the second encoding block (which includes one convolutional layer and one pooling layer), and the encoding result of the feature encoding block within the second encoding block (i.e., a 64 × 20 tensor) is obtained. The encoding result of the feature encoding block within the first encoding block (i.e., a 64 × 20 tensor) is output to the third encoding block, and the above processing is performed sequentially until the output of the last encoding block is the low-frequency feature. Here, the embodiment of this application does not limit the number of encoding blocks, and can be any positive integer such as 2, 3, 4, 5, etc.

[0125] In step 1023B, high-frequency analysis is performed on the high-frequency subband signal to obtain the high-frequency features of the high-frequency subband signal.

[0126] Because low-frequency subband signals have a greater impact on audio coding than high-frequency subband signals, differentiated signal processing is performed on the low-frequency and high-frequency subband signals so that the feature dimension of the high-frequency features is lower than that of the low-frequency features. For example, the feature dimension of the low-frequency features is 56, and the feature dimension of the high-frequency features is 8. Here, high-frequency analysis is used to perform dimensionality reduction on the high-frequency subband signals and achieve data compression. High-frequency coding features are features that represent the high-frequency subband signals, and the feature dimension of the high-frequency coding features is smaller than that of the high-frequency subband signals.

[0127] In some embodiments, a fifth neural network can be invoked to extract high-frequency features from a high-frequency subband signal. The structure of the fifth neural network and the second neural network are similar. Here, the dimension of the high-frequency features is smaller than the dimension of the low-frequency features.

[0128] In some embodiments, high-frequency analysis can be performed on a high-frequency subband signal to obtain its high-frequency characteristics, which can be achieved by the following method: frame division is performed on the high-frequency subband signal to obtain multiple subframes of the high-frequency subband signal; frequency bandwidth expansion processing is performed on each subframe to obtain the subband spectral envelope of each subframe; and the subband spectral envelopes corresponding to each of the multiple subframes are used as the high-frequency characteristics of the high-frequency subband signal. Here, the number of multiple subframes may be 2, 4, 6, etc., and the embodiments of this application do not limit the number of subframes.

[0129] Here, since high-frequency subband signals are of relatively lower importance to quality compared to low-frequency subband signals, they can be compressed by another method, namely frequency band expansion (recovering a wideband audio signal from a narrowband audio signal with a limited frequency band). This allows for rapid compression of the high-frequency subband signals and extraction of their high-frequency characteristics.

[0130] In some embodiments, frequency bandwidth expansion processing can be performed on each subframe to obtain the subband spectral envelope for each subframe by the following method: a frequency domain transformation is performed based on multiple sample points contained in the subframe to obtain transformation coefficients corresponding to each of the multiple sample points, these transformation coefficients are divided into multiple subbands, the transformation coefficients contained in each subband are averaged to obtain the average energy corresponding to each subband, and the average energy is used as the subband spectral envelope corresponding to each subband.

[0131] It should be noted that the frequency domain transformation method in the embodiments of this application includes modified discrete cosine transform (MDCT), discrete cosine transform (DCT), fast fourier transform (FFT), etc., and the embodiments of this application are not limited to the method of frequency domain transformation. The averaging process in the embodiments of this application includes arithmetic mean and geometric mean, and the embodiments of this application are not limited to the method of averaging.

[0132] In some embodiments, frequency domain transformation based on multiple sample points contained in a subframe and obtaining transformation coefficients corresponding to each of the multiple sample points can be achieved by the following method: Perform the following process for each subframe, namely, obtain adjacent subframes adjacent to the subframe, and perform a discrete cosine transform on the multiple sample points contained in the subframe based on the multiple sample points contained in the adjacent subframes and the multiple sample points contained in the subframe, thereby obtaining transformation coefficients corresponding to each of the multiple sample points contained in the subframe.

[0133] For example, a high-frequency subband signal x containing 320 points. HB For (n), the high-frequency subband signal x of 320 points HB (n) is divided into two subframes (a first subframe and a second subframe), each containing 160 points. For any subframe, i.e., a high-frequency subband signal containing 160 points, a Modified Discrete Cosine Transform (MDCT) is called to generate MDCT coefficients for the 160 points (i.e., transformation coefficients corresponding to each of the multiple sample points contained in the subframe). Specifically, in the case of a 50% overlap, the first subframe of the (n+1)th frame can be merged with the second subframe of the nth frame to calculate the MDCT for 320 points and obtain the MDCT coefficients for the 160 points of the first subframe of the (n+1)th frame. The second subframe of the (n+1)th frame can be merged with the first subframe of the (n+1)th frame to calculate the MDCT for 320 points and obtain the MDCT coefficients for the 160 points of the second subframe of the (n+1)th frame.

[0134] In some embodiments, the process of performing a geometric mean on the transformation coefficients included in each subband is as follows: determine the sum of squares of the transformation coefficients corresponding to the sample points included in each subband, and determine the ratio of the sum of squares to the number of sample points included in the subband as the average energy corresponding to each subband.

[0135] Continuing from step 1023B, in step 1024B, the low-frequency and high-frequency features are determined as the encoded features of the audio signal.

[0136] Continuing from step 102, in step 103, signal encoding is performed on the encoded features using the target bitrate mode to obtain the audio bitstream of the audio signal.

[0137] Here, the bitrate mode is used to perform signal coding using a specified codebook and the corresponding bitrate. Therefore, signal coding is performed on the coding features of the audio signal using the codebook and bitrate corresponding to the target bitrate mode to obtain the audio bitstream of the audio signal. In the field of digital signal processing, step 103 can be implemented in the following way: Digital signal-based coding is performed on the coding features using the codebook and bitrate corresponding to the target bitrate mode to obtain the audio bitstream of the audio signal.

[0138] In some embodiments, when the target coding mode is a broadband coding mode, the target bitrate mode is used to indicate that signal coding is performed on the coding features of the audio signal using a target codebook. Step 103 can be implemented in the following manner: the target codebook is used to quantize the coding features of the audio signal to obtain quantized values ​​of the coding features, and the target bitrate corresponding to the target codebook is used to entropy code the quantized values ​​to obtain an audio bitstream of the audio signal.

[0139] As shown in Figure 4E, Figure 4E is a schematic flowchart of an audio coding method according to an embodiment of the present application, and when the target coding mode is an ultra-wideband coding mode, the target bitrate mode includes indicating that a first codebook is used to perform signal coding on low-frequency features and a second codebook is used to perform signal coding on high-frequency features. Figure 4E shows that step 103 in Figure 4A can be realized by steps 1031 to 1033. In step 1031, the first codebook is used to perform quantization on low-frequency features to obtain quantized values ​​of the low-frequency features, and entropy coding is performed on the quantized values ​​of the low-frequency features using the bitrate corresponding to the first codebook to obtain a low-frequency bitstream of the low-frequency subband signal. In step 1032, the second codebook is used to perform quantization on high-frequency features to obtain quantized values ​​of the high-frequency features, and entropy coding is performed on the quantized values ​​of the high-frequency features using the bitrate corresponding to the second codebook to obtain a high-frequency bitstream of the high-frequency subband signal. In step 1033, the audio bitstream of the audio signal is constructed based on the low-frequency bitstream and the high-frequency bitstream.

[0140] Here, the quantization levels (i.e., quantization precision) corresponding to different codebooks may be different. For example, the quantization precision of the first codebook is higher than that of the second codebook. Since the interval of values ​​for each dimension in low-frequency features is [-1,1], the interval [-1,1] is divided into 11 equal parts to form a first codebook containing 11 elements, and the low-frequency features can be quantized using the first codebook. Since the interval of values ​​for each dimension in high-frequency features is [-1,1], the interval [-1,1] is divided into 5 equal parts to form a second codebook containing 5 elements, and the high-frequency features can be quantized using the second codebook.

[0141] Here, the low-frequency feature F of the low-frequency subband signal LB (n) and the high-frequency coding feature F of the high-frequency subband signalHB For (n), scalar quantization (quantizing each component individually) and entropy coding can be performed. Furthermore, the embodiments of this application are not limited to a combination of vector quantization (combining multiple adjacent components into a single vector and quantizing them together) and entropy coding techniques.

[0142] To explain further, if the target bitrate mode only indicates that a first codebook is used to perform signal coding for low-frequency features and a second codebook is used to perform signal coding for high-frequency features, then the low-frequency bitstream and the high-frequency bitstream are combined to obtain the audio bitstream of the audio signal.

[0143] In some embodiments, the target bitrate mode further includes indicating that a third codebook is used to perform signal coding on the residual features of the high-frequency features. Prior to step 1033, the subbands corresponding to the subframes of the high-frequency subband signal are determined, the subbands are divided into multiple subsets, the transformation coefficients contained in each subset are averaged to obtain the average energy corresponding to each subset, and the average energy is taken as the envelope value corresponding to each subset. In the quantized value of the high-frequency features, the quantized value of the subband spectral envelope of the subband corresponding to the subset is determined, the difference between the envelope value corresponding to the subset and the quantized value of the subband spectral envelope is taken as the first residual value, the residual features of the high-frequency features are determined based on the first residual value, the residual features of the high-frequency features are quantized using the third codebook to obtain the quantized value of the residual features of the high-frequency features, and the quantized value of the residual features of the high-frequency features is entropy coded using the bitrate corresponding to the third codebook to obtain the residual bitstream of the residual features. Accordingly, step 1033 can be implemented in the following manner: The low-frequency bitstream, high-frequency bitstream, and residual bitstream are used as the audio bitstream of the audio signal. Here, the quantization level of the third codebook may differ from the quantization levels of the first and second codebooks.

[0144] In some embodiments, if the number of residual values ​​is N, N is a positive integer greater than 1, and determining the residual features of high-frequency features based on the first residual value can be achieved by the following method: Determine the quantized value of the nth residual value, determine the sum of the quantized value of the nth residual value and the quantized value of the subband spectral envelope of the corresponding subband, and take the difference between the envelope value corresponding to the subset and the sum as the (n+1)th residual value, thereby determining the N residual values ​​as the residual features of high-frequency features. Here, n is a progressively increasing positive integer, and 1 ≤ n ≤ N.

[0145] Using the example of a case where the number of residual values ​​is 2, we will explain how to determine the residual values ​​using the following method.

[0146] 1) For each 10ms subframe, divide the subframe into four subbands, and then further divide each subband into two subsets equally.

[0147] For example, the set of envelope values ​​for four subbands is

number

[0148] 2) For each subset of each subband, calculate the envelope value for that subset.

[0149] For example, the set of envelope values ​​for all subsets is

number

[0150] 3) Subtract the quantized value of the corresponding high-frequency feature vector (i.e., the quantized value of the high-frequency feature) from the envelope value of each subset to obtain the first high-frequency feature vector residual (abbreviated as the first residual value).

[0151] For example, the quantization value of a high-frequency feature vector

number

number

[0152] 4) Using the above-mentioned first high-frequency feature residual, the quantized value of the first high-frequency feature vector residual (abbreviated as the quantized value of the first residual) is obtained by using similar quantization and entropy coding.

[0153] Here, considering that the dynamic range of the residual values ​​is much lower than that of the original values, it is therefore sufficient to allocate only 2 bits when performing scalar quantization on each residual. Thus, to obtain the quantized value of the first high-frequency feature vector residual, a reference bitrate of 2 × 16 × 50 = 1.6 kbps is required (similarly, since it is a non-uniformly distributed entropy encoding, the actual bitrate is generally less than 1.6 kbps).

[0154] 5) From the envelope value of each subset, subtract the sum of the quantized value of the corresponding high-frequency feature vector and the quantized value of the first high-frequency feature vector residual to obtain the second high-frequency feature vector residual (abbreviated as the second residual value).

[0155] In step 104, the frame header is determined based on the target coding mode and target bitrate mode.

[0156] Here, the shell frame header can be constructed first. This shell frame header includes an encoding mode bit and a bitrate mode bit. The target encoding mode is written to the encoding mode bit included in the shell frame header, and the target bitrate mode is written to the bitrate mode bit included in the shell frame header, thereby determining the frame header. The embodiments of this application do not limit the number of bits in the frame header, nor do they limit the positions of the target encoding mode and target bitrate mode within the frame header. For example, the target encoding mode may be written to the first bit of the frame header, or to the last bit of the frame header, and the number of bits occupied by the encoding mode bit and bitrate mode bit in the frame header is not limited.

[0157] In step 105, an audio bitstream package of the audio signal is generated based on the audio bitstream and frame header.

[0158] For example, after obtaining the audio bitstream and frame header, the audio bitstream and frame header are packaged into a single file according to a specific format to obtain an audio bitstream package, which contains the audio bitstream and frame header.

[0159] In some embodiments, the flatness side information of the high-frequency subband signal is determined before step 105. Therefore, step 105 can be implemented by the following method: The audio bitstream, frame header, and flatness side information are combined to obtain an audio bitstream package of the audio signal.

[0160] For example, the frame header may be placed at the beginning of the audio bitstream package, and the audio bitstream and flatness side information may be placed after the frame header. Embodiments of this application do not limit the order of the audio stream and flatness side information. In some embodiments, the target coding mode and target bitrate mode may be located elsewhere in the audio bitstream package.

[0161] Here, the flatness of a high-frequency subband signal refers to the degree to which the amplitudes of each frequency component in the frequency domain of the high-frequency subband signal are uniformly distributed. Flatness is an important indicator for evaluating the quality of a high-frequency subband signal. Flatness side information of a high-frequency subband signal refers to additional information about the flatness of a high-frequency subband signal when processing the signal. When performing frequency domain analysis on a high-frequency subband signal, flatness side information includes the amplitude distribution of the high-frequency subband signal at each frequency point and is used to determine whether the high-frequency subband signal maintains consistency within a particular frequency range or whether there are prominent peaks. When balancing a signal, flatness side information helps identify valleys and peaks in the high-frequency subband signal, thereby enabling appropriate gain adjustment for the high-frequency subband signal, making the overall frequency response flatter and improving sound quality. Flatness side information can also be used to analyze the degree of distortion of a signal; if the amplitude of the high-frequency subband signal at one frequency point differs significantly from that at other frequency points, it can cause distortion and affect sound quality. In summary, flatness side information of high-frequency subband signals is of crucial importance for signal pre-processing, analysis, balancing, and post-processing, and helps improve the accuracy and effectiveness of audio signal processing.

[0162] In some embodiments, determining the flatness side information of a high-frequency subband signal can be achieved by the following method: The conversion coefficients included in the subframe of the high-frequency subband signal are divided into multiple blocks, a first flatness is determined for each block, and a second flatness is determined for a low-frequency specified band of the audio signal. If the first flatness is less than the second flatness, or if the first flatness is less than the flatness threshold, the flatness side information is set to a first value, where the first value indicates that flattening processing is required during audio decoding. If the first flatness is greater than or equal to the second flatness, and the first flatness is greater than or equal to the flatness threshold, the flatness side information is set to a second value, where the second value indicates that flattening processing is not required during audio decoding.

[0163] Here, the low-frequency specified bandwidth is a low-frequency bandwidth set according to the actual application requirements. The flatness threshold is also a threshold set according to the actual application requirements.

[0164] For example, here, the first value may be 1, and the second may be 0. The first value may be 0, and the second may be 1.

[0165] In some embodiments, the frame header further includes at least one audio channel bit, which is used to indicate that audio coding is performed on the audio signal using either a monaural or stereo coding method. If the audio signal is a stereo signal downmixed by the input signals of the left audio channel and the right audio channel, the audio channel bit in the frame header is set to a first value (e.g., 1). The first value indicates that coding is performed on the audio signal using a stereo coding method. Parametric stereo coding is performed on the input signals of the left audio channel and the right audio channel to obtain a stereo feature vector, and an audio bitstream package of the audio signal is generated based on the stereo feature vector, the audio bitstream, and the frame header. If the audio signal is a monaural input signal, the audio channel bit in the frame header is set to a second value (e.g., 0), which indicates that coding is performed on the audio signal using a monaural coding method. An audio bitstream package of the audio signal is generated based on the audio bitstream and the frame header.

[0166] As described above, the audio decoding method provided in the embodiments of this application can be implemented by various types of electronic devices. Referring to Figure 5A, Figure 5A is a schematic flowchart of the audio decoding method according to the embodiments of this application, and the audio decoding method realizes the audio decoding function, and the audio decoding method is the reverse process of the audio encoding method described above. The following explanation will be given with reference to the steps shown in Figure 5A.

[0167] Before describing the following steps, the audio decoding method provided in the embodiments of this application will be described. The audio decoding method provided in the embodiments of this application includes two decoding methods: a signal decoding method in signal processing techniques and a feature decoding (i.e., a reconstruction method) in artificial intelligence techniques. For details of the signal decoding method in signal processing techniques, see step 202 below. For details of feature decoding in artificial intelligence techniques, see step 203 below.

[0168] In step 200, obtain the audio bitstream package.

[0169] Here, the audio bitstream package includes an audio bitstream, which is obtained by performing audio encoding on an audio signal using a target encoding mode and a target bitrate mode. The target encoding mode is obtained from multiple candidate encoding modes, and the target bitrate mode is obtained from multiple candidate bitrate modes.

[0170] For example, after obtaining an audio bitstream package using the audio encoding method shown in Figure 4A, the audio bitstream package is transmitted to the decoding side, which then receives the audio bitstream package.

[0171] In step 201, in response to a decoding request for the audio bitstream package, the target encoding mode and target bitrate mode are obtained from the frame header contained in the audio bitstream package.

[0172] Here, the decryption request is used to indicate that audio decoding should be performed on the audio bitstream.

[0173] As an example, after obtaining an audio bitstream package using the audio encoding method shown in Figure 4A, the audio bitstream package is transmitted to the decoding side. After the decoding side receives the audio bitstream package, it performs audio decoding on the audio bitstream package. First, the target encoding mode and target bitrate mode are obtained from the frame header contained in the audio bitstream package. For example, the target encoding mode is obtained from the encoding mode bit contained in the frame header, and the target bitrate mode is obtained from the bitrate mode bit contained in the frame header.

[0174] In step 202, signal decoding is performed on the audio bitstream using the target coding mode and target bitrate mode to obtain coding feature estimates corresponding to the audio bitstream.

[0175] Here, since signal decoding is the inverse process of signal coding, signal decoding is performed on the audio bitstream using the decoding mode and target bitrate mode corresponding to the target coding mode, and an estimated coding feature value corresponding to the audio bitstream is obtained. For example, if the target coding mode is a broadband coding mode, signal decoding is performed on the audio bitstream using the broadband decoding mode, and an estimated coding feature value corresponding to the audio bitstream is obtained. If the target coding mode is an ultra-broadband coding mode, signal decoding is performed on the audio bitstream using the ultra-broadband decoding mode, and an estimated coding feature value corresponding to the audio bitstream is obtained.

[0176] It should be explained that signal decoding is the inverse process of signal coding. The decoding process on a received bitstream is the inverse process of the coding process on the encoding side. Therefore, the values ​​generated during decoding are estimates of the values ​​in the coding process. For example, the data values ​​representing coding features generated during decoding are estimates of the coding features in the coding process. In other words, the former is not necessarily equivalent to the data values ​​of the original coding features. For example, there may be subtle differences between the two caused by the coding and decoding operations, and there may be situations where decoding cannot completely restore the data values ​​of the original coding features. For this reason, the former is also called a "coding feature estimate."

[0177] In some embodiments, the target coding mode is a broadband coding mode, and the target bitrate mode is used to indicate that signal coding is being performed on the coding features of the audio signal using a target codebook. Therefore, step 202 can be implemented in the following manner: entropy decoding is performed on the audio bitstream using the target bitrate corresponding to the target codebook to obtain quantized values ​​corresponding to the audio bitstream, and inverse quantization is performed on the quantized values ​​corresponding to the audio bitstream using the target codebook to obtain estimated coding features corresponding to the audio bitstream.

[0178] Here, the inverse quantization process is performed by querying a quantization table, which is a mapping table generated by quantization during the encoding process. For example, upon receiving a bitstream, entropy decoding is performed first, and then the estimated feature vectors, i.e., the encoded feature estimates corresponding to the audio bitstream, are obtained by querying the quantization table (i.e., the inverse quantization, which is a mapping table generated by quantization during the encoding process).

[0179] Referring to Figure 5B, which is a schematic flowchart of an audio decoding method according to an embodiment of the present application, the target coding mode is an ultra-wideband coding mode. The target bitrate mode includes indicating that a first codebook is used to perform signal coding on the low-frequency features of the audio signal and a second codebook is used to perform signal coding on the high-frequency features of the audio signal. The audio bitstream includes a low-frequency bitstream and a high-frequency bitstream. Therefore, Figure 5B shows that step 202 can be achieved by steps 2021 to 2024. In step 2021, the target coding mode is used to obtain a low-frequency bitstream and a high-frequency bitstream from the audio bitstream. In step 2022, entropy decoding is performed on the low-frequency bitstream using the bitrate corresponding to the first codebook to obtain a quantized value corresponding to the low-frequency bitstream, and inverse quantization is performed on the quantized value corresponding to the low-frequency bitstream using the first codebook to obtain an estimated low-frequency feature value corresponding to the low-frequency bitstream. In step 2023, entropy decoding is performed on the high-frequency bitstream using the bitrate corresponding to the second codebook to obtain the quantized value corresponding to the high-frequency bitstream. Then, inverse quantization is performed on the quantized value corresponding to the high-frequency bitstream using the second codebook to obtain the high-frequency feature estimate corresponding to the high-frequency bitstream. In step 2024, the encoded feature estimate corresponding to the audio bitstream is determined based on the low-frequency feature estimate and the high-frequency feature estimate.

[0180] Here, the data values ​​representing low-frequency features generated during decoding are estimates with respect to the low-frequency features in the encoding process. That is, the former are not necessarily equivalent to the original low-frequency feature data values. For example, there may be subtle differences between the two caused by the encoding and decoding operations, and there may be situations where decoding cannot completely restore the original encoded feature data values. Therefore, the former are also called "low-frequency feature estimates." Similarly, the high-frequency feature estimates generated during decoding are also estimates with respect to the high-frequency features in the encoding process.

[0181] For example, for a low-frequency bitstream, entropy decoding is first performed on the low-frequency bitstream using the bitrate corresponding to the first codebook to obtain the quantization value corresponding to the low-frequency bitstream, and then by querying the first codebook, an estimate of the low-frequency features corresponding to the low-frequency bitstream is obtained.

number

number

[0182] In some embodiments, the audio bitstream further includes a residual bitstream, and the target bitrate mode further indicates that signal coding is being performed on the residual features of the high-frequency features corresponding to the audio bitstream using a third codebook. Prior to step 2024, entropy decoding is performed on the residual bitstream at the bitrate corresponding to the third codebook to obtain quantized values ​​corresponding to the residual bitstream, and inverse quantization is performed on the quantized values ​​corresponding to the residual bitstream using the third codebook to obtain residual feature estimates corresponding to the residual bitstream. Correspondingly, step 2024 can be implemented in the following manner: the sum of the high-frequency feature estimates and the residual feature estimates is determined as the final high-frequency feature estimate, and the low-frequency feature estimates and the final high-frequency feature estimates are determined as coded feature estimates corresponding to the audio bitstream.

[0183] Here, the i-th value among the high-frequency feature estimates is added to the 2i-1 and 2i-th values ​​among the residual feature estimates, respectively, to obtain the 2i-1 and 2i-th values ​​among the final high-frequency feature estimates. For example, the high-frequency feature estimates are

number

number

number

[0184] In step 203, the target coding mode is used to reconstruct the coded feature estimates and obtain the reconstructed audio signal corresponding to the audio bitstream.

[0185] It should be explained that the reconstruction process is the reverse process of the extraction process. The decoding process on the receiving bitstream is the reverse process of the encoding process; therefore, the values ​​generated during decoding are estimates of the values ​​in the encoding process. For example, the reconstructed audio signal generated during decoding is an estimate of the audio signal in the encoding process. In other words, the reconstructed audio signal is not necessarily equivalent to the original audio signal. There are subtle differences between the two caused by the encoding and decoding operations, and there may be situations where decoding cannot completely restore the original audio signal.

[0186] In some embodiments, the target coding mode is a broadband coding mode, and step 203 can be implemented in the following manner: A third neural network is invoked based on the broadband coding mode, and the third neural network reconstructs the coded feature estimates corresponding to the audio bitstream to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0187] It should be explained that the reconstruction process using the third neural network is the reverse process of the extraction process using the first neural network.

[0188] For example, reconstructing the encoded feature estimates corresponding to the audio bitstream and obtaining the reconstructed audio signal corresponding to the audio bitstream can be achieved by the following method: using at least one residual unit included in the third neural network, residual processing is performed on the encoded feature estimates corresponding to the audio bitstream to obtain audio feature estimates corresponding to the audio bitstream, feature reconstruction is performed on the audio feature estimates corresponding to the audio bitstream to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0189] Here, the third neural network includes four decoding blocks, each decoding block containing four or five residual units. Each embodiment of this application is not limited to the number of decoding blocks, which can be any positive integer such as 2, 3, 4, or 5. Each embodiment of this application is also not limited to the number of residual units within a decoding block, which can be any positive integer such as 2, 3, 4, 5, or 6.

[0190] Based on the characteristics of the residual unit, residual processing is performed on the estimated encoded features corresponding to the audio bitstream using at least one residual unit included in the third neural network, and the residual of the encoded features is calculated on the decoding side. For example, the residual of the encoded features is obtained by adding the encoded features with the output of the decoding side's residual unit. That is, the encoded features are used as input to the residual unit, the encoded features are processed by the decoding side's residual unit, the output of the residual unit is obtained, and due to the skip connection characteristics of the residual unit, the input of the decoding side's residual unit and the output of the residual unit are added to obtain the residual of the encoded features.

[0191] Referring to Figure 5C, which is a schematic flowchart of an audio decoding method according to an embodiment of the present application, where the target coding mode is an ultra-wideband coding mode, and Figure 5C shows that step 203 can be realized by steps 2031 to 2033.

[0192] In step 2031, the fourth neural network performs feature reconstruction on the low-frequency feature estimates included in the encoded feature estimates to obtain low-frequency subband signal estimates corresponding to the audio bitstream. Here, the structure of the fourth neural network is similar to that of the third neural network.

[0193] In some embodiments, step 2031 can be achieved by steps 20311 to 20312.

[0194] In step 20311, residual processing is performed on the low-frequency feature estimates using at least one residual unit in the fourth neural network to obtain low-frequency coded feature estimates. Here, the fourth neural network includes four decoding blocks, each decoding block including four or five residual units.

[0195] In some embodiments, step 20311 can be achieved by steps 203111 to 203112.

[0196] In step 203111, feature decoding is performed on the low-frequency feature estimates to obtain residual features.

[0197] For example, feature decoding is the inverse process of feature coding, where feature decoding is performed on low-frequency feature estimates to obtain residual features (a type of estimate).

[0198] In some embodiments, step 203111 can be implemented by the following method: convolution is performed on the low-frequency features to obtain convolutional features, where the number of channels in the convolutional features is less than the number of channels in the low-frequency features; and upsampling is performed on the convolutional features to obtain residual features.

[0199] In the field of audio codecs, upsampling operations are used to increase the resolution of feature maps (i.e., convolutional features) and reconstruct audio signals more accurately. Upsampling involves interpolation or other forms of upsampling techniques to generate more accurate feature maps, thereby helping to better restore the original details and characteristics of the audio signal during decoding. By using neural network techniques such as convolution, pooling, and upsampling in audio decoding, useful features can be effectively extracted, computational complexity can be reduced, and the original content of the audio signal can be reconstructed more accurately. These techniques are of great significance to improving the performance and efficiency of audio decoding and help to promote the development and application of audio codec technology.

[0200] Of course, before step 203111, it is also possible to perform causal convolution on the low-frequency features to obtain the low-frequency features after causal convolution, and then perform step 203111 based on the low-frequency features after causal convolution. That is, feature decoding is performed on the low-frequency features after causal convolution to obtain residual features.

[0201] In step 203112, feature residual processing is performed on the residual features by at least one residual unit included in the fourth neural network to obtain low-frequency coded feature estimates.

[0202] By performing residual processing on the residual features, we can ensure that the model learns the residual features comprehensively, while also making better use of the shallow feature information of the residual features and avoiding overlooking it.

[0203] The residual features obtained by performing feature decoding on low-frequency feature estimates are estimates of the residual features on the encoding side. Since the encoding and decoding processes are inverse processes of each other, the residual features obtained by performing feature decoding on low-frequency feature estimates are not the residual results obtained by residual calculation on the decoding side. After obtaining the residual features obtained by performing feature decoding on low-frequency feature estimates, residual calculations can be performed on the residual features obtained by performing feature decoding on low-frequency feature estimates.

[0204] In some embodiments, if at least one residual unit is a plurality of cascaded residual units, step 203112 can be implemented in the following manner: one residual unit performs one residual process on the residual features to obtain a low-frequency coded feature estimate.

[0205] In some embodiments, if at least one residual unit is a plurality of cascaded residual units, step 203112 can be implemented in the following manner: The first residual unit of the plurality of cascaded residual units performs one residual process on the residual features, outputs the residual result output by the first residual unit to the subsequent cascaded residual units, which then perform one residual process and output the residual result, and the residual result output by the last residual unit is taken as the low-frequency coded feature estimate.

[0206] In some embodiments, the processing process for residual units is as follows: The k-th residual unit of a cascaded set of residual units performs the following operation: a convolution operation is performed on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit; the convolution result of the k-th residual unit and the input of the k-th residual unit are added together to obtain the residual result output by the k-th residual unit. Here, k is a progressively increasing positive integer, 1 ≤ k ≤ J, where J is the number of residual units. If k is 1, the input of the k-th residual unit is a residual feature; if k is not 1, the input of the k-th residual unit is the residual result output by the (k-1)-th residual unit.

[0207] Continuing from the above embodiments, each residual unit includes a dilated convolution operator. Performing convolution processing on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit can be implemented in the following manner: the k-th residual unit of the plurality of cascaded residual units executes the following processing, that is, performing dilated convolution processing on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. That is, the dilated convolution operator included in the first residual unit performs dilated convolution processing on the residual feature to obtain the dilated convolution result of the first residual unit. The j-th residual unit of the plurality of cascaded residual units executes the following processing, that is, the dilated convolution operator included in the j-th residual unit performs dilated convolution processing on the residual result output by the (j-1)-th residual unit to obtain the dilated convolution result of the j-th residual unit. Here, j is a positive integer that increases sequentially, 1<j≦J, and J is the number of residual units.

[0208] Continuing from the above embodiment, each residual unit includes not only an extended convolution operator but also at least one causal convolution operator. After performing a convolution operation on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit, a causal convolution operation is performed on the obtained extended convolution result using at least one causal convolution operator included in the k-th residual unit, and the obtained causal convolution result is taken as the convolution result of the k-th residual unit. That is, a causal convolution operation is performed on the extended convolution result of the first residual unit using at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is taken as the convolution result output from the first residual unit. An extended convolution operation is performed on the residual result output by the (j-1)-th residual unit using the extended convolution operator contained in the j-th residual unit to obtain the extended convolution result of the j-th residual unit. Then, a causal convolution operation is performed on the extended convolution result of the j-th residual unit using at least one causal convolution operator contained in the j-th residual unit, and the causal convolution result of the j-th residual unit is taken as the convolution result of the j-th residual unit.

[0209] In some embodiments, when applying grouped convolution to an extended convolution operator included in a residual unit, extending convolution to low-frequency features can be achieved by the following method: grouping is performed on the input channels of the residual features to obtain multiple groups, where each group includes a first element corresponding to at least two channels in the residual features, and extending convolution is performed on the first element within each group. When applying grouped convolution to a causal convolution operator included in a residual unit, causal convolution can be achieved by the following method: grouping is performed on the input channels of the extended convolution result to obtain multiple groups, where each group includes a second element corresponding to at least two channels in the extended convolution result, and causal convolution is performed on the second element within each group.

[0210] In some embodiments, the fourth neural network for audio decoding includes a plurality of cascaded decoding blocks, each decoding block including a feature decoding block and at least one residual unit. Step 203111 can be implemented by step 203111A, and step 203112 can be implemented by step 203112A. In step 203111A, the feature decoding block in the plurality of cascaded decoding blocks performs feature decoding on the low-frequency features to obtain residual features. Correspondingly, in step 203112A, the residual unit in the plurality of cascaded decoding blocks performs a single residual operation on the residual features to obtain an estimate of the low-frequency coded features.

[0211] In some embodiments, step 203111A may be implemented in the following manner: performing feature decoding processing on a low-frequency feature by a feature decoding block in the first decoding block among the plurality of cascaded decoding blocks, outputting the decoding result output by the feature decoding block in the first decoding block to at least one residual unit in the first decoding block, performing feature decoding processing on the residual result output by at least one residual unit in the (i-1)-th decoding block by a feature decoding block in the i-th decoding block among the plurality of cascaded decoding blocks, outputting the decoding result output by the feature decoding block in the i-th decoding block to at least one residual unit in the i-th decoding block, and taking the decoding result output by the feature decoding block in the last decoding block as a residual feature. Here, i is a positive integer that increases sequentially, 1<i≦I, and I is the number of decoding blocks. Here, the decoding result output by the feature decoding block in the first decoding block is obtained by the feature decoding block in the first decoding block among the plurality of cascaded decoding blocks performing the following processing, that is, performing convolution processing on the low-frequency feature to obtain a convolved feature, where the number of channels of the convolved feature is less than the number of channels of the low-frequency feature, and performing upsampling processing on the convolved feature to obtain the decoding result output by the feature decoding block in the first decoding block. Here, the decoding result output by the feature decoding block in the i-th decoding block is obtained by the feature decoding block in the i-th decoding block performing the following processing, that is, performing convolution processing on the residual result output by at least one residual unit in the (i-1)-th decoding block to obtain a convolved feature, where the number of channels of the convolved feature is less than the number of channels of the residual result output by the at least one residual unit, and performing upsampling processing on the convolved feature to obtain the decoding result output by the feature decoding block in the i-th decoding block.

[0212] As an example, the fourth neural network includes four decoding blocks, and each decoding block includes four or five residual units.

[0213] In some embodiments, step 203112A may be implemented in the following manner: that is, at least one residual unit in the last decoding block among the plurality of cascaded decoding blocks performs one residual processing on the residual feature to obtain a low-frequency coded feature estimated value.

[0214] In step 2032, high-frequency reconstruction is performed on the high-frequency feature estimated value included in the coded feature estimated value, so as to obtain a high-frequency subband signal estimated value corresponding to the audio bitstream.

[0215] It should be noted that the feature reconstruction on the decoding side is the inverse process of the extraction process on the encoding side, and the high-frequency reconstruction on the decoding side is the inverse process of the high-frequency analysis on the encoding side.

[0216] In some embodiments, performing high-frequency reconstruction on the high-frequency feature estimated value included in the coded feature to obtain the high-frequency subband signal estimated value corresponding to the audio bitstream may be implemented in the following manner: that is, the sixth neural network is invoked to perform feature reconstruction on the high-frequency feature estimated value included in the coded feature, so as to obtain the high-frequency subband signal estimated value corresponding to the audio bitstream. Here, the structure of the sixth neural network is similar to that of the third neural network.

[0217] In some embodiments, step 2032 may be implemented through step 20321 to step 20324.

[0218] In step 20321, frequency domain transformation is performed on the first half sample points and the second half sample points included in the low-frequency subband signal estimated value, so as to obtain a first transform coefficient corresponding to the first half sample points and a second transform coefficient corresponding to the second half sample points.

[0219] It should be noted that the frequency domain transformation method in the embodiments of the present application includes Modified Discrete Cosine Transform (MDCT), Discrete Cosine Transform (DCT), Fast Fourier Transform (FFT), etc., and the embodiments of the present application do not limit the mode of frequency domain transformation.

[0220] In step 20322, inverse processing of frequency band extension is performed on the high-frequency feature estimated value based on the first transform coefficient, to obtain a first high-frequency subband signal estimated value.

[0221] In some embodiments, step 20322 can be implemented in the following manner: that is, spectrum copying processing is performed on the transform coefficients in the latter half of the first transform coefficient to obtain the first reference transform coefficient of the reference high-frequency subband signal, gain processing is performed on the first reference transform coefficient based on the subband spectrum envelope in the first half corresponding to the high-frequency feature estimated value to obtain the gained first reference transform coefficient, and inverse frequency domain transformation is performed on the gained first reference transform coefficient to obtain the first high-frequency subband signal estimated value.

[0222] In some embodiments, obtaining a gained first reference transformation coefficient by applying a gain to the first reference transformation coefficient based on the subband spectral envelope of the first half corresponding to the high-frequency feature estimate includes: dividing the first reference transformation coefficient of the reference high-frequency subband signal into multiple first subbands based on the subband spectral envelope of the first half; determining the first average energy corresponding to the first subband in the subband spectral envelope of the first half for any of the multiple first subbands; determining the second average energy corresponding to the first subband; determining a gain factor based on the ratio of the first average energy to the second average energy; and multiplying each first reference transformation coefficient included in the first subband by the gain factor to obtain a gained first reference transformation coefficient.

[0223] For example, the estimated low-frequency subband signal generated on the decoding side.

number

[0224] Next, for the MDCT coefficients of the first set of 160 points (i.e., the first transformation coefficients corresponding to the sample points in the first half),

number

[0225] Next, the eight subband spectral envelopes obtained previously (i.e., the eight subband spectral envelopes obtained by querying the quantization table, i.e., the subband spectral envelopes corresponding to the high-frequency features) are recalled, and these eight subband spectral envelopes correspond to eight high-frequency subbands, and the reference values ​​of the MDCT coefficients of the 160 generated reference high-frequency subband signals are divided into eight reference high-frequency subbands (i.e., the first reference transformation coefficients of the reference high-frequency subband signals are divided into multiple first subbands), and for each band, one high-frequency subband Based on the corresponding reference high-frequency subband, a gain is applied (multiplied in the frequency domain) to the reference values ​​of the MDCT coefficients of the generated 320-point reference high-frequency subband signal. For example, a gain factor is calculated based on the average energy of the high-frequency subband (i.e., the first average energy) and the average energy of the corresponding reference high-frequency subband (the second average energy). By multiplying the MDCT coefficient corresponding to each point in the corresponding reference high-frequency subband by this gain factor, it is ensured that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to the original coefficient energy on the encoding side.

[0226] For example, assuming that the average energy of the reference high-frequency subband (i.e., the first subband divided by the reference values ​​of the MDCT coefficients for 160 points of the generated high-frequency portion of the signal) is Y_L, and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectral envelope decoded based on the bitstream) is Y_H, then the gain factor a = sqrt(Y_H / Y_L) is calculated, where sqrt() represents the square root calculation function and is used to calculate the square root of (Y_H / Y_L). After obtaining the gain factor a, the MDCT coefficients for each point in the reference high-frequency subband are directly multiplied by a. The average energy of the gained MDCT coefficients (virtually generated) will be very close to those of the original encoded side.

[0227] Finally, the inverse MDCT transform is called to generate an estimate of the first subframe of the high-frequency subband signal (i.e., the first high-frequency subband signal estimate).

[0228] In step 20323, based on the second transformation coefficient, the high-frequency feature estimate is subjected to the inverse process of frequency bandwidth expansion to obtain the second high-frequency subband signal estimate.

[0229] Here, step 20323 is similar to step 20322, but differs in that the processing target in step 20323 is the second conversion coefficient, while the processing target in step 20322 is the first conversion coefficient.

[0230] In some embodiments, step 20323 can be implemented in the following manner: spectral copying is performed on the latter half of the second transformation coefficient to obtain the second reference transformation coefficient of the reference high-frequency subband signal; gain is applied to the second reference transformation coefficient based on the latter half of the subband spectral envelope corresponding to the high-frequency feature estimate to obtain the gained second reference transformation coefficient; and inverse frequency domain transformation is performed on the gained second reference transformation coefficient to obtain the second high-frequency subband signal estimate.

[0231] In some embodiments, obtaining a gained second reference transformation coefficient by applying a gain to the second reference transformation coefficient based on the latter half of the subband spectral envelope corresponding to the high-frequency feature estimate includes: dividing the second reference transformation coefficient of the reference high-frequency subband signal into multiple second subbands based on the latter half of the subband spectral envelope; determining the first average energy corresponding to the second subband in the latter half of the subband spectral envelope for any of the multiple second subbands; determining the second average energy corresponding to the second subband; determining a gain factor based on the ratio of the first average energy to the second average energy; and multiplying each second reference transformation coefficient included in the second subband by the gain factor to obtain a gained second reference transformation coefficient.

[0232] In step 20324, the first high-frequency subband signal estimate and the second high-frequency subband signal estimate are combined to obtain a high-frequency subband signal estimate corresponding to the audio bitstream.

[0233] In step 2033, subband synthesis is performed on the estimated low-frequency subband signal and the estimated high-frequency subband signal to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0234] For example, subband synthesis is the reverse process of subband decomposition. On the decoding side, subband synthesis is performed on the estimated low-frequency subband signals and the estimated high-frequency subband signals to reconstruct the reconstructed audio signal, where the reconstructed audio signal is the reconstructed signal.

[0235] In some embodiments, performing subband synthesis on the estimated low-frequency subband signal and the estimated high-frequency subband signal to obtain a reconstructed audio signal includes performing upsampling processing on the estimated low-frequency subband signal to obtain a low-pass filtered signal, performing upsampling processing on the estimated high-frequency subband signal to obtain a high-frequency filtered signal, and performing filtering synthesis processing on the low-pass filtered signal and the high-frequency filtered signal to obtain the reconstructed audio signal.

[0236] For example, after acquiring the estimated low-frequency subband signal and the estimated high-frequency subband signal, subband synthesis is performed on the estimated low-frequency subband signal and the estimated high-frequency subband signal by a QMF synthesis filter to restore the reconstructed audio signal.

[0237] In some embodiments, the frame header further includes at least 1 bit of audio channel bit, which is used to indicate that audio decoding is performed on the audio bitstream using a monaural decoding method or a stereo decoding method. When the audio channel bit in the frame header included in the audio bitstream package is a first value, it indicates that the stereo decoding method needs to be used to decode the audio bitstream, that is, a synthesized stereo signal is reconstructed from the decoded reconstructed audio signal and the stereo feature vector in the audio bitstream package. When the audio channel bit in the frame header included in the audio bitstream package is a second value, it indicates that the monaural decoding method needs to be used to perform audio decoding on the audio bitstream, that is, the decoded reconstructed audio signal is taken as a monaural reconstructed audio signal.

[0238] Hereinafter, exemplary applications of the embodiments of the present application in actual application scenarios will be described.

[0239] The embodiments of this application can be applied to various audio scenarios, such as voice calls and instant messaging. The following description will use voice calls as an example.

[0240] In related technologies, the principle of speech coding is roughly as follows: Speech coding can be performed directly on a sample-by-sample basis on a speech waveform sample, or, based on the principles of human speech, relevant low-dimensional features are extracted, the features are coded on the coding side, and the speech signal is reconstructed on the decoding side based on these parameters.

[0241] The encoding principles described above all originate from audio signal modeling, i.e., compression methods based on signal processing. To guarantee audio encoding efficiency and improve encoding quality compared to compression methods based on signal processing, the embodiments of this application provide a multimode multirate audio encoding method (i.e., an audio encoding method and an audio decoding method). Based on the characteristics of the audio signal, important parts (low-frequency subband signals) are processed using neural network (NN) technology to obtain feature vectors with lower dimensions than the input low-frequency subband signals. Here, an operation similar to "blocking" is employed within the neural network to reduce the complexity of the algorithm and improve the encoding effect. For low-frequency feature vectors (i.e., the aforementioned low-frequency features), quantization and encoding can be performed using codebooks with different quantization accuracies based on the same low-frequency feature vector, thereby achieving multirate encoding and decoding effects. For high-frequency feature vectors (i.e., the aforementioned high-frequency features), multilevel quantization and encoding are used to achieve multirate encoding and decoding effects.

[0242] The embodiment of this application can be applied to the voice communication link shown in Figure 6C. Using a Voice over Internet Protocol (VoIP) conferencing system as an example, the voice encoding and decoding technology according to the embodiment of this application is arranged in an encoding unit and a decoding unit to solve the basic function of voice compression. The encoder is located in the uplink client 601, and the decoder is located in the downlink client 602. The uplink client collects the voice, performs processing such as pre-processing enhancement and encoding, transmits the bitstream obtained by encoding to the downlink client 602 via the network, performs processing such as decoding and enhancement in the downlink client 602, and plays back the decoded voice in the downlink client 602.

[0243] Considering forward compatibility (i.e., compatibility between the new encoder and existing encoders), it is necessary to place the transcoder in the system's backend (i.e., server) to resolve the interoperability issues between the new encoder and existing encoders. For example, if the transmitting side (uplink client) is a new NN encoder and the receiving side (downlink client) is a Public Switched Telephone Network (PSTN) (G.722), the backend needs to generate the audio signal using an NN decoder, then call the G.722 encoder to generate a specific bitstream and implement code conversion functionality, thereby allowing the receiving side to correctly decode based on that specific bitstream.

[0244] The following describes the audio encoding method and audio decoding method provided in the embodiment of this application, which combine the high-frequency and low-frequency portions within a monaural signal.

[0245] The audio encoding method and audio decoding method provided in the embodiment of this application will be described below with reference to Figure 7.

[0246] The following process is performed on the encoding side.

[0247] For the input audio signal x(n) of the nth frame, an analysis filter is used to obtain the low-frequency subband signal x LB (n) and high-frequency subband signal x HB Decompose it into (n).

[0248] Low-frequency subband signal x LB For (n), call the second NN and the low-dimensional feature vector F LB (n), i.e., low-frequency features are obtained. Feature vector F LB The dimension of (n) is smaller than the dimension of the low-frequency subband signal in order to reduce the amount of data. For example, each frame x LB For (n), call the neural network (encoding part) and create a lower-dimensional feature vector F LB (n) is generated. The embodiments of this application do not restrict other NN structures, such as autoencoders, full-connection (FC) networks, long short-term memory (LSTM) networks, convolutional neural networks (CNN) + LSTM, etc. Here, an operation similar to "blocking" is employed within the neural network, which can reduce the complexity of the algorithm and improve the encoding effect. F LB For (n), quantization and coding are performed using codebooks with different quantization accuracies to achieve multirate coding and decoding effects.

[0249] High-frequency subband signal x HB For (n), considering that the importance of high-frequency quality is less important than that of low frequencies, the high-frequency subband signal x HB (n) The feature vector F is obtained using other solutions. HB(n) can be extracted. For example, frequency band extension techniques based on audio signal analysis can generate high-frequency subband signals using only a small number of bits. The same NN structure as the low-frequency subband signal, or a simpler network (e.g., the output feature vector is the low-frequency feature vector F) can be extracted. LB You can also use (smaller than n). High-frequency feature vector F HB For (n), multi-level quantization and coding are used to achieve multi-rate coding and decoding effects.

[0250] Feature vector corresponding to the subband signal (F LB (n) and F HB Vector quantization or scalar quantization is performed on (n), entropy coding is performed on the quantized value after quantization, and the bitstream obtained by coding (low-frequency bitstream and high-frequency bitstream) is transmitted to the decoding side.

[0251] The following process is performed on the decryption side.

[0252] The bitstream received by the decoding side (including low-frequency and high-frequency bitstreams) is decoded, and the estimated low-frequency feature vectors are obtained.

number

number

[0253] For the low-frequency portion, estimates of low-frequency feature vectors.

number

number

number

number

[0254] For the high-frequency portion, estimates of the high-frequency feature vectors.

number

number

number

number

[0255] Finally, the QMF synthesis filter is called, and the reconstructed synthesized speech signal

number

[0256] The following describes in detail the audio encoding method and audio decoding method provided in the embodiments of this application.

[0257] In some embodiments, an audio signal with a sampling rate Fs = 32000 Hz is used as an example (it should be noted that the methods provided in the embodiments of this application also apply to other sampling rate scenarios, including but not limited to 8000 Hz, 32000 Hz, 48000 Hz, etc.). Simultaneously, assuming a frame length of 20 ms, then, for Fs = 32000 Hz, each frame corresponds to containing 640 sample points.

[0258] The encoding and decoding sides will be explained in detail below, referring to the flowchart shown in Figure 7.

[0259] The encoding process for the low-frequency and high-frequency portions of a monaural signal is as follows:

[0260] For an audio signal (monaural signal) with a sampling rate Fs = 32000 Hz, the input signal of the nth frame contains 640 sample points and is denoted as input signal x(n), i.e., a monaural input signal.

[0261] In step 11, the QMF analysis filter is called to perform signal decomposition.

[0262] By calling the QMF analysis filter (2-channel QMF) and performing downsampling, the low-frequency subband signal x LB (n) and high-frequency subband signal x HB Two subband signals, (n) and (n), can be obtained. Low frequency subband signal x LB (n) and high-frequency subband signal x HB The effective bandwidths of (n) are 0-8kHz and 8-16kHz, respectively, and the low-frequency subband signal x LB (n) and high-frequency subband signal x HB The sample size for (n) is 320.

[0263] In step 12, the second neural network (NN) is invoked based on the low-frequency subband signal.

[0264] Low-frequency subband signal x LB Based on (n), we call a second NN and a lower-dimensional feature vector F LB (n) is generated. What needs to be explained is x LB The dimension of (n) is 320, and F LB The dimension of (n) is 56. In terms of data volume, the second NN plays the role of "dimensionality reduction" and realizes the function of data compression. Here, the embodiment of this application is F LB Without limiting the dimension of (n), x LB It may be any other dimension smaller than (n).

[0265] Referring to the network structure diagram of the second neural network shown in Figure 11, the process by which the second neural network performs data compression will be explained in detail below.

[0266] First, we can call a 16-channel causal convolution to extend the input tensor (i.e., a vector) to a 16x320 tensor.

[0267] Next, preprocessing is performed on the 16x320 tensor. For example, after performing one convolution operation on the 16x320 tensor, a pooling operation with a factor of 2 is performed, and the activation function may be PReLU, generating a 16x160 tensor.

[0268] Next, four coding blocks with different downsampling factors (Down_factor) are cascaded. Here, each coding block contains one residual block, one convolutional layer, and one pooling layer. Here, each residual block contains five residual units based on extended convolution (the input and output feature dimensions of the residual units remain unchanged). One convolutional layer is used to double the number of input channels, and the activation function may be PReLU, thereby guaranteeing the amount of data and avoiding data loss. The pooling layer is a pooling operation that includes a Down_factor to perform downsampling and achieve data compression. Here, the Down_factors of the four coding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four coding blocks are set to 32, 64, 128, and 256, respectively. After processing by four coding blocks, the input 16×160 tensor is transformed into tensors of 32×80, 64×20, 128×5, and 256×1, respectively. Herein, the embodiments of this application do not limit the number of coding blocks, which can be any positive integer such as 2, 3, 4, or 5. Similarly, the embodiments of this application do not limit the number of residual units within a coding block, which can be any positive integer such as 2, 3, 4, 5, or 6. The number of residual units in multiple coding blocks may be the same or different; for example, one coding block may contain four residual units, while another coding block may contain five residual units.

[0269] Here, we will further explain residual units. A residual unit is a module within a deep neural network that introduces cross-layer connections to the neural network, making it easier to optimize the neural network during training and avoiding problems such as gradient vanishing or gradient explosion. Its core idea is to perform residual learning on the input within the module, that is, to pass input information directly to the output by bypassing some layers through a direct path, allowing the network to better utilize shallow feature information during the learning process. Figure 12A is a schematic diagram of the residual block structure used in the coding block within the 3rd NN. This residual block contains four residual units based on augmented convolutions, each residual unit containing an augmented convolution block with a specified dilation rate, that is, each augmented convolution block contains a convolution operator with a specified dilation rate (e.g., Dilation rate = 3). In the embodiments of this application, five progressively expanding convolutional blocks are used, which corresponds to extracting input features at different resolutions using different receptive fields, allowing for a more comprehensive analysis of the data. After residual processing of the five specified expanding convolutional blocks, the inputs from the skip connections are added together to obtain the output of the residual block, and the output is output to a convolutional layer connected to the residual block.

[0270] Herein, we will further describe an arbitrary residual unit in Figure 12A, and as shown in Figure 12B, any one residual unit contains at least one expansion convolution of a specified expansion rate (used for receptive field expansion), and PReLU can be used as the activation function. Alternatively, one or more causal convolutions (used for local information extraction) can be cascaded, and PReLU can be used as the activation function. The convolution kernel size of the expansion convolution of the specified expansion rate described above may be 3, 5, 7, 9, etc., and the convolution kernel size of the causal convolution described above may be 1, 3, etc. The embodiments of this application do not limit the convolution kernel size of the expansion convolution or causal convolution of the specified expansion rate described above. Furthermore, the causal convolution or expansion convolution in the embodiments of this application may be carried out by other convolution units having similar or equivalent functions.

[0271] Furthermore, for the residual unit, a grouped convolution algorithm is introduced to reduce the complexity of the algorithm. Grouped convolution involves dividing the input channels into multiple groups and performing a convolution operation, where input channels and output channels are associated only within each group. Here, we assume that there are 16 input channels and 32 output channels. If the number of groups is 1, each input channel is associated with one of the 32 output channels. If the number of groups is 2, the 16 input channels are first divided into two groups, 0-7 and 8-15. Within each of the two groups, the input channels are associated with the output channels within that group. For example, input channels 0-7 in the first group are associated with output channels 0-15, and input channels 8-15 in the second group are associated with output channels 16-31. For example, output channel 0 is associated only with input channels 0-7 and not with input channels 8-15. Output channel 25 is associated only with input channels 8-15 and not with input channels 0-7. As this comparison shows, introducing grouped convolution can avoid any input channel being associated with all output channels, thereby reducing the number of connections and complexity. Of course, a larger number of groups does not necessarily mean better, as it reduces the degree of association between input and output channels and affects the coding effect. In the embodiments of this application, the extended convolution contained in the five residual blocks corresponding to the four coding blocks can use different grouping number configurations, and the specific grouping number configurations are shown in Table 1.

[0272] [Table 1]

[0273] Finally, for a 256x1 tensor, we perform a causal convolution similar to the preprocessing to obtain a 56-dimensional feature vector F. LB(n) can be output. The calculation of the second NN yields a 56-dimensional feature vector F LB Each value of (n) lies between [-1, 1].

[0274] In step 13, the high-frequency subband signal x HB High-frequency analysis is performed on (n).

[0275] The purpose of high-frequency analysis is to analyze the high-frequency subband signal x HB Extract important information from (n) and create a lower-dimensional feature vector F HB The objective is to generate (n). Here, the embodiment of this application is F HB Without limiting the dimension of (n), x LB It can be any other dimension smaller than (n), but F HB The dimension of (n) is F LB It should be smaller than the dimension of (n).

[0276] In some embodiments, referring to step 12, another structure similar to the second NN can be introduced to generate low-dimensional feature vectors. Since high-frequency subband signals are of relatively lower importance to quality compared to low-frequency subband signals, the NN structure for high-frequency subband signals does not need to be as complex as that of the second NN. The NN structure for high-frequency subband signals is similar to that of the second NN, but the number of channels in the NN structure for high-frequency subband signals is significantly reduced compared to the structure of the second NN.

[0277] The embodiments of this application propose an alternative method for compressing high-frequency subband signals, namely frequency bandwidth extension (recovering a wideband audio signal from a narrowband audio signal with a limited frequency bandwidth). The application of frequency bandwidth extension in the embodiments of this application will be described in detail below.

[0278] High-frequency subband signal x containing 320 points (i.e., sample points) HB For (n), the high-frequency subband signal x of 320 points HB(n) is divided into two subframes (the first subframe and the second subframe), each containing 160 points. By dividing the frame and using a 320-point MDCT transformation with 50% overlap, compared to a 640-point MDCT transformation with 50% overlap, the algorithmic delay can be reduced by 10ms. Therefore, for every 20ms frame, two 320-point MDCT transformations can be performed to extract features.

[0279] For any subframe, i.e., a high-frequency subband signal containing 160 points, a Modified Discrete Cosine Transform (MDCT) is called to generate MDCT coefficients for the 160 points. Specifically, in the case of a 50% overlap, the first subframe of the (n+1)th frame can be merged with the second subframe of the (n)th frame to calculate the MDCT for 320 points, obtaining the MDCT coefficients for the 160 points of the first subframe of the (n+1)th frame. The second subframe of the (n+1)th frame can then be merged with the first subframe of the (n+1)th frame to calculate the MDCT for 320 points, obtaining the MDCT coefficients for the 160 points of the second subframe of the (n+1)th frame.

[0280] For the MDCT coefficients of 160 points in either of the two subframes mentioned above, the same method is used to perform the following processing: the MDCT coefficients of the 160 points are divided into N subbands, where each subband forms a group of multiple adjacent MDCT coefficients, and the MDCT coefficients of the 160 points can be divided into four subbands. For example, the 160 points can be evenly distributed, i.e., the number of points in each subband is the same. Of course, embodiments of this application may also distribute the 160 points unevenly. For example, fewer MDCT coefficients (higher frequency resolution) may be included in lower frequency subbands, and more MDCT coefficients (lower frequency resolution) may be included in higher frequency subbands.

[0281] According to the Nyquist sampling theorem (which states that to reconstruct the original signal from a sampled signal without distortion, the sampling frequency must be greater than twice the highest frequency of the original signal; if the sampling frequency is less than twice the highest frequency of the spectrum, aliasing occurs in the signal spectrum; and if the sampling frequency is greater than twice the highest frequency of the spectrum, aliasing does not occur in the signal spectrum), the MDCT coefficients of the aforementioned 320 points represent the spectrum from 8 to 16 kHz. However, in ultra-wideband voice communication, a spectrum up to 16 kHz is not necessarily required. For example, if the spectrum is set to 14 kHz, only the MDCT coefficients of the previous 240 points need to be considered, and accordingly, the number of subbands can be controlled to 6.

[0282] For each subband, the average energy of all MDCT coefficients within the current subband is calculated and used as the subband spectral envelope (the spectral envelope is a smooth curve that passes through each major peak point of the spectrum). For example, if the MDCT coefficients included in the current subband are x(n) and n=1, 2, ..., 40, then the average energy Y=((x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40). When the MDCT coefficients of 160 points are divided into four subbands, four subband spectral envelopes can be obtained, and these four subband spectral envelopes correspond to the subbands.

number

number

[0283] In summary, using either of the two methods described above (NN structure and frequency band extension), it is possible to output 160-dimensional high-frequency subband signals as 4-dimensional feature vectors. Therefore, high-frequency information can be represented with a minimum of 8-dimensional data for each 20ms frame, significantly improving coding efficiency.

[0284] Step 14 involves quantization and encoding.

[0285] Feature vector F of low-frequency subband signal LB (n) and the feature vector F of the high-frequency subband signal HB For (n), both scalar quantization (quantizing each component individually) and entropy coding methods can be performed. Furthermore, the embodiments of this application are not limited to a combination of vector quantization (combining multiple adjacent components into a single vector and quantizing them in association) and entropy coding techniques.

[0286] Here, the low-frequency feature vector F LB The quantization and coding of (n) will be explained. As mentioned above, after processing the low-frequency subband signal with the second NN, a 56-dimensional feature vector F LB (n) is obtained. Embodiments of this application provide a method based on scalar quantization and entropy coding, and include the following 1) to 3): 1) F LB (n) In each dimension, the interval [-1,1] is divided into 11 equal parts to form a codebook containing 11 elements, and the value of each dimension is quantized to one of the 11 elements. 2) According to Shannon's entropy theorem, in the case of a uniformly distributed codebook containing 11 elements, the entropy (average bits) is:

number

[0287] Also, the low-frequency feature vector F LB To achieve the multirate coding and decoding target of (n), the following implementations are possible.

[0288] The codebook for the above implementation, that is, the codebook containing 11 elements in each dimension, is named Low Frequency Codebook-1, and its reference bitrate is 9.69 kbps.

[0289] Similarly, for each dimension, the interval [-1,1] is divided into 9 equal parts, the entropy is 3.17, and the average bitrate is 3.17 * 56 * 50 / 1000 = 8.88 kbps. This is named Low Frequency Codebook-2, and its reference bitrate is 8.88 kbps. Thus, at least two bitrate schemes exist, with different bitrates corresponding to different qualities, enabling multirate coding.

[0290] By analogy, for each dimension, we divide the interval [-1,1] into 7 equal parts, resulting in an average bitrate of 7.86kbps, which we name Low Frequency Codebook-3. We also divide the interval [-1,1] into 5 equal parts, resulting in an average bitrate of 6.50kbps, which we name Low Frequency Codebook-4.

[0291] Thus, for the original 56-dimensional feature vector obtained by data calculation for each frame, a low-frequency feature encoding mode of at least 4 bits per minute is realized by a quantization precision configuration. It should be noted that the embodiments of this application do not limit other multi-bitrate construction schemes or the number of codebooks.

[0292] Feature vector F of the 8-dimensional high-frequency subband signal for each frame (including two subframes) HB For (n), the specific implementation of its quantization and coding includes the following 1) to 3): 1) Scalar quantization is performed on each dimension alone, allocating 5 bits, and in this way, 40 bits are allocated to each frame. 2) For a frame division scheme every 20ms, there are 50 frames per second, so the average bitrate is 2kbps. Entropy coding (including but not limited to Huffman coding or range coding) is performed on the quantized values ​​of each dimension above, and the overall bitrate is less than 2kbps. The above codebook is defined as the high-frequency codebook-1, and the quantized values ​​are defined as high-frequency feature vector quantized values ​​(i.e., quantized values ​​of high-frequency features).

[0293] Also, the high-frequency feature vector F HB To achieve the multirate coding and decoding target of (n), residual processing can be further introduced, and the following implementations are possible.

[0294] 1) For each 10ms subframe, the four subbands of the subframe are further divided equally into two subsets.

[0295] For example, the set of envelope values ​​for four subbands is

number

[0296] 2) For each subset of each subband, calculate the envelope value for that subset.

[0297] For example, the set of envelope values ​​for all subsets is

number

[0298] 3) The first high-frequency feature vector residual (abbreviated as the first residual value) is obtained by subtracting the corresponding high-frequency feature vector quantization value from the envelope value of each subset.

[0299] For example, high-frequency feature vector quantization values

number

number

[0300] 4) The quantized value of the first high-frequency feature vector residual (abbreviated as the quantized value of the first residual) is obtained from the first high-frequency feature residual by similar quantization and entropy coding.

[0301] Here, considering that the dynamic range of the residual values ​​is much lower than that of the original values, it is therefore sufficient to allocate only 2 bits when performing scalar quantization on each residual. Thus, to obtain the quantized value of the first high-frequency feature vector residual, a reference bitrate of 2 × 16 × 50 = 1.6 kbps is required (similarly, since it is a non-uniformly distributed entropy encoding, the actual bitrate is generally less than 1.6 kbps).

[0302] 5) From the envelope value of each subset, subtract the sum of the quantized value of the corresponding high-frequency feature vector and the quantized value of the first high-frequency feature vector residual to obtain the second high-frequency feature vector residual (abbreviated as the second residual value).

[0303] Here, similarly, the quantization and entropy coding of the second high-frequency feature vector residual can be performed using a reference bitrate of 1.6 kbps.

[0304] By analogy, as more residuals are calculated, it can be guaranteed that the high-frequency feature vector reconstructed on the decoding side will approximate the original high-frequency feature vector infinitely. Furthermore, the effects of multirate coding are realized based on the presence or absence of high-frequency feature vector residuals, or the number of high-frequency feature vector residuals. The embodiments of this application can define the number of residual coding types (no residual coding, first high-frequency feature vector residual, second high-frequency feature vector residual, third high-frequency feature vector residual, etc.) according to the bitrate requirements, and are not limited thereto.

[0305] Furthermore, embodiments of this application provide a solution for additionally extracting side information on the encoding side, thereby leading to more detailed processing on the decoding side and improving audio quality. In principle, the main function of frequency band expansion is to copy the low-frequency spectrum to the high-frequency spectrum and then gain it. However, the high-frequency spectrum is flatter and has fewer harmonic wave components than the low-frequency spectrum. Direct copying would result in the artificially generated high frequencies containing excessive harmonic wave components. Therefore, embodiments of this application propose to perform specific side information estimation on the encoding side and write it to the bitstream. The decoding side determines whether additional processing is necessary on the decoding side based on the flatness side information described above. It should be noted that the flatness side information described above indicates only on the decoding side whether the current frame requires additional processing. In special cases, the entire audio does not require additional processing.

[0306] The above side information estimation is implemented on the encoding side as follows:

[0307] 1) For each 10ms subframe, divide the MDCT coefficients of 160 points into two blocks. They may be divided equally, or the MDCT coefficients of the first two subbands may be used as the first block, and the MDCT coefficients of the last two subbands as the second block.

[0308] 2) For each block, calculate the square of each MDCT coefficient, i.e.,

number

[0309] 3) The arithmetic mean of all MDCT coefficients in the current block

number

[0310] 4) Geometric mean of all MDCT coefficients in the current block

number

[0311] 5) The first degree of flatness of the current block

number

number

[0312] 6) Similarly, the second flatness of the MDCT coefficient for a specified low-frequency band.

number

[0313] 7)

number

number

[0314] Here, if Sfp_FLAG=1, it means that additional operations are required on the decoding side to avoid the artificial generation of excessive harmonic waves at high frequencies. Following the above operations, each frame requires an additional 4 bits to represent the flatness side information of two blocks within two 10ms subframes. The embodiments of this application are not limited to the above extraction method, and the above flatness side information may also be extracted by other analysis methods to guide subsequent decoding operations on the decoding side.

[0315] After quantization and encoding, a bitstream can be generated. Experiments have shown that high-quality compression can be achieved for 16kHz wideband signals in the 5-10kbps range, and high-quality compression can be achieved for 32kHz ultrawideband signals in the 8-15kbps range.

[0316] Furthermore, in order to achieve the aforementioned goal of multimode multirate encoding and decoding, it is necessary to appropriately design the bitstream structure. As shown in Figures 13A to 13E, the implementation of the bitstream structure is as follows.

[0317] Each frame bitstream typically includes an 8-bit frame header. This 8-bit frame header includes at least one encoding mode bit, which indicates whether broadband encoding (value of the encoding mode bit is 0) or ultra-broadband encoding (value of the encoding mode bit is 1) is used. That is, there are two encoding modes: broadband and ultra-broadband. It also includes two encoding rate bits, which indicate which bitrate mode is used for the corresponding broadband or ultra-broadband encoding mode. It should be noted that the embodiments of this application do not limit the positions of the one encoding mode bit and the two encoding rate bits in the frame header. For example, the one encoding mode bit may be located in the first bit of the frame header, and the two encoding rate bits may be located in the last two bits of the frame header.

[0318] As shown in Figure 13A, in broadband coding mode, a 56-dimensional low-frequency feature vector index is placed following the frame header. Since it is entropy coding, the coding lengths are not the same. As previously mentioned, based on the 2-bit bitrate mode bits, the coded low-frequency bitstream is quantized using different low-frequency codebooks and stored in the portion of the 56-dimensional low-frequency feature vector index.

[0319] In broadband coding mode, the 2-bit coding rate bits are interpreted as follows:

[0320] 1) 00 indicates the use of the low-frequency codebook-4, with a reference average bitrate of 6.5 kbps, which is the lowest bitrate in broadband coding mode.

[0321] 2)01 indicates that the low-frequency codebook-3 is used and the reference average bitrate is 7.86kbps.

[0322] 3) 10 indicates that the low-frequency codebook-2 is used and the reference average bitrate is 8.88 kbps.

[0323] 4)11 indicates that the low-frequency codebook-1 is used and the reference average bitrate is 9.69 kbps, which is the highest bitrate in broadband coding mode.

[0324] For ultra-wideband coding mode, two bitrate mode bits are defined based on the usage mode, whether or not residual coding is performed. The two coding rate bits in ultra-wideband coding mode are interpreted as follows:

[0325] 1)00 (without residual coding) indicates that the low-frequency codebook-4 is used for the low-frequency portion, the bitstream package structure is encoded and quantized using the low-frequency codebook-4, the four spectral envelopes are ultra-wideband coded, and 4 bits of flatness are included, with a reference average bitrate of 6.5 + 2 + 0.2 = 8.7 kbps.

[0326] 2)01 (without residual coding) indicates that the low-frequency codebook-1 is used for the low-frequency portion, the bitstream package structure is encoded and quantized using the low-frequency codebook-1, the four spectral envelopes are ultra-wideband coded, and 4 bits of flatness are included, and the reference average bitrate is 9.69 + 2 + 0.2 = 11.9 kbps.

[0327] 3) 10 (including first residual coding) indicates that the low-frequency portion uses low-frequency codebook-1, the bitstream package structure is coded and quantized using low-frequency codebook-1, the four spectral envelopes are ultra-wideband coded, there are eight first residuals, and four bits of flatness, and the reference average bitrate is 9.69 + 2 + 1.6 + 0.2 = 13.5 kbps.

[0328] 4)11 (including first and second residual coding) indicates that the low-frequency portion uses low-frequency codebook-1, the bitstream package structure is coded and quantized using low-frequency codebook-1, the four spectral envelopes are ultra-wideband coded, there are eight first residuals, eight second residuals, and four bits of flatness, and the reference average bitrate is 9.69 + 2 + 1.6 + 1.6 + 0.2 = 15.0 kbps.

[0329] As shown in Figures 13A to 13C, based on 2 bits of bitrate mode bits, the associated bitstreams of low-frequency and high-frequency feature vector indices follow immediately after the frame header. As shown in Figures 13D to 13E, if the bitstream package structure includes 4 bits of flatness side information, then 4 bits of side information can follow immediately after the 8-dimensional high-dimensional feature vector indices.

[0330] In summary, when performing audio encoding using an encoder, the user inputs the audio data to be encoded and precise configuration parameters into the encoder. For example, if the input audio data is broadband, the sampling rate is 16,000 Hz, and if the input audio data is ultra-broadband, the sampling rate is 32,000 Hz. For the sake of clarity, we will use a broadband signal sampled at 16,000 Hz and an ultra-broadband signal sampled at 32,000 Hz as examples, but the embodiments of this application are not limited to other combinations.

[0331] Furthermore, the user must set the configuration parameters for the bitrate mode to be used, and the encoder can select an appropriate codebook based on the set bitrate mode. Generally, the higher the bitrate mode, the more bits the encoder uses to encode, resulting in higher quality reconstructed audio. As described above, the embodiments of this application provide four bitrate modes for both broadband and ultra-broadband encoding modes, but the embodiments of this application are not limited to four bitrate modes.

[0332] For example, if the user inputs a broadband signal sampled based on 16000Hz, the encoder can encode it using broadband encoding mode. The second neural network (NN) is used directly on the input broadband signal to extract low-frequency features, and the extracted low-frequency features are quantized and encoded to obtain a bitstream of the broadband signal. If the user inputs an ultra-broadband signal sampled based on 32000Hz, the encoder can encode it using ultra-broadband encoding mode. The input ultra-broadband signal is first separated into low-frequency and high-frequency subband signals by an analysis filter, and the low-frequency subband signal is quantized and encoded using the second NN to extract low-frequency features F LB Extract (n), F LB Quantization and encoding are performed on (n) to obtain a low-frequency bitstream, and high-frequency analysis is performed on the high-frequency subband signal to obtain high-frequency features F HB Extract (n), F HB Quantization and encoding are performed on (n) to obtain a high-frequency bitstream. Furthermore, based on the bitrate mode input on the encoding side, the encoder can select a corresponding codebook and perform quantization and encoding.

[0333] The decoding process for the low-frequency and high-frequency portions is as follows:

[0334] Step 21 involves quantization and decoding.

[0335] Quantization and decoding are the reverse processes of quantization and coding. For a received bitstream (including high-frequency and low-frequency bitstreams), entropy decoding is first performed, and by querying the quantization table, an estimate of the feature vector of the low-frequency bitstream is obtained.

number

number

[0336] Referring to Figure 13E, we will explain the following using a configuration that is in ultra-wideband coding mode and has 10 bitrate mode bits as an example.

[0337] First, the frame header of the bitstream package is analyzed. If the encoding mode bit is 1, it means that ultra-wideband encoding was used during encoding. If the bitrate mode bit is 10, it means that the broadband portion was encoded using low-frequency codebook-1. After encoding the ultra-wideband portion, the bitstream package contains an 8-dimensional high-frequency feature vector index and a 16-dimensional first high-frequency feature vector residual index.

[0338] For the broadband portion, the low-frequency bitstream was analyzed using entropy coding and decoding techniques, and the estimated 56-dimensional low-frequency feature vector was obtained using the Low-Frequency Codebook-1.

number

[0339] For the ultra-wideband portion, first, an 8-dimensional high-frequency feature vector estimate is obtained for each frame based on a 5-bit codebook. Next, a 16-dimensional first high-frequency feature vector residual estimate is obtained for each frame based on a 2-bit codebook. The corresponding 8-dimensional high-frequency feature vector estimate (i.e., the aforementioned high-frequency feature estimate) and the 16-dimensional first high-frequency feature vector residual estimate (i.e., the aforementioned first residual feature estimate) can be used to obtain the 16-dimensional final high-frequency feature vector estimate (i.e., the aforementioned final high-frequency feature estimate). The i-th value of the high-frequency feature vector estimate is added to the 2i-1 and 2i values ​​of the first high-frequency feature vector residual estimate, respectively, to obtain the 2i-1 and 2i values ​​of the final high-frequency feature vector estimate, respectively. For example, if the high-frequency feature estimate is

number

number

number

[0340] Referring to Figure 13D, we will explain the following using a configuration that is in ultra-wideband coding mode and has a bitrate mode bit of 01 as an example.

[0341] First, the frame header of the bitstream package is analyzed. If the encoding mode bit is 1, it means that ultra-wideband encoding is used during encoding. If the bitrate mode bit is 10, it means that the wideband portion uses low-frequency codebook-1 during encoding. After encoding the ultra-wideband portion, the bitstream package contains an 8-dimensional high-frequency feature vector index.

[0342] For the broadband portion, the low-frequency bitstream was analyzed using entropy coding and decoding techniques, and the estimated 56-dimensional low-frequency feature vector was obtained using the Low-Frequency Codebook-4.

number

[0343] For the ultra-wideband portion, first, an 8-dimensional high-frequency feature vector estimate is obtained for each frame based on a 5-bit codebook, and the 8-dimensional high-frequency feature vector estimate is copied to obtain the 16-dimensional final high-frequency feature vector estimate (i.e., the aforementioned final high-frequency feature estimate). For example, if the high-frequency feature estimate is

number

number

[0344] Here, we will explain using a configuration that is in ultra-wideband coding mode and has 11 bitrate mode bits as an example.

[0345] First, the frame header of the bitstream package is analyzed. If the encoding mode bit is 1, it means that ultra-wideband encoding is used during encoding. If the bitrate mode bit is 11, it means that the wideband portion uses low-frequency codebook-1 during encoding. After encoding the ultra-wideband portion, the bitstream package contains an 8-dimensional high-frequency feature vector index, a 16-dimensional first high-frequency feature vector residual index, and a 16-dimensional second high-frequency feature vector residual index.

[0346] For the broadband portion, the low-frequency bitstream was analyzed using entropy coding and decoding techniques, and the estimated 56-dimensional low-frequency feature vector was obtained using the Low-Frequency Codebook-1.

number

[0347] For the ultra-wideband portion, first, an 8-dimensional high-frequency feature vector estimate is obtained for each frame based on a 5-bit codebook. Next, a 16-dimensional first high-frequency feature vector residual estimate and a 16-dimensional second high-frequency feature vector residual estimate are obtained for each frame based on a 2-bit codebook. Using the corresponding 8-dimensional high-frequency feature vector estimate (i.e., the aforementioned high-frequency feature estimate), the 16-dimensional first high-frequency feature vector residual estimate (i.e., the aforementioned first residual feature estimate), and the 16-dimensional second high-frequency feature vector residual estimate (i.e., the aforementioned second residual feature estimate), a 16-dimensional final high-frequency feature vector estimate (i.e., the aforementioned final high-frequency feature estimate) can be obtained. The i-th value among the high-frequency feature vector estimates is added to the (2i-1)-th value among the first high-frequency feature vector residual estimates and the (2i-1)-th value among the second high-frequency feature vector residual estimates to obtain the (2i-1)-th value among the final high-frequency feature vector estimates. The i-th value among the high-frequency feature vector estimates is added to the (2i)-th value among the first high-frequency feature vector residual estimates and the (2i)-th value among the second high-frequency feature vector residual estimates to obtain the (2i)-th value among the final high-frequency feature vector estimates. For example, if the high-frequency feature estimates are

number

number

number

number

[0348] In step 22, the estimated feature vector of the low-frequency bitstream

number

[0349] First, estimates of the feature vectors of low-frequency bitstreams.

number

number

[0350] First, we call a causal convolution and the input tensor

number

[0351] Next, four decoding blocks with different upsampling factors (Up_factors) are cascaded. Each decoding block includes one convolutional layer, one upsampling module, and one residual block, where the convolutional layer is used to halve the number of input channels. The upsampling module contains a specific Up_factor for upsampling. The residual block contains five residual units based on extended convolutions. The Up_factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Thus, the number of output channels of the four decoding blocks are set to 128, 64, 32, and 16, respectively. Through processing of the four decoding blocks, the 256×1 tensor is transformed into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. Herein, the embodiments of this application are not limited to the number of decoding blocks, which can be any positive integer such as 2, 3, 4, or 5.

[0352] Here, for upsampling modules containing a specific Up_factor, the upsampling operation can be performed using a copy (Repeat) operation in a repeating padding manner. In this way, complexity can be reduced.

[0353] Here, the configuration of the five residual units based on the extended convolution on the decoding side is similar to the configuration of the residual units on the encoding side, including, but not limited to, the internal structure of the residual units, the convolution kernel size, and the extension ratio. The configuration of the grouping number used in the extended convolution in the decoding block is shown in Table 2. Here, in the decoding block, a grouping number of 2 is used more often to associate more input channels with output channels and improve the quality of speech reconstruction.

[0354] [Table 2]

[0355] Next, post-processing is performed on the 16x160 tensor output by the cascaded decoding blocks. For example, a Repeat operation with factor 2 is performed on the 16x160 tensor output by the cascaded decoding blocks, followed by upsampling, then a convolution operation is performed, and a 16x320 tensor is generated using an activation function that may be PReLU.

[0356] Finally, a single causal convolution can be called to transform the input 16x320 tensor into a 1x320 tensor and reconstruct the low-frequency subband signal.

[0357] In step 23, the estimated feature vector of the high-frequency subband signal

number

[0358] Similar to high-frequency analysis on the encoding side, the high-frequency reconstruction in the embodiments of this application includes two solutions.

[0359] The first realization of high-frequency reconstruction corresponds to the first realization of high-frequency analysis on the encoding side. Estimated feature vectors of high-frequency subband signals.

number

number

[0360] The second implementation of high-frequency reconstruction corresponds to the frequency bandwidth extension technique for high-frequency analysis on the encoding side. Based on 16 MDCT subband spectral envelopes decoded from the high-frequency bitstream (each subframe containing 8 subband spectral envelopes), i.e., the final estimate of the high-frequency feature vectors, the following operations are performed.

[0361] First, the estimated value of the low-frequency subband signal generated on the decoding side.

number

[0362] next,

number

[0363] It should be explained that if the bitstream package contains flatness side information, additional operations are required. If Sfp_FLAG==0, no operations are performed. If Sfp_FLAG==1, an additional flattening operation is required. The implementation of the flattening operation is as follows:

[0364] Taking the MDCT coefficient c(i) at the i-th position as an example, the mean power spectrum of the neighborhood of c(i) is as shown in equation (3).

number

[0365]

number

[0366]

number

number

number

number

[0367] After performing the flattening operation, the eight previously acquired subband spectral envelopes (i.e., the eight subband spectral envelopes obtained by querying the quantization table) are recalled. These eight subband spectral envelopes correspond to eight high-frequency subbands, and the reference values ​​of the MDCT coefficients of the 160 points of the generated high-frequency subband signal are divided into eight reference high-frequency subbands. Based on one high-frequency subband and its corresponding reference high-frequency subband, a gain is applied (multiplication in the frequency domain) to the reference values ​​of the MDCT coefficients of the 160 points of the generated high-frequency subband signal. For example, a gain factor is calculated based on the average energy of the high-frequency subband and the average energy of the corresponding reference high-frequency subband, and the MDCT coefficient corresponding to each point in the corresponding reference high-frequency subband is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to that of the original encoded version.

[0368] For example, assuming that the average energy of the reference high-frequency subband (i.e., the subband separated from the reference values ​​of the MDCT coefficients for 160 points of the generated high-frequency subband) is Y_L, and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectral envelope decoded based on the bitstream) is Y_H, then we calculate the gain factor a = sqrt(Y_H / Y_L), where sqrt() represents the square root calculation function and is used to calculate the square root of (Y_H / Y_L). After obtaining the gain factor a, we directly multiply the MDCT coefficients of each point in the reference high-frequency subband by a, so that the average energy of the gained MDCT coefficients (virtually generated) is very close to that of the original encoded ones.

[0369] Finally, the inverse MDCT transform is called to generate estimates of the first subframe of the high-frequency subband signal (subframe estimates calculated from the MDCT coefficients of the first set of 160 points) and the second subframe of the high-frequency subband signal (subframe estimates calculated from the MDCT coefficients of the second set of 160 points). The estimates of the first and second subframes are then combined to obtain the estimate of the high-frequency subband signal.

number

number

[0370] In step 24, the composite filter is called.

[0371] Estimated low-frequency subband signal on the decoding side

number

number

number

[0372] The embodiment of this application allows for the joint training of related encoding and decoding networks by collecting data, thereby obtaining optimal parameters. Users only need to prepare the data and configure the corresponding network structure; after background training is completed, the trained model can be used for input.

[0373] In a voice communication system, the communication method includes monaural voice communication and stereo voice communication. Therefore, the embodiments of this application are not limited to the aforementioned monaural voice communication coding and decoding technology, but may also include stereo coding and decoding technology. Here, stereo coding and decoding technology enables the encoder to receive signals for both the left and right audio channels (i.e., the input signal for the left audio channel and the input signal for the right channel), and when a user wears headphones and makes a call, they can have a certain sense of left and right spatial awareness.

[0374] The following describes the audio encoding method and audio decoding method provided in the embodiments of this application based on a stereo signal.

[0375] It should be noted that the stereo coding schemes of the embodiments of this application include, but are not limited to, the following two methods: 1) Using the aforementioned monaural signal coding and decoding techniques, the left audio channel and the right audio channel are coded and decoded respectively, and the stereo signal is restored by combining the single-channel reconstructed signals decoded on the decoding side. 2) Coding and decoding are performed using parametric stereo techniques.

[0376] Here, parametric stereo coding is considered a technique for efficiently coding a stereo signal by adding a small amount of parametric overhead to a mono signal to describe stereo information. On the coding side, the input signals of the left and right audio channels (i.e., the stereo signal) are downmixed into a single signal, which is considered the aforementioned input signal x(n), and then coded using the mono signal coding method described above in the embodiment of this application. Parametric stereo coding also extracts feature information related to the left and right audio channels (i.e., stereo feature vectors, usually 1-2 kbps), and these stereo feature vectors are used to mix the side information of the mono channel. On the decoding side, decoding yields a single downmix signal (i.e., the reconstructed signal described above), and the stereo signals of the left and right audio channels are reconstructed using the decoded single downmix signal and the stereo feature vectors describing the correlation between the left and right audio channels.

[0377] Building upon the above embodiment, a 1-bit audio channel bit can be assigned to the 8-bit frame header in the bitstream package. Based on the 1-bit stereo encoding bit, it is possible to determine whether mono or stereo encoding is used. A specific embodiment is as follows.

[0378] On the encoding side, if the number of input audio channels is 1, the audio channel flag (Flag) is set to 0, and 0 is written to the audio channel bit corresponding to the frame header, meaning that monaural encoding is performed on the input signal, that is, the encoding method for the input signal x(n) described above is reused for encoding. If the number of input audio channels is 2, i.e., stereo, the audio channel flag is set to 1, and 1 is written to the audio channel bit corresponding to the frame header, that is, stereo encoding is performed on the input signal, that is, the parametric stereo encoding method described above is reused for encoding.

[0379] On the decoding side, the audio channel bits corresponding to the frame header are analyzed. If the audio channel bits are 0, it indicates monaural decoding, and the reconstructed signal is decoded by reusing the decoding method described above. If the audio channel bits are 1, it indicates stereo decoding, and the parametric stereo decoding method described above is reused to perform decoding.

[0380] The bitstream packages shown in Figures 15A to 15E include not only the bitstream packages shown in Figures 13A to 13E above, but also stereo feature vectors. On the decoding side, the stereo signals of the left and right audio channels are reconstructed using the decoded single downmix signal and the stereo feature vectors that describe the correlation between the left and right audio channels.

[0381] Furthermore, audio channel bits do not need to be assigned to the 8-bit frame header in the bitstream package. For example, if audio channel bits are not present, mono encoding is used by default.

[0382] The frame header in each embodiment of this application is not limited to 8 bits; it may have more or fewer bits. In another embodiment, more bits may be allocated to the audio channel bits, for example, 2 audio channel bits, thereby representing encoding and decoding of more audio channels, such as 3.1 channels or 4.1 channels.

[0383] In summary, the multimode multirate coding and decoding methods provided in the embodiments of this application, through an organic combination of signal decomposition, signal processing techniques, and deep neural networks, significantly improve audio quality compared to signal processing solutions, while ensuring acceptable complexity.

[0384] To date, exemplary applications and implementations of the terminal equipment provided in the embodiments of this application have been described in combination with the audio encoding or decoding methods provided in the embodiments of this application. The embodiments of this application further provide a method for processing a bitstream, the bitstream being decoded based on the audio decoding method described above or generated based on the audio encoding method described above.

[0385] To date, the audio encoding method or audio decoding method provided in the embodiments of this application has been described by combining exemplary applications and implementations of the terminal equipment provided in the embodiments of this application. The embodiments of this application further provide an audio encoding device and an audio decoding device. In actual applications, each functional module in the audio encoding device and audio decoding device can be realized in cooperation with hardware resources of an electronic device (e.g., terminal equipment, server, or server cluster), such as computing resources such as a processor, communication resources (used to support the realization of various types of communication such as fiber optics and cellular), and memory. Figure 3A shows an audio encoding device 555 stored in memory 550, and Figure 3B shows an audio decoding device 655 stored in memory 650. These can be implemented as software in the form of a program or plug-in, for example, a software module designed in a programming language such as software C / C++ or Java, application software designed in a programming language such as C / C++ or Java, or a dedicated software module in a large-scale software system, an application programming interface, a plug-in, or a cloud service. Examples of different implementations will be described below.

[0386] Here, the audio coding device 555 includes a series of modules, including a feature extraction module 5551, an encoding module 5552, and a signal coding module 5553. The solution in which each module in the audio coding device 555 provided in the embodiment of this application works together to achieve audio coding will be described below.

[0387] The second acquisition module 5551 is configured to acquire a target coding mode for an audio signal from a plurality of coding modes and a target bitrate mode for an audio signal from a plurality of bitrate modes in response to a coding request for an audio signal. The extraction module 5552 is configured to extract the coding features of the audio signal from the audio signal based on the target coding mode. The signal coding module 5553 is configured to perform signal coding on the coding features based on the target bitrate mode to obtain an audio bitstream of the audio signal. The configuration module 5554 is configured to determine a frame header based on the target coding mode and the target bitrate mode. The generation module 5555 is configured to generate an audio bitstream package of the audio signal based on the audio bitstream and the frame header.

[0388] In some embodiments, if the target coding mode is a broadband coding mode, the extraction module 5552 is further configured to invoke a first neural network based on the broadband coding mode, and to use the first neural network to extract the coded features of the audio signal from the audio signal.

[0389] In some embodiments, the extraction module 5552 is further configured to perform feature extraction on the audio signal to obtain audio features of the audio signal, and to perform residual processing on the audio features using at least one residual unit included in the first neural network to obtain encoded features of the audio signal.

[0390] In some embodiments, the first neural network includes four coding blocks, each coding block including four or five residual units.

[0391] In some embodiments, the target bitrate mode is used to indicate that signal coding is performed on the coded features of the audio signal using a target codebook, and the signal coding module 5553 is further configured to perform quantization on the coded features of the audio signal using the target codebook to obtain quantized values ​​of the coded features, and to perform entropy coding on the quantized values ​​using a target bitrate corresponding to the target codebook to obtain an audio bitstream of the audio signal.

[0392] In some embodiments, when the target coding mode is an ultra-wideband coding mode, the extraction module 5552 is configured to further perform subband decomposition on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal, extract low-frequency features of the low-frequency subband signal from the low-frequency subband signal using a second neural network, perform high-frequency analysis on the high-frequency subband signal to obtain high-frequency features of the high-frequency subband signal, and determine the low-frequency features and high-frequency features as coding features of the audio signal.

[0393] In some embodiments, the extraction module 5552 is further configured to perform frame division on the high-frequency subband signal to obtain a plurality of subframes of the high-frequency subband signal, perform frequency bandwidth expansion processing on each of the subframes to obtain a subband spectral envelope for each of the subframes, and make the subband spectral envelopes corresponding to each of the plurality of subframes the high-frequency features of the high-frequency subband signal.

[0394] In some embodiments, the extraction module 5552 is further configured to perform frequency domain transformation based on a plurality of sample points included in the subframe, obtain transformation coefficients corresponding to each of the plurality of sample points, divide the transformation coefficients corresponding to each of the plurality of sample points into a plurality of subbands, perform averaging on the transformation coefficients included in each of the subbands to obtain an average energy corresponding to each of the subbands, and use the average energy as the subband spectral envelope corresponding to each of the subbands.

[0395] In some embodiments, the target bitrate mode includes indicating that a first codebook is used to perform signal coding on the low-frequency features and a second codebook is used to perform signal coding on the high-frequency features, the signal coding module 5553 is further configured to perform quantization on the low-frequency features using the first codebook to obtain quantized values ​​of the low-frequency features, entropy coding on the quantized values ​​of the low-frequency features using the bitrate corresponding to the first codebook to obtain a low-frequency bitstream of the low-frequency subband signal, perform quantization on the high-frequency features using the second codebook to obtain quantized values ​​of the high-frequency features, entropy coding on the quantized values ​​of the high-frequency features using the bitrate corresponding to the second codebook to obtain a high-frequency bitstream of the high-frequency subband signal, and construct an audio bitstream of the audio signal based on the low-frequency bitstream and the high-frequency bitstream.

[0396] In some embodiments, the target bitrate mode further includes indicating that a third codebook is used to perform signal coding on the residual features of the high-frequency features, the signal coding module 5553 further determines the subbands corresponding to the subframes of the high-frequency subband signal, divides the subbands into a plurality of subsets, averages the conversion coefficients contained in each subset to obtain the average energy corresponding to each subset, sets the average energy as the envelope value corresponding to each subset, determines the quantized value of the subband spectral envelope of the subbands corresponding to the subsets in the quantized value of the high-frequency features, and sets the envelope corresponding to the subsets The difference between the subband value and the quantized value of the subband spectral envelope is taken as the first residual value, the residual features of the high-frequency features are determined based on the first residual value, quantization processing is performed on the residual features of the high-frequency features using the third codebook to obtain the quantized value of the residual features of the high-frequency features, entropy coding is performed on the quantized value of the residual features of the high-frequency features using the bitrate corresponding to the third codebook to obtain the residual bitstream of the residual features, and the configuration module 5554 is further configured to make the low-frequency bitstream, the high-frequency bitstream, and the residual bitstream into the audio bitstream of the audio signal.

[0397] In some embodiments, when the number of residual values ​​is N, N is a positive integer greater than 1, and the signal coding module 5553 is further configured to determine the quantized value of the nth residual value, determine the sum of the quantized value of the nth residual value and the quantized value of the subband spectral envelope of the corresponding subband, and take the difference between the envelope value corresponding to the subset and the sum as the (n+1)th residual value, thereby determining the N residual values ​​as residual features of the high-frequency feature, where n is a progressively increasing positive integer and 1 ≤ n ≤ N.

[0398] In some embodiments, the signal coding module 5553 is further configured to determine flatness side information for the high-frequency subband signal before generating an audio bitstream package of the audio signal based on the audio bitstream and the frame header, and the generation module 5555 is further configured to combine the audio bitstream, the frame header and the flatness side information to obtain an audio bitstream package of the audio signal.

[0399] In some embodiments, the signal coding module 5553 further divides the conversion coefficients included in the subframe of the high-frequency subband signal into a plurality of blocks, determines a first flatness for each block, determines a second flatness for a low-frequency specified band of the audio signal, sets the flatness side information to a first value if the first flatness is less than the second flatness or if the first flatness is less than a flatness threshold, the first value indicating that flattening processing is required during audio decoding, and sets the flatness side information to a second value if the first flatness is greater than or equal to the second flatness and the first flatness is greater than or equal to the flatness threshold, the second value indicating that flattening processing is not required during audio decoding.

[0400] In some embodiments, the frame header further includes at least one audio channel bit, which is used to indicate that audio coding is performed on the audio signal using a monaural or stereo coding method.

[0401] Here, the audio decoding device 655 includes a series of modules, including a first acquisition module 6551, a signal decoding module 6552, and a reconstruction module 6553. The solution in which each module in the audio coding device 555 provided in the embodiment of this application works together to achieve audio coding will be described below.

[0402] The first acquisition module 6551 acquires an audio bitstream package, the audio bitstream package includes an audio bitstream, the audio bitstream is obtained by performing audio coding on an audio signal using a target coding mode and a target bitrate mode, the target coding mode is obtained from a plurality of candidate coding modes, and the target bitrate mode is obtained from a plurality of candidate bitrate modes, and is configured to acquire the target coding mode and the target bitrate mode from the frame header included in the audio bitstream package in response to a decoding request for the audio bitstream package; the signal decoding module 6552 is configured to perform signal decoding on the audio bitstream using the target coding mode and the target bitrate mode and obtain an estimated coding feature value corresponding to the audio bitstream; and the reconstruction module 6553 is configured to reconstruct the estimated coding feature value using the target coding mode and obtain a reconstructed audio signal corresponding to the audio bitstream.

[0403] In some embodiments, when the target coding mode is a broadband coding mode, the target bitrate mode is used to indicate that signal coding is being performed on the coding features of the audio signal using a target codebook, and the signal decoding module 6552 is further configured to perform entropy decoding on the audio bitstream using the target bitrate corresponding to the target codebook to obtain a quantized value corresponding to the audio bitstream, and to perform inverse quantization on the quantized value corresponding to the audio bitstream using the target codebook to obtain an estimated coding feature value corresponding to the audio bitstream.

[0404] In some embodiments, if the target coding mode is a broadband coding mode, the reconstruction module 6553 is further configured to invoke a third neural network based on the broadband coding mode, and to use the third neural network to reconstruct the coding feature estimates corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0405] In some embodiments, the reconstruction module 6553 is further configured to use at least one residual unit included in the third neural network to perform residual processing on the encoded feature estimates corresponding to the audio bitstream to obtain audio feature estimates corresponding to the audio bitstream, to perform feature reconstruction on the audio feature estimates corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0406] In some embodiments, the third neural network includes four decoding blocks, each of which includes four or five residual units.

[0407] In some embodiments, when the target coding mode is an ultra-wideband coding mode, the target bitrate mode includes indicating that a first codebook is used to signal code the low-frequency features of the audio signal and a second codebook is used to signal code the high-frequency features of the audio signal, the audio bitstream includes a low-frequency bitstream and a high-frequency bitstream, the signal decoding module 6552 further obtains the low-frequency bitstream and the high-frequency bitstream from the audio bitstream by the target coding mode and entropy decoding the low-frequency bitstream by the bitrate corresponding to the first codebook. The system is configured to perform the following steps to obtain a quantized value corresponding to the low-frequency bitstream, perform inverse quantization on the quantized value corresponding to the low-frequency bitstream using the first codebook to obtain an estimated low-frequency feature value corresponding to the low-frequency bitstream, perform entropy decoding on the high-frequency bitstream using the bitrate corresponding to the second codebook to obtain a quantized value corresponding to the high-frequency bitstream, perform inverse quantization on the quantized value corresponding to the high-frequency bitstream using the second codebook to obtain an estimated high-frequency feature value corresponding to the high-frequency bitstream, and determine an estimated encoded feature value corresponding to the audio bitstream based on the estimated low-frequency feature value and the estimated high-frequency feature value.

[0408] In some embodiments, the audio bitstream further includes a residual bitstream, the target bitrate mode further includes indicating that a third codebook is used to signal encode the residual features of the high-frequency features corresponding to the audio bitstream, the signal decoding module 6552 is further configured to entropy decode the residual bitstream at the bitrate corresponding to the third codebook to obtain a quantized value corresponding to the residual bitstream, dequantize the quantized value corresponding to the residual bitstream using the third codebook to obtain a residual feature estimate corresponding to the residual bitstream, determine the sum of the high-frequency feature estimate and the residual feature estimate as the final high-frequency feature estimate, and determine the low-frequency feature estimate and the final high-frequency feature estimate as the encoded feature estimate corresponding to the audio bitstream.

[0409] In some embodiments, the reconstruction module 6553 is further configured, when the target coding mode is an ultra-wideband coding mode, to perform feature reconstruction on the low-frequency feature estimates included in the coding feature estimates using a fourth neural network to obtain a low-frequency subband signal estimate corresponding to the audio bitstream, to perform high-frequency reconstruction on the high-frequency feature estimates included in the coding feature estimates to obtain a high-frequency subband signal estimate corresponding to the audio bitstream, and to perform subband synthesis on the low-frequency subband signal estimates and the high-frequency subband signal estimates to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0410] In some embodiments, the reconstruction module 6553 is configured to further perform frequency domain transformation on the sample points in the first half and the sample points in the second half included in the low-frequency subband signal estimate to obtain a first transformation coefficient corresponding to the sample points in the first half and a second transformation coefficient corresponding to the sample points in the second half, perform inverse frequency band expansion on the high-frequency feature estimate based on the first transformation coefficient to obtain a first high-frequency subband signal estimate, perform inverse frequency band expansion on the high-frequency feature estimate based on the second transformation coefficient to obtain a second high-frequency subband signal estimate, and combine the first high-frequency subband signal estimate and the second high-frequency subband signal estimate to obtain a high-frequency subband signal estimate corresponding to the audio bitstream.

[0411] In some embodiments, the reconstruction module 6553 is further configured to perform spectral copying on the latter half of the first transformation coefficients to obtain a first reference transformation coefficient of a reference high-frequency subband signal, to gain the first reference transformation coefficient based on the first half of the subband spectral envelope corresponding to the high-frequency feature estimate to obtain a gained first reference transformation coefficient, and to perform inverse frequency domain transformation on the gained first reference transformation coefficient to obtain a first high-frequency subband signal estimate.

[0412] In some embodiments, the audio bitstream package includes flatness side information, spectral copying processing is performed on the transform coefficients in the latter half of the first transform coefficients to obtain the first reference transform coefficients of the reference high-frequency subband signal, then the reconstruction module 6553 is further configured to: if the flatness side information indicates that a flattening process is required during audio decoding, determine the (i-1)th first reference transform coefficient, the ith first reference transform coefficient, and the (i+1)th first reference transform coefficient; determine an average power spectrum based on the (i-1)th first reference transform coefficient, the ith first reference transform coefficient, and the (i+1)th first reference transform coefficient; and set the ratio of the ith first reference transform coefficient to the average power spectrum as the new ith first reference transform coefficient, where 1<i<I, i is a positive integer, and I is the number of the first reference transform coefficients.

[0413] In some embodiments, the frame header further includes at least 1 bit of audio channel bit, and the audio channel bit is used to indicate that audio decoding is performed on the audio signal using a monaural decoding method or a stereo decoding method.

[0414] The embodiments of the present application provide a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions to cause the electronic device to perform the above audio encoding method or audio decoding method according to the embodiments of the present application.

[0415] The embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, cause the processor to perform the audio encoding method or audio decoding method provided in the embodiments of the present application, for example, the audio encoding method shown in Figure 4A.

[0416] In some embodiments, the computer-readable storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM, and may be various electronic devices containing one or any combination of the above-mentioned memories.

[0417] In some embodiments, a computer-executable instruction (abbreviated as "executable instruction") may take the form of a program, software, software module, script, or code, and may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0418] For example, executable instructions do not necessarily correspond to files in the file system, but may be stored in parts of files that store other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program being discussed, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code sections).

[0419] For example, executable instructions may be configured to run on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0420] It is understandable that, if the embodiments of this application apply to a specific product or technology, the user's permission or consent must be obtained for any relevant data, such as user information, related to the embodiments of this application. The collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0421] The foregoing description is limited to embodiments of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made in the spirit and scope of this application are all included within the scope of protection of this application.

Claims

1. An audio decoding method performed by an electronic device, A step of obtaining an audio bitstream package, wherein the audio bitstream package includes an audio bitstream, the audio bitstream is obtained by performing audio coding on an audio signal using a target coding mode and a target bitrate mode, the target coding mode is obtained from a plurality of candidate coding modes, and the target bitrate mode is obtained from a plurality of candidate bitrate modes. In response to a decoding request for the audio bitstream package, the steps include obtaining the target encoding mode and the target bitrate mode from the frame header included in the audio bitstream package, The steps include: performing signal decoding on the audio bitstream using the target coding mode and the target bitrate mode to obtain an estimated coding feature value corresponding to the audio bitstream; An audio decoding method comprising the steps of: reconstructing the encoded feature estimates using the target encoding mode to obtain a reconstructed audio signal corresponding to the audio bitstream.

2. When the target coding mode is a broadband coding mode, the target bitrate mode is used to indicate that signal coding is being performed on the coding features of the audio signal using a target codebook. The step of performing signal decoding on the audio bitstream using the target coding mode and the target bitrate mode, and obtaining an estimated coding feature value corresponding to the audio bitstream, is: The steps include: performing entropy decoding on the audio bitstream using the target bitrate corresponding to the target codebook to obtain a quantization value corresponding to the audio bitstream; The method is characterized by including the step of performing inverse quantization on the quantized values ​​using the target codebook to obtain an estimated encoded feature value corresponding to the audio bitstream. The audio decoding method according to claim 1.

3. When the target coding mode is a broadband coding mode, the step of reconstructing the coded feature estimates using the target coding mode to obtain a reconstructed audio signal corresponding to the audio bitstream is: The steps include calling a third neural network based on the aforementioned broadband coding mode, The method is characterized by including the step of reconstructing the encoded feature estimates using the third neural network to obtain a reconstructed audio signal corresponding to the audio bitstream. The audio decoding method according to claim 1 or 2.

4. The step of reconstructing the encoded feature estimates to obtain a reconstructed audio signal corresponding to the audio bitstream is: The steps include: performing residual processing on the encoded feature estimate using at least one residual unit included in the third neural network to obtain an audio feature estimate corresponding to the audio bitstream; The method is characterized by including the step of performing feature reconstruction on the estimated audio features to obtain a reconstructed audio signal corresponding to the audio bitstream. The audio decoding method according to claim 3.

5. The third neural network is characterized in that it includes four decoding blocks, and each decoding block includes four or five residual units. The audio decoding method according to claim 4.

6. When the target coding mode is an ultra-wideband coding mode, the target bitrate mode includes indicating that a first codebook is used to signal code the low-frequency features of the audio signal and a second codebook is used to signal code the high-frequency features of the audio signal, and the audio bitstream includes a low-frequency bitstream and a high-frequency bitstream. The step of performing signal decoding on the audio bitstream using the target coding mode and the target bitrate mode, and obtaining an estimated coding feature value corresponding to the audio bitstream, is: The steps include obtaining the low-frequency bitstream and the high-frequency bitstream from the audio bitstream using the target encoding mode, The steps include: performing entropy decoding on the low-frequency bitstream using the bitrate corresponding to the first codebook to obtain a quantized value corresponding to the low-frequency bitstream; performing inverse quantization on the quantized value corresponding to the low-frequency bitstream using the first codebook to obtain an estimated low-frequency feature value corresponding to the low-frequency bitstream; The steps include: performing entropy decoding on the high-frequency bitstream using the bitrate corresponding to the second codebook to obtain a quantization value corresponding to the high-frequency bitstream; performing inverse quantization on the quantization value corresponding to the high-frequency bitstream using the second codebook to obtain an estimated high-frequency feature value corresponding to the high-frequency bitstream; The method is characterized by including the step of determining an estimated encoding feature corresponding to the audio bitstream based on the estimated low-frequency feature and the estimated high-frequency feature. The audio decoding method according to any one of claims 1 to 5.

7. The audio bitstream further includes a residual bitstream, and the target bitrate mode further includes indicating that a third codebook is being used to signal encode the residual features of the high-frequency features corresponding to the audio bitstream. The aforementioned audio decoding method further, The process includes the steps of performing entropy decoding on the residual bitstream using the bitrate corresponding to the third codebook to obtain a quantized value corresponding to the residual bitstream, performing inverse quantization on the quantized value corresponding to the residual bitstream using the third codebook to obtain an estimated residual feature value corresponding to the residual bitstream, and / or, The step of determining the coding feature estimate corresponding to the audio bitstream based on the low-frequency feature estimate and the high-frequency feature estimate is: The step of determining the sum of the estimated high-frequency features and the estimated residual features as the final estimated high-frequency features, The method is characterized by including the step of determining the low-frequency feature estimate and the high-frequency feature final estimate as the encoded feature estimate corresponding to the audio bitstream. The audio decoding method according to claim 6.

8. The step of reconstructing the estimated encoded features using the target encoding mode to obtain a reconstructed audio signal corresponding to the audio bitstream is: When the target coding mode is an ultra-wideband coding mode, the fourth neural network performs feature reconstruction on the low-frequency feature estimates included in the coding feature estimates to obtain low-frequency subband signal estimates corresponding to the audio bitstream. The steps include: performing high-frequency reconstruction on the high-frequency feature estimates included in the encoded feature estimates to obtain high-frequency subband signal estimates corresponding to the audio bitstream; The method is characterized by including the step of performing subband synthesis on the estimated low-frequency subband signal and the estimated high-frequency subband signal to obtain a reconstructed audio signal corresponding to the audio bitstream. The audio decoding method according to any one of claims 1 to 7.

9. The step of performing high-frequency reconstruction on the high-frequency feature estimates included in the aforementioned encoded feature estimates to obtain high-frequency subband signal estimates corresponding to the audio bitstream is: The steps include: performing frequency domain transformation on the sample points in the first half and the sample points in the second half included in the estimated low-frequency subband signal, to obtain a first transformation coefficient corresponding to the sample points in the first half and a second transformation coefficient corresponding to the sample points in the second half; The steps include: performing an inverse process of frequency bandwidth expansion on the high-frequency feature estimate based on the first conversion coefficient to obtain a first high-frequency subband signal estimate; The steps include: performing the inverse process of frequency bandwidth expansion on the high-frequency feature estimate based on the second conversion coefficient to obtain a second high-frequency subband signal estimate; The method is characterized by including the step of combining the first high-frequency subband signal estimate and the second high-frequency subband signal estimate to obtain a high-frequency subband signal estimate corresponding to the audio bitstream. The audio decoding method according to claim 8.

10. The step of performing the inverse process of frequency bandwidth expansion on the high-frequency feature estimate based on the first conversion coefficient to obtain the first high-frequency subband signal estimate is: The steps include performing a spectral copy on the latter half of the first conversion coefficient to obtain the first reference conversion coefficient of the reference high-frequency subband signal, The steps include: applying gain to the first reference transformation coefficient based on the subband spectral envelope of the first half corresponding to the estimated high-frequency features, and obtaining the first reference transformation coefficient after gain; The method is characterized by including the step of performing an inverse frequency domain transformation on the first reference transformation coefficient after gain to obtain an estimated first high-frequency subband signal. The audio decoding method according to claim 9.

11. The aforementioned audio bitstream package includes flatness side information, After performing a spectral copy on the latter half of the first conversion coefficient to obtain the first reference conversion coefficient of the reference high-frequency subband signal, the audio decoding method further: If the flatness side information indicates that flattening processing is required during audio decoding, the steps include determining the (i-1)th first reference transformation coefficient, the i-th first reference transformation coefficient, and the (i+1)th first reference transformation coefficient, A step of determining the average power spectrum based on the i-1th first reference transformation coefficient, the ith first reference transformation coefficient, and the i+1th first reference transformation coefficient, The step includes setting the ratio of the i-th first reference transformation coefficient to the average power spectrum as a new i-th first reference transformation coefficient, The characteristic is that 1 < i < I, where i is a positive integer and I is the number of the first reference transformation coefficients. The audio decoding method according to claim 10.

12. The frame header further includes at least one audio channel bit, the audio channel bit being used to indicate that audio decoding is performed on the audio signal using a mono decoding method or a stereo decoding method. The audio decoding method according to any one of claims 1 to 11.

13. An audio encoding method performed by an electronic device, The steps include: obtaining a target encoding mode for an audio signal from a plurality of encoding modes in response to an encoding request for an audio signal, and obtaining a target bitrate mode for an audio signal from a plurality of bitrate modes; The steps include: extracting the encoded features of the audio signal from the audio signal using the target encoding mode; The steps include: performing signal coding on the encoded features using the target bitrate mode to obtain an audio bitstream of the audio signal; The steps include determining the frame header based on the target encoding mode and the target bitrate mode, An audio encoding method comprising the steps of generating an audio bitstream package of the audio signal based on the audio bitstream and the frame header.

14. When the target coding mode is a broadband coding mode, the step of extracting the coding features of the audio signal from the audio signal using the target coding mode is: The steps include calling a first neural network based on the aforementioned broadband coding mode, The method is characterized by including the step of extracting encoded features of the audio signal from the audio signal using the first neural network. The audio encoding method according to claim 13.

15. The step of extracting the encoded features of the audio signal from the audio signal is: The steps include: performing feature extraction on the aforementioned audio signal to obtain the audio characteristics of the aforementioned audio signal; The method is characterized by including the step of performing residual processing on the audio features using at least one residual unit included in the first neural network to obtain the encoded features of the audio signal. The audio encoding method according to claim 14.

16. The first neural network is characterized in that it includes four coding blocks, and each coding block includes four or five residual units. The audio encoding method according to claim 15.

17. The aforementioned target bitrate mode is used to indicate that signal coding is performed on the coding features of the audio signal using a target codebook. The step of performing signal coding on the encoded features using the target bitrate mode to obtain an audio bitstream of the audio signal is: The steps include: performing quantization on the encoded features using the target codebook to obtain the quantized values ​​of the encoded features; The method is characterized by including the step of performing entropy coding on the quantization value using a target bitrate corresponding to the target codebook to obtain an audio bitstream of the audio signal. The audio encoding method according to any one of claims 13 to 16.

18. When the target coding mode is an ultra-wideband coding mode, the step of extracting the coded features of the audio signal from the audio signal using the target coding mode is: The steps include: performing subband decomposition on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal; The second neural network performs the step of extracting the low-frequency features of the low-frequency subband signal from the low-frequency subband signal, The steps include: performing high-frequency analysis on the high-frequency subband signal to obtain the high-frequency characteristics of the high-frequency subband signal; The method is characterized by including the step of determining the low-frequency features and the high-frequency features as encoding features of the audio signal. The audio encoding method according to any one of claims 13 to 17.

19. The step of performing high-frequency analysis on the high-frequency subband signal to obtain the high-frequency characteristics of the high-frequency subband signal is: The steps include: performing frame division on the high-frequency subband signal to obtain a plurality of subframes of the high-frequency subband signal; The steps include performing frequency bandwidth expansion processing on each subframe to obtain the subband spectral envelope of each subframe, The method is characterized by including the step of making the subband spectral envelopes corresponding to each of the plurality of subframes high-frequency features of the high-frequency subband signal. The audio encoding method according to claim 18.

20. The step of performing frequency bandwidth expansion processing on each subframe to obtain the subband spectral envelope of each subframe is: The steps include: performing a frequency domain transformation based on a plurality of sample points included in the subframe to obtain a transformation coefficient corresponding to each of the plurality of sample points; The steps include dividing the transformation coefficients corresponding to each of the aforementioned multiple sample points into multiple subbands, The method is characterized by including the step of performing an averaging process on the transformation coefficients included in each of the subbands to obtain the average energy corresponding to each of the subbands, and using the average energy as the subband spectral envelope corresponding to each of the subbands. The audio encoding method according to claim 19.

21. The target bitrate mode includes indicating that a first codebook is used to perform signal coding for the low-frequency features and a second codebook is used to perform signal coding for the high-frequency features. The step of performing signal coding on the encoded features using the target bitrate mode to obtain an audio bitstream of the audio signal is: The steps include: performing quantization on the low-frequency features using the first codebook to obtain the quantized values ​​of the low-frequency features; performing entropy coding on the quantized values ​​of the low-frequency features using the bitrate corresponding to the first codebook to obtain a low-frequency bitstream of the low-frequency subband signal; The steps include: performing quantization processing on the high-frequency features using the second codebook to obtain the quantized values ​​of the high-frequency features; performing entropy coding on the quantized values ​​of the high-frequency features using the bitrate corresponding to the second codebook to obtain a high-frequency bitstream of the high-frequency subband signal; The method is characterized by including the step of configuring an audio bitstream of the audio signal based on the low-frequency bitstream and the high-frequency bitstream. The audio encoding method according to claim 18.

22. The target bitrate mode further includes indicating that signal coding is performed on the residual features of the high-frequency features using a third codebook, The aforementioned audio encoding method further, The steps include determining the subband corresponding to the subframe of the high-frequency subband signal and dividing the subband into a plurality of subsets, The steps include: performing an averaging process on the conversion coefficients included in each subset to obtain the average energy corresponding to each subset, and using the average energy as the envelope value corresponding to each subset; The steps include determining the quantized value of the subband spectral envelope of the subband corresponding to the subset in the quantized value of the high-frequency feature, and taking the difference between the envelope value corresponding to the subset and the quantized value of the subband spectral envelope as the first residual value, The steps include determining the residual characteristics of the high-frequency features based on the first residual value, The process includes the steps of: performing quantization on the residual features of the high-frequency features using the third codebook to obtain the quantized values ​​of the residual features of the high-frequency features; performing entropy coding on the quantized values ​​of the residual features of the high-frequency features using the bitrate corresponding to the third codebook to obtain the residual bitstream of the residual features; and / or, The step of configuring the audio bitstream of the audio signal based on the low-frequency bitstream and the high-frequency bitstream is: The method is characterized by including the step of making the low-frequency bitstream, the high-frequency bitstream, and the residual bitstream into the audio bitstream of the audio signal. The audio encoding method according to claim 21.

23. If the number of residual values ​​is N (where N is a positive integer greater than 1), the step of determining the residual features of the high-frequency features based on the first residual value is: The steps include determining the quantized value of the nth residual value and determining the sum of the quantized value of the nth residual value and the quantized value of the subband spectral envelope of the corresponding subband, The step of taking the difference between the envelope value corresponding to the subset and the sum as the (n+1)th residual value, The process includes the step of determining N of the aforementioned residual values ​​as residual features of the high-frequency feature, n is a progressively increasing positive integer, and 1 ≤ n ≤ N. The audio encoding method according to claim 22.

24. Before generating an audio bitstream package of the audio signal based on the audio bitstream and the frame header, the audio encoding method further: The step includes determining flatness side information of the high-frequency subband signal, The step of generating an audio bitstream package for the audio signal based on the audio bitstream and the frame header is: The method is characterized by including the step of combining the audio bitstream, the frame header, and the flatness side information to obtain an audio bitstream package of the audio signal. The audio encoding method according to any one of claims 21 to 23.

25. The step of determining the flatness side information of the high-frequency subband signal is: The steps include dividing the conversion coefficients included in the subframe of the high-frequency subband signal into multiple blocks, The steps include determining the first flatness of each block and determining the second flatness of the low-frequency specified band of the audio signal, If the first flatness is less than the second flatness, or if the first flatness is less than the flatness threshold, the flatness side information is set to a first value, wherein the first value indicates that flattening processing is required during audio decoding. The method is characterized by including the step of setting the flatness side information to a second value if the first flatness is equal to or greater than the second flatness and the first flatness is equal to or greater than the flatness threshold, wherein the second value indicates that flattening processing is not required during audio decoding. The audio encoding method according to claim 24.

26. The frame header further includes at least one audio channel bit, the audio channel bit is used to indicate that audio coding is performed on the audio signal using a monaural coding method or a stereo coding method. The audio encoding method according to any one of claims 13 to 25.

27. A bitstream processing method, wherein the bitstream is decoded based on the audio decoding method described in any one of claims 1 to 12, or generated based on the audio encoding method described in any one of claims 13 to 26.

28. An audio decoding device, A step of obtaining an audio bitstream package, wherein the audio bitstream package includes an audio bitstream, the audio bitstream is obtained by performing audio encoding on an audio signal using a target encoding mode and a target bitrate mode, and the target bitrate mode is obtained from a plurality of candidate bitrate modes. A first acquisition module is configured to perform the steps of: obtaining the target encoding mode and the target bitrate mode from the frame header included in the audio bitstream package in response to a decoding request for the audio bitstream package; A signal decoding module configured to perform signal decoding on the audio bitstream using the target coding mode and the target bitrate mode, and to obtain an estimated coding feature value corresponding to the audio bitstream, An audio decoding device comprising: a reconstruction module configured to reconstruct the estimated encoded features using the target encoding mode to obtain a reconstructed audio signal corresponding to the audio bitstream.

29. An audio encoding device, A second acquisition module is configured to respond to an encoding request for an audio signal by acquiring a target encoding mode for the audio signal from a plurality of encoding modes and a target bitrate mode for the audio signal from a plurality of bitrate modes. An extraction module configured to extract the encoded features of the audio signal from the audio signal using the target coding mode, A signal coding module configured to perform signal coding on the coding features in the target bitrate mode to obtain an audio bitstream of the audio signal, A configuration module configured to determine the frame header based on the target encoding mode and the target bitrate mode, An audio encoding device comprising: a generating module configured to generate an audio bitstream package of the audio signal based on the audio bitstream and the frame header.

30. Memory for storing computer-executable instructions, An electronic device comprising a processor for implementing the audio decoding method according to any one of claims 1 to 12, or the audio encoding method according to any one of claims 13 to 26, when executing a computer-executable instruction stored in the memory.

31. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the audio decoding method according to any one of claims 1 to 12, or the audio encoding method according to any one of claims 13 to 26.

Images