Audio encoding method and apparatus, audio decoding method and apparatus, and computer program

Abstract

An audio encoding method and apparatus, an audio decoding method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product, the audio decoding method including: a step (21) of performing a signal decoding process on an audio bitstream to obtain coding features corresponding to the audio bitstream, where the audio bitstream is obtained by audio encoding an audio signal; a step (22) of performing decoding-side residual processing on the coding features corresponding to the audio bitstream using at least one residual unit to obtain audio features corresponding to the audio bitstream; and a step (23) of performing feature reconstruction processing on the audio features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application bearing application number 202311006978.4, filed with the China Patent Office on August 10, 2023, the entire contents of which are incorporated herein by reference. The present application relates to artificial intelligence technology, and in particular to an audio encoding method and apparatus, an audio decoding method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0002] Artificial intelligence (AI) is a comprehensive field of computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology is a comprehensive field that covers a wide range of fields and is related to several major directions, such as natural language processing technology and machine learning / deep learning. As technology develops, AI technology is expected to be applied in more fields and to play an increasingly important role.

[0003] Audio coding and decoding technology is an important application in the field of artificial intelligence and a core technology for communication services, including long-distance audio and video calls. Simply put, audio coding is a technique for transmitting as much audio information as possible using limited network bandwidth resources. From the perspective of Shannon information theory, audio coding is a type of source coding, whose purpose is to compress the data volume of the information to be transmitted as much as possible on the encoding side, remove redundancies within the information, and simultaneously enable lossless (or nearly lossless) restoration on the decoding side.

[0004] In the related art, in the decoding process, the quality of the audio decoding is significantly reduced in order to ensure the efficiency of the audio decoding. Summary of the Invention

[0005] Embodiments of the present application provide an audio encoding method and apparatus, an audio decoding method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product that improve the quality of audio decoding.

[0006] The technical solution of the embodiment of the present application is realized as follows:

[0007] An embodiment of the present application provides an audio encoding method applied to an electronic device, the method comprising: performing a feature extraction process on the audio signal to obtain audio features of the audio signal; performing residual processing on the audio features using at least one residual unit to obtain coding features of the audio signal; and performing a signal coding process on the coded features of the audio signal to obtain an audio bitstream of the audio signal.

[0008] An embodiment of the present application provides an audio decoding method applied to an electronic device, the method comprising: performing a signal decoding process on an audio bitstream to obtain coding features corresponding to the audio bitstream, the audio bitstream being obtained by audio encoding an audio signal; performing residual processing on coding features corresponding to the audio bitstream using at least one residual unit to obtain audio features corresponding to the audio bitstream; performing a feature reconstruction process on audio features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0009] An embodiment of the present application provides an audio encoding device, the device comprising: a feature extraction module configured to perform a feature extraction process on an audio signal to obtain audio features of the audio signal; an encoding module configured to perform residual processing on the audio features using at least one residual unit to obtain coded features of the audio signal; a signal encoding module configured to perform a signal encoding process on the encoded features of the audio signal to obtain an audio bitstream of the audio signal.

[0010] An embodiment of the present application provides an audio decoding device, the device comprising: a signal decoding module configured to perform a signal decoding process on an audio bitstream to obtain coding features corresponding to said audio bitstream, said audio bitstream being obtained by audio encoding an audio signal; a decoding module configured to perform residual processing on coding features corresponding to the audio bitstream using at least one residual unit to obtain audio features corresponding to the audio bitstream; and a feature reconstruction module configured to perform a feature reconstruction process on audio features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0011] An embodiment of the present application provides an electronic device, the electronic device comprising: a memory and a processor; the memory stores computer-executable instructions; The processor, when executing the computer-executable instructions stored in the memory, implements the audio encoding method or audio decoding method provided in the embodiments of the present application.

[0012] An embodiment of the present application provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, implements an audio encoding method or an audio decoding method provided in the embodiment of the present application.

[0013] An embodiment of the present application provides a computer program product including computer-executable instructions, which, when executed by a processor, implement an audio encoding method or an audio decoding method provided in the embodiment of the present application.

[0014] The embodiments of the present application have the following beneficial effects: By performing residual processing on the coding features corresponding to the audio bitstream, comprehensive learning of the audio bitstream can be ensured based on the characteristics of residual processing, while better utilizing the shallow information of the coding features, thereby improving the feature representation ability of the audio features of the audio bitstream and further improving the quality of audio decoding. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram of spectrum comparison at different bit rates according to an embodiment of the present application; [Figure 2] 1 is a schematic diagram of the architecture of an audio encoding / decoding system according to an embodiment of the present application; [Figure 3A] 1 is a schematic diagram illustrating a configuration of an electronic device according to an embodiment of the present application. [Figure 3B] 1 is a schematic diagram illustrating a configuration of an electronic device according to an embodiment of the present application. [Figure 4A] 1 is a first flowchart of an audio encoding method according to an embodiment of the present application; [Figure 4B] 2 is a second flowchart of an audio encoding method according to an embodiment of the present application; [Figure 4C] 3 is a third flowchart of an audio encoding method according to an embodiment of the present application; [Figure 4D] 4 is a fourth flowchart of the audio encoding method according to an embodiment of the present application. [Figure 4E] 5 is a fifth flowchart of an audio encoding method according to an embodiment of the present application; [Figure 4F] 6 is a sixth flowchart of the audio encoding method according to an embodiment of the present application. [Figure 5A] 1 is a first flowchart of an audio decoding method according to an embodiment of the present application; [Figure 5B] 2 is a second flowchart of an audio decoding method according to an embodiment of the present application; [Figure 5C] 3 is a third flowchart of an audio decoding method according to an embodiment of the present application; [Figure 5D] 4 is a fourth flowchart of the audio decoding method according to an embodiment of the present application; [Figure 6A] FIG. 1 is a schematic diagram of a channel that does not employ grouped convolution according to an embodiment of the present application. [Figure 6B] FIG. 1 is a schematic diagram of a channel employing grouped convolution according to an embodiment of the present application. [Figure 6C] FIG. 1 is a schematic diagram of an audio communication link according to an embodiment of the present application. [Figure 7A] 1 is a flowchart of an audio encoding / decoding method according to an embodiment of the present application. [Figure 7B] 1 is a flowchart of a low-complexity, low-bit-rate neural network audio compression method according to an embodiment of the present application; [Figure 8] FIG. 1 is a schematic diagram of a filter bank according to an embodiment of the present application; [Figure 9A] FIG. 1 is a schematic diagram of a general convolutional network according to an embodiment of the present application. [Figure 9B] FIG. 1 is a schematic diagram of a dilated convolutional network according to an embodiment of the present application. [Figure 10] FIG. 1 is a schematic diagram of bandwidth extension according to an embodiment of the present application. [Figure 11]FIG. 10 is a schematic diagram of a third neural network according to an embodiment of the present application. [Figure 12A] FIG. 2 is a schematic diagram illustrating the structure of a residual block used in a coding block according to an embodiment of the present application; [Figure 12B] FIG. 2 is a schematic diagram illustrating a configuration of a residual unit according to an embodiment of the present application. [Figure 13] FIG. 10 is a schematic diagram of a fourth neural network according to an embodiment of the present application. [Figure 14] FIG. 2 is a schematic diagram of a first neural network according to an embodiment of the present application. [Figure 15] FIG. 2 is a schematic diagram of a second neural network according to an embodiment of the present application. [Figure 16A] 1 is a flowchart of an audio encoding method according to an embodiment of the present application; [Figure 16B] 4 is another flowchart of an audio encoding method according to an embodiment of the present application; [Figure 17A] 1 is a flowchart of an audio decoding method according to an embodiment of the present application; [Figure 17B] 4 is another flowchart of an audio decoding method according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0016] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. The described embodiments do not limit the present application, and all other embodiments that can be obtained by those skilled in the art without any creative efforts are included in the protection scope of the present application.

[0017] The terms "first / second" and the like used in the present specification do not imply a particular order to objects, but rather distinguish between similar objects. As will be understood, "first / second" and the like can be used to interchange a particular order or order when appropriate, so that the embodiments of the present specification described herein can be performed in an order other than that shown or described herein.

[0018] In the following, references to "some embodiments" describe a subset of all possible embodiments, but it will be understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without contradiction.

[0019] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a portion of a computer program having a certain function, working together with other related portions to achieve a certain goal, and may be implemented in whole or in part by using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, one or more modules or units may be implemented using a processor (or multiple processors or memories). Furthermore, each module or unit may be part of an overall module or unit that includes the functionality of that module or unit.

[0020] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terms used herein are used only to describe the embodiments of the present application and are not intended to limit the present application.

[0021] Before describing the embodiments of the present application in detail, the nouns and terms related to the embodiments of the present application will be explained. The explanation of the nouns and terms related to the embodiments of the present application is as follows.

[0022] 1) Neural Network (NN): A mathematical model of an algorithm that performs distributed parallel information processing by imitating the behavioral characteristics of animal neural networks. Such networks depend on the complexity of the system and achieve the purpose of information processing by adjusting the relationships between a large number of internal nodes.

[0023] 2) Deep Learning (DL): A new research direction in the field of machine learning (ML). Deep learning learns the internal rules and representation hierarchy of sample data, and the information gained from this learning process is extremely useful for interpreting data such as text, images, and audio. The ultimate goal is for machines to have human-like analytical capabilities and be able to recognize data such as text, images, and audio.

[0024] 3) Quantization: It is a process of approximating continuous values ​​(or a large number of discrete values) of a signal to a finite number of discrete values ​​(or a small number of discrete values). Here, quantization includes vector quantization (VQ) and scalar quantization.

[0025] Vector quantization is an effective lossy compression technique, and its theoretical basis is Shannon's rate-distortion theory. The basic principle of vector quantization is that the index of the codeword in the codebook that best matches the input vector is transmitted and stored instead of the input vector, requiring only a simple table lookup operation during decoding. For example, several scalar data are organized into a vector space, and the vector space is divided into several small regions. During quantization, vectors corresponding to the small regions are used instead of the input vector using the corresponding index.

[0026] Scalar quantization is a quantization performed on a scalar, i.e., one-dimensional vector quantization, in which the dynamic range is divided into several small intervals, and each small interval has one representative value (i.e., index). If the input signal falls into a certain interval, the input signal is quantized to the representative value.

[0027] 4) Entropy coding: This is a lossless coding method that follows the entropy principle during the coding process and does not lose any information. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, exponential Columbus coding (Exp-Golomb), and arithmetic coding.

[0028] 5) Quadrature Mirror Filters (QMF): A filter pair including an analysis filter and a synthesis filter, where the QMF analysis filter is used to decompose the subband signal to reduce the signal bandwidth so that each subband signal can smoothly pass through its own channel for processing, and the QMF synthesis filter is used to synthesize the restored subband signals at the decoding side, for example, by zero-value insertion and band-pass filtering to reconstruct the original audio signal.

[0029] Before specifically describing the audio encoding method and audio decoding method provided in the embodiments of the present application, the following will first explain the QMF filter bank, the dilated convolutional network (also called the "dilated convolutional network"), and the band extension.

[0030] A QMF filter bank is a filter pair that includes an analysis filter and a synthesis filter. In the case of a QMF analysis filter, an input signal with a sampling rate of Fs can be decomposed into two-pass signals with a sampling rate of Fs / 2, which represent the QMF low-pass signal and the QMF high-pass signal, respectively. Figure 8 shows the spectral responses of the low-pass part H_Low(z) and the high-pass part H_High(z) of the QMF filter. Based on the relevant theoretical knowledge of the QMF analysis filter bank, the relationship between the coefficients of the above low-pass filtering and high-pass filtering can be simply described as shown in Equation 1.

number

[0031] Similarly, according to the QMF related theory, based on the QMF analysis filter banks H_Low(z) and H_High(z), the QMF synthesis filter bank can be written as shown in Equation 2:

number

[0032] The low-pass signal and high-pass signal restored on the decoding side are synthesized by a QMF synthesis filter bank, so that a reconstructed signal (ie, a synthesized signal) with a sampling rate Fs corresponding to the input signal can be restored.

[0033] 9A and 9B, FIG. 9A is a schematic diagram of a general convolutional (e.g., causal) network provided in an embodiment of the present application, and FIG. 9B is a schematic diagram of a dilated convolutional network according to an embodiment of the present application. Compared to a general convolutional network, a dilated convolution can expand the receptive field while maintaining the size of the feature map, and can also avoid errors due to upsampling and downsampling. The convolution kernel sizes shown in FIGS. 9A and 9B are both 3×3. However, while the receptive field 901 of the general convolution shown in FIG. 9A is 3, the receptive field 902 of the dilated convolution shown in FIG. 9B reaches 5. That is, for a convolution kernel size of 3×3, the receptive field of the general convolution shown in FIG. 9A is 3 and the dilation rate (the number of intervals between points in the convolution kernel) is 1, while the receptive field of the dilated convolution shown in FIG. 9B is 5 and the dilation rate is 2.

[0034] The convolution kernel can also move on a plane as shown in Figure 9A or 9B, where the concept of stride rate (step length) comes into play. For example, every time the convolution kernel shifts by one square, the corresponding stride rate is 1.

[0035] Furthermore, there is the concept of the number of convolution channels, which is the number of convolution kernels and corresponding parameters used for convolution analysis. In theory, the more channels there are, the more comprehensive and accurate the signal analysis will be. However, the higher the number of channels, the higher the complexity. For example, for a 1x320 tensor, a 24-channel convolution operation can be used, and the output will be a 24x320 tensor.

[0036] It should be noted that the dilation convolution kernel size (for example, for an audio signal, the size of the convolution kernel can be set to 1×3), dilation rate, stride rate, and number of channels can be independently defined according to the needs of the actual application, and the embodiments of the present application do not specifically limit these.

[0037] In the schematic diagram of band expansion (or band duplication) shown in Figure 10, the wideband signal is first reconstructed, then the wideband signal is duplicated into an ultra-wideband signal, and finally shaping is performed based on the ultra-wideband envelope. The frequency domain implementation method shown in Figure 10 specifically includes the following: 1) Implementing core layer coding at a low sampling rate; 2) Selecting and duplicating the low-frequency spectrum to a high-frequency portion; and 3) Performing gain control on the duplicated high-frequency spectrum based on pre-recorded boundary information (such as the energy correlation between high and low frequencies). The effect of doubling the sampling rate can be achieved with a bit rate of just 1 to 2 kbps.

[0038] Audio coding technology is a technique for transmitting as much audio information as possible using minimal network bandwidth resources. Audio codecs can achieve compression rates of over 10x, meaning that 10MB of original audio data can be transmitted in just 1MB after compression by an encoder, significantly reducing the bandwidth required for transmitting information. For example, for a wideband audio signal with a sampling rate of 16,000 Hz, using a 16-bit sampling depth (the fineness of the audio intensity recorded during sampling) results in a bitrate (amount of data transmitted per unit time) of 256 kbps for the uncompressed version. Even with lossy encoding, audio coding technology can achieve a reconstructed audio signal with a bitrate between 10 and 20 kbps, close to the uncompressed version, making it virtually indistinguishable from the uncompressed version. For services requiring higher sampling rates, such as ultra-wideband audio at 32,000 Hz, the bitrate range must reach 30 kbps or higher.

[0039] To ensure smooth communication in communication systems, industry-wide standards for speech coding and decoding have been adopted. These include standards from international and national standardization organizations such as ITU-T, 3GPP, IETF, AVS, and CCSA, as well as standards such as G.711, G.722, the AMR series, EVS, and OPUS. Figure 1 shows a schematic diagram of spectrum comparisons at different bit rates to demonstrate the relationship between compression bit rate and quality. Curve 101 is the original spectrum curve, i.e., the uncompressed signal; curve 102 is the spectrum curve of the OPUS encoder at a bit rate of 20 kbps; and curve 103 is the spectrum curve of the OPUS encoder at a bit rate of 6 kbps. As can be seen from Figure 1, as the coding bit rate increases, the compressed signal becomes closer to the original signal.

[0040] The principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples sample by sample, or based on the principle of human speech production, extract relevant low-dimensional features, encode the features on the encoding side, and reconstruct the speech signal based on these parameters on the decoding side.

[0041] The above coding principles all originate from audio signal modeling, i.e., compression methods based on signal processing, and cannot guarantee audio coding quality. To improve coding efficiency while ensuring audio quality, embodiments of the present application provide an audio encoding method and apparatus, an audio decoding method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The following describes exemplary applications of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application may be implemented as a terminal device, a server, or a combination of a terminal device and a server. The following describes an example in which the electronic device is implemented as a terminal device.

[0042] For example, refer to FIG. 2, which is a schematic diagram of the architecture of an audio encoding / decoding system 10 provided in an embodiment of the present application, where the audio decoding system 10 includes a server 200, a network 300, a terminal device 400 (i.e., encoding side), and a terminal device 500 (i.e., decoding side), where the network 300 may be a local area network, a wide area network, or a combination thereof.

[0043] In some embodiments, a client 410 runs on the terminal device 400, and the client 410 may be various types of clients, such as an instant messaging client, a web conference client, a live client, a browser, etc. In response to an audio collection command triggered by a sender (e.g., a host of a web conference, a broadcaster, an initiator of a voice call, etc.), the client 410 calls a microphone built into the terminal device 400 to collect an audio signal, and performs audio encoding processing on the collected audio signal to obtain bitstreams (high-frequency bitstream and low-frequency bitstream).

[0044] For example, the client 410 invokes the audio encoding method provided in the embodiments of the present application to encode the collected audio signal. That is, the client 410 performs a feature extraction process on the audio signal to obtain audio features of the audio signal, performs residual processing on the audio features using at least one residual unit to obtain coding features of the audio signal, and performs signal coding on the coding features of the audio signal to obtain an audio bitstream of the audio signal. In one embodiment, the client 410 optionally performs subband decomposition on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal, and then performs the audio encoding method provided in the embodiments of the present application on the low-frequency subband signals to obtain a low-frequency bitstream of the audio signal. Of course, it is also necessary to perform audio coding on the high-frequency subband signals of the audio signal to obtain a high-frequency bitstream of the audio signal. Note that the audio coding method for the high-frequency subband signals is not limited to the audio encoding method provided in the embodiments of the present application and may be other audio coding methods. The encoding side (i.e., the terminal device 400) combines signal processing technology and artificial intelligence technology to perform residual processing on the audio features of the audio signal, thereby ensuring comprehensive learning of the audio features and at the same time making better use of shallow information of the encoding features, thereby improving the feature representation ability of the encoding features and further improving the quality of audio encoding.In addition, the embodiments of the present application are not limited to the number of subband signals (including low-frequency subband signals and high-frequency subband signals) obtained by subband decomposition, and may be any positive integer such as 2, 3, 4, 5, etc., i.e., the number of low-frequency subband signals is at least 1 and the number of high-frequency subband signals is at least 1.

[0045] The client 410 can transmit the audio bitstream to the server 200 over the network 300 such that the server 200 transmits the audio bitstream to a terminal device 500 associated with a recipient (e.g., a participant in a web conference, a viewer, a recipient of a voice call, etc.).

[0046] A client 510 (e.g., an instant messaging client, a web conferencing client, a live client, a browser, etc.) running on the terminal device 500 receives the audio bitstream transmitted from the server 200, and then performs an audio decoding process on the bitstream to obtain a reconstructed audio signal, thereby enabling audio communication.

[0047] For example, the client 510 invokes the audio decoding method provided in the embodiment of the present application to decode a received audio bitstream. That is, the audio bitstream is subjected to signal decoding processing to obtain coding features corresponding to the audio bitstream, where the audio bitstream is obtained by audio coding an audio signal, and residual processing is performed on the coding features corresponding to the audio bitstream using at least one residual unit to obtain audio features corresponding to the audio bitstream, and feature reconstruction processing is performed on the audio features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream. Of course, if the received audio bitstream is a low-frequency bitstream in a full-frequency bitstream, the audio decoding method of the embodiment of the present application is performed on the low-frequency bitstream to obtain a low-frequency subband signal (an estimated value of the low-frequency subband signal in the subband decomposition at the encoding side). Of course, the full-frequency bitstream may also include a high-frequency bitstream. The high-frequency bitstream is subjected to audio decoding to obtain a high-frequency subband signal (an estimated value of the high-frequency subband signal in the subband decomposition at the encoding side), and subband synthesis processing is performed on the low-frequency subband signal and the high-frequency subband signal to obtain a reconstructed audio signal. It should be noted that the audio decoding method for the high frequency bitstream is not limited to the audio decoding method provided in the embodiments of the present application, and may be other audio decoding methods.

[0048] In some embodiments, embodiments of the present application may be realized by Cloud Technology, which is a hosting technology that integrates a set of resources, such as hardware, software, and networks, into a wide area network or a local area network to realize data computation, storage, processing, and sharing.

[0049] Cloud computing technology is a collective term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model, which allows for the creation of a resource pool that can be used flexibly and conveniently according to requirements. Cloud computing technology is an important support. The above-mentioned service interaction function between servers 200 may be realized by cloud computing technology.

[0050] 2 may be an independent physical server, a server cluster or a distributed system configured with multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, big data, and artificial intelligence platforms. The terminal devices 400 and 500 shown in FIG. 2 may be, but are not limited to, smartphones, tablet computers, notebook computers, desktop computers, smart speakers, smart watches, in-vehicle terminals, etc. The terminal devices (e.g., the terminal devices 400 and 500) and the server 200 may be directly or indirectly connected via wired or wireless communication, and the present application is not limited thereto.

[0051] In some embodiments, the terminal device or server 200 may implement the audio encoding method or audio decoding method provided in the embodiments of the present application by executing a computer program. For example, the computer program may be a native program or software module within an operating system, a local application (APP) that needs to be installed in an operating system to run, such as a live APP, web conferencing APP, or instant messaging APP, or a mini-program that can be run simply by downloading it to a browser environment or can be embedded in any APP. In short, the computer program may be an application, module, or plug-in in any form.

[0052] In some embodiments, multiple servers may constitute one blockchain, server 200 is a node on the blockchain, and information connections exist between nodes in the blockchain, and information transmission between nodes may be performed via the information connections. Here, data related to the audio encoding method or audio decoding method provided in the embodiments of the present application (e.g., audio encoding logic, audio decoding logic, high-frequency bitstream, and low-frequency bitstream) may be stored on the blockchain.

[0053] Referring to FIG. 3A, FIG. 3A is a schematic diagram illustrating the configuration of an electronic device 500 provided in an embodiment of the present application, and will be described as an example in which the electronic device 500 is a terminal device. The electronic device 500 illustrated in FIG. 3A includes at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The components within the electronic device 500 are coupled via a bus system 550. It should be understood that the bus system 550 is configured to realize communication connections between these components. The bus system 550 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for clarity of explanation, various buses are represented as the bus system 550 in FIG. 3A.

[0054] The processor 520 may be an integrated circuit chip with signal processing capabilities, such as a general purpose processor, a digital signal processor (DSP), a programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., where the general purpose processor may be a microprocessor or any conventional processor, etc.

[0055] Memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 550 includes one or more storage devices physically remote from processor 520.

[0056] The memory 550 may include volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 of the present embodiment may include any suitable type of memory.

[0057] In some embodiments, memory 550 may store data to support various operations, examples of which include programs, modules, data structures, or a subset or superset thereof, as exemplified below.

[0058] The operating system 551 includes system programs for processing various basic system services and executing hardware-related tasks, such as a frame layer, a core library layer, and a drive layer, and is configured to realize various basic operations and process hardware-based tasks.

[0059] The network communication module 552 is configured to reach other computing devices via one or more (wired or wireless) network interfaces 530, example network interfaces 530 including Bluetooth, wireless compatibility (WiFi), and Universal Serial Bus (USB), etc.

[0060] In some embodiments, the audio encoding device provided in the embodiments of the present application may be realized in software, and FIG. 3A shows an audio encoding device 555 stored in a memory 550, which may be software in the form of a program or plug-in, and includes software modules such as a feature extraction module 5551, an encoding module 5552, and a signal encoding module 5553, where the feature extraction module 5551, the encoding module 5552, and the signal encoding module 5553 are used to realize audio encoding functions, and these modules are logical and can be arbitrarily combined or further divided according to the functions to be realized.

[0061] Referring to FIG. 3B, FIG. 3B is a schematic diagram illustrating the configuration of an electronic device 600 provided in an embodiment of the present application, and will be described taking as an example that the electronic device 600 is a terminal device. The electronic device 600 shown in FIG. 3B includes at least one processor 620, a memory 650, at least one network interface 630, and a user interface 640. The components in the electronic device 600 are coupled via a bus system 650. The memory 650 includes an operating system 651 and a network communication module 652. Note that the functions of the configuration in FIG. 3B are similar to those of the configuration in FIG. 3A. The audio encoding device provided in the embodiments of the present application may be realized by software, and Figure 3B shows an audio decoding device 655 stored in a memory 650, which may be software in the form of a program or plug-in, and includes software modules such as a signal decoding module 6551, a decoding module 6552, and a feature reconstruction module 6553, where the signal decoding module 6551, the decoding module 6552, and the feature reconstruction module 6553 are used to realize audio decoding functions, and these modules are logical and can be arbitrarily combined or further divided according to the functions to be realized.

[0062] As described above, the audio encoding method provided in the embodiments of the present application can be implemented by various types of electronic devices. Referring to Fig. 16A, Fig. 16A is a flowchart of the audio encoding method provided in the embodiments of the present application, and the audio encoding function is realized by the audio encoding method. The following description will be made with reference to Steps 11 to 13 shown in Fig. 16A.

[0063] In step 11, a feature extraction process is performed on the audio signal to obtain the audio features of the audio signal.

[0064] Here, the embodiment of the present application may invoke a third neural network (NN) based on the audio signal, and subsequently extract audio features from the audio signal through the third neural network to continuously perform feature extraction based on the significant audio features. Note that the embodiment of the present application is not limited to the structure of the third NN, and the third NN may be a convolutional neural network, a deep neural network, etc.

[0065] In some embodiments, step 11 can be implemented as follows: performing a causal convolution process on the audio signal to obtain causal convolution features, and performing a pooling process on the causal convolution features to obtain audio features of the audio signal.

[0066] In the field of audio coding and decoding, neural network (NN) operations such as causal convolution and pooling play an important role in processing audio signals and extracting features within them. In audio coding and decoding, causal convolution can be used to extract local features within an audio signal. By applying a convolution kernel (a type of learnable filter), convolution operations can be performed on the time dimension of an audio signal to capture patterns and resonances within the signal. Causal convolution can extract time- and frequency-domain features within an audio signal for tasks such as noise reduction, feature extraction, and signal separation. Pooling operations are used to reduce the time dimension of an audio signal, thereby reducing data complexity and computational effort. Causal convolution is a convolution operation used to process sequence data in neural network models. Because causal convolution operations are performed only in relation to the current and previous elements of the sequence, they preserve the time ordering of the sequence data and prevent backpropagation of information. Causal convolution ensures forward propagation of information within a sequence without backpropagating information by restricting the propagation direction of the convolution kernel. In specific neural network implementations, causal convolution is usually achieved by introducing edge padding or truncation before the convolution operation. In this way, the convolution calculation is performed only within the sequence, ensuring that the output at each point in time depends only on the input at that point and the previous one. For example, when processing a fixed-length sequence, special padding elements (0 or other discriminant values) can be added to the beginning of the sequence. In this way, these padding elements do not affect the actual convolution result because they are not within the range of the convolution kernel. In this way, the convolution kernel can only operate on the actual data of the sequence, maintaining the causal relationship of the information.In deep learning frameworks, causal convolution can be achieved by configuring parameters of the convolution layer, such as specifying the size and step length of the convolution kernel. Overall, causal convolution plays an important role in ensuring the fidelity of temporal causality in sequence data processing, and is widely applied in fields that require strict temporal ordering, such as natural language processing and speech recognition.

[0067] Pooling operations can generate more compact feature representations by sampling local regions of the input signal and aggregating information from those regions, such as the maximum and average values. In audio signals, pooling operations can help improve the robustness and generalization ability of networks and reduce the risk of overfitting. In the field of audio coding and decoding, operations such as convolution and pooling can achieve tasks such as feature extraction, encoding, and decoding of audio signals by constructing appropriate neural network configurations. These operations help improve the efficiency and quality of audio signal processing and expand the scope of application of audio coding and decoding technologies in fields such as audio processing, speech recognition, and music generation.

[0068] In step 12, at least one residual unit is used to perform coding-side residual processing on the audio features to obtain coding features of the audio signal.

[0069] In neural network models, the residual unit refers to a special structure for constructing residual networks (ResNets). The residual unit aims to solve the problems of vanishing gradients and exploding gradients during the training process of deep neural networks and help the network better learn features. The residual unit introduces skip connections, which means that the input is directly added to the output (i.e., residual processing) rather than simply passing it from layer to layer. This skip connection allows the network to learn the residual function, i.e., the difference between the input and the output, rather than learning a direct mapping relationship. This design makes the network easier to optimize and also helps alleviate the vanishing gradient problem.

[0070] According to the characteristics of the residual unit, the residual processing in step 12 calculates the residual of the audio feature at the encoding side, determines the residual of the audio feature as the coding feature, and uses it for subsequent signal coding. For example, the residual of the audio feature is obtained by adding the audio feature and the output of the residual unit, that is, the audio feature is used as the input of the residual unit, and after the audio feature is processed by the residual unit, the output of the residual unit is obtained, and the input of the residual unit and the output of the residual unit are added together to obtain the residual of the audio feature by utilizing the skip connection characteristics of the residual unit.

[0071] Here, by performing residual processing on audio features, comprehensive learning of audio features can be ensured based on the characteristics of residual processing, while better utilizing shallow layer information of coding features and avoiding leakage of shallow layer feature information of audio features.

[0072] In some embodiments, referring to FIG. 16B, FIG. 16B is another flowchart of the audio encoding method provided in the embodiments of the present application, and FIG. 16B shows that step 12 of FIG. 16A can be realized by steps 121 to 122.

[0073] In step 121, feature residual processing is performed on the audio features by at least one residual unit to obtain residual features of the audio signal.

[0074] Here, the feature residual processing in step 121 calculates the residual of the audio feature, and determines the residual of the audio feature as the residual feature of the audio signal, which is used for subsequent feature encoding.

[0075] In some embodiments, when the at least one residual unit is one residual unit, step 1021 can be realized by performing one residual processing on the audio features by one residual unit to obtain the residual features of the audio signal, where one residual processing of one residual unit is used to calculate one residual corresponding to the audio features at the encoding side.

[0076] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 121 can be realized by: performing one residual processing on the audio features by a first residual unit of the plurality of cascaded residual units, where the one residual processing of the first residual unit is used to calculate a residual of the audio features once and determine the residual of the audio features as a residual result of the first residual unit; outputting the residual result output from the first residual unit to a subsequent cascaded residual unit; and continuously performing one residual processing and outputting the residual result by the subsequent cascaded residual unit, where the one residual processing of the subsequent cascaded residual unit is used to calculate a residual of the residual result input to the subsequent cascaded residual unit; and using the residual result output from the last residual unit as the residual feature of the audio signal.

[0077] In some embodiments, the processing process of performing one residual processing on a residual unit is as follows. The k-th residual unit of a plurality of cascaded residual units performs a convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit, and adds the convolution result of the k-th residual unit and the input of the k-th residual unit to obtain the residual result output from the k-th residual unit. Here, k is a positive integer that sequentially increases, 1≦k≦J, J is the number of residual units. When k is 1, the input of the k-th residual unit is an audio feature. When k is not 1, the input of the k-th residual unit is a residual feature, which is the residual result output from the (k - 1)-th residual unit. That is, performing one residual processing on an audio feature by the first residual unit of a plurality of cascaded residual units can be realized by the first residual unit of a plurality of cascaded residual units performing a convolution process on the audio feature to obtain the convolution result of the first residual unit, and adding the convolution result of the first residual unit and the audio feature to obtain the residual result output from the first residual unit. The step of continuously performing one residual processing and outputting the residual result by the subsequent cascaded residual units can be realized by the j-th residual unit of a plurality of cascaded residual units performing a convolution process on the residual result output from the (j - 1)-th residual unit to obtain the convolution result of the j-th residual unit, adding the convolution result of the j-th residual unit and the residual result output from the (j - 1)-th residual unit to obtain the residual result output from the j-th residual unit, and outputting the residual result output from the j-th residual unit to the (j + 1)-th residual unit. Here, j is a positive integer that sequentially increases, 1<j<J, and J is the number of residual units.

[0078] Continuing from the above embodiments, each residual unit includes a dilated convolution operator. The k-th residual unit among a plurality of cascaded residual units performs a convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. This process can be realized by the k-th residual unit among a plurality of cascaded residual units performing a dilated convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. That is, through the dilated convolution operator included in the first residual unit, a dilated convolution process is performed on the audio features to obtain the dilated convolution result of the first residual unit. The j-th residual unit among a plurality of cascaded residual units performs a dilated convolution process on the residual result output from the (j - 1)-th residual unit through the dilated convolution operator included in the j-th residual unit to obtain the dilated convolution result of the j-th residual unit. Here, j is a positive integer that increases sequentially, 1 < j ≤ J, and J is the number of residual units. Each residual unit includes a dilated convolution operator with a specified dilation rate. Using dilated convolution operators with a progressive dilation rate is equivalent to extracting features input at different resolutions using different receptive fields, and a more comprehensive analysis of the data can be performed. After the convolution process by the dilated convolution operator with the dilation rate, each residual unit is added to the shallow features from the skip connection (i.e., the input of each residual unit). Thereby, by directly using the shallow feature information, the network can make full use of the shallow feature information during the learning process.

[0079] Following the above embodiment, each residual unit further includes at least one causal convolution operator in addition to the dilation and convolution operator, and performs a convolution process on the input of the kth residual unit to obtain the convolution result of the kth residual unit. After that, the kth residual unit performs a causal convolution process on the obtained dilation and convolution result through at least one causal convolution operator included in the kth residual unit, and the obtained causal convolution result is used as the convolution result of the kth residual unit. That is, the first residual unit performs a causal convolution process on the dilation and convolution result of the first residual unit through at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result output from the first residual unit. The jth residual unit performs a dilation and convolution process on the residual result output from the j-1th residual unit through a dilation and convolution operator included in the jth residual unit to obtain the dilation and convolution result of the jth residual unit. Then, the jth residual unit performs a causal convolution process on the dilation and convolution result of the jth residual unit through at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit. Note that each residual unit further includes at least one causal convolution operator, and continues to extract local information of the features input to the causal convolution operator through the causal convolution operator.

[0080] In neural network models, causal convolution is a special type of algorithm for processing time sequence data (audio signals are a type of time sequence data). It ensures that the output of a neural network depends only on the current and previous time steps, thereby maintaining temporal causality. In practical applications, causal convolution adjusts the size of the convolution kernel to ensure that the convolution kernel does not extend beyond the current time step. This effectively captures long-term dependencies within a time sequence while avoiding the problems of gradient vanishing or gradient explosion caused by future information. Causal convolution is particularly important in fields such as natural language processing, speech recognition, and time sequence prediction. This is because causal convolution follows the time order of the data, avoids the confusion of past information, and can effectively process and predict long-term sequence data. In tasks such as speech recognition and time sequence prediction, causal convolution has demonstrated excellent performance due to its ability to maintain time order.

[0081] In some embodiments, when applying grouped convolution to a dilated convolution operator included in the residual unit, performing a dilated convolution process on the audio features can be achieved by: performing a grouping process on input channels of the audio features to obtain a plurality of first groups, each first group including first elements (i.e., first feature values) corresponding to at least two channels in the audio features; and performing a dilated convolution process on the first elements in each first group. When applying grouped convolution to a causal convolution operator included in the residual unit, performing a causal convolution process on the obtained dilated convolution result can be achieved by: performing a grouping process on input channels of the dilated convolution result to obtain a plurality of second groups, each second group including second elements (i.e., second feature values) corresponding to at least two channels in the dilated convolution result; and performing a causal convolution process on the second elements in each second group.

[0082] For example, grouped convolution can be applied to the convolution operators (including dilated convolution operators and causal convolution operators) of the residual unit. Grouped convolution divides the input channels into multiple groups and performs convolution operations, associating only the input channels and output channels within each group. After the input channels are divided into multiple groups, the corresponding output channels are also divided into multiple groups. That is, the number of groups of input channels is the same as the number of groups of output channels, and after convolution within a group, only the input channels and output channels within each group are associated. Here, assume that a feature input to a certain convolution operator has four input channels and four output channels. If the number of groups is 1, each input channel is associated with four output channels. If the number of groups is 2, the four input channels are first divided into two groups, 0-1 and 2-3, and within each of the two groups, the input channels are associated with the output channels within that group. For example, input channels 0 to 1 in the first group are associated with output channels 0 to 1, and input channels 2 to 3 in the second group are associated with output channels 2 to 3. As shown in FIG. 6A, when the grouped convolution method is not employed, each input channel is associated with four output channels. As shown in FIG. 6B, when the grouped convolution method is not employed, the 0th output channel is associated only with the 0th to 1st input channels and not with the 2nd to 3rd input channels, and the 2nd output channel is associated only with the 2nd to 3rd input channels and not with the 0th to 1st input channels. In this way, by introducing grouped convolution, it is possible to avoid associating any input channel with all output channels, thereby reducing the number of connections and reducing complexity.

[0083] Following step 121, in step 122, feature coding processing is performed on the residual feature to obtain coded features of the audio signal.

[0084] In some embodiments, step 122 can be implemented by performing a convolution process on the residual features to obtain convolutional features, where the number of channels of the convolutional features is greater than the number of channels of the residual features, and performing a pooling process on the convolutional features to obtain coded features of the audio signal.

[0085] In some embodiments, the third neural network for audio coding includes a plurality of cascaded coding blocks, each coding block including at least one residual unit and a feature coding block, and step 12 can be realized by the plurality of cascaded coding blocks, and step 12 can be realized by: performing residual processing on the audio features by at least one residual unit in the plurality of cascaded coding blocks to obtain residual features of the audio signal, wherein the residual processing of the at least one residual unit in the plurality of cascaded coding blocks calculates a residual of the audio features and uses the calculated residual of the audio signal to determine the residual of the audio signal as residual features of the audio signal; and performing feature coding processing on the residual features via the feature coding blocks in the plurality of cascaded coding blocks to obtain coded features of the audio signal.

[0086] In some embodiments, performing residual processing on audio features by at least one residual unit in at least one of a plurality of cascaded encoding blocks to obtain residual features of an audio signal is to perform residual processing on audio features by at least one residual unit in the first encoding block of the plurality of cascaded encoding blocks, and output the residual result output from at least one residual unit in the first encoding block to the feature encoding block in the first encoding block. The residual processing of at least one residual unit in the first encoding block is used to calculate the residual of the audio features and determine the residual of the audio features obtained by calculation as the residual result output from at least one residual unit in the first encoding block. Also, performing residual processing on the encoded result output from the feature encoding block in the (i - 1)-th encoding block by at least one residual unit in the i-th encoding block of the plurality of cascaded encoding blocks, and output the residual result output from at least one residual unit in the i-th encoding block to the feature encoding block in the i-th encoding block. The residual processing of at least one residual unit in the i-th encoding block is used to calculate the residual of the encoded result output from the feature encoding block in the (i - 1)-th encoding block and determine the residual of the encoded result output from the feature encoding block in the (i - 1)-th encoding block obtained by calculation as the residual result output from at least one residual unit in the i-th encoding block. And it can be achieved by taking the residual result output from at least one residual unit in the last encoding block as the residual features of the audio signal. Here, i is a positive integer that increases sequentially, 1 < i ≤ I, and I is the number of encoding blocks. Obtaining the encoded features of the audio signal by performing feature encoding processing on the residual features through the feature encoding blocks in the plurality of cascaded encoding blocks can be achieved by performing feature encoding processing on the residual features through the feature encoding block in the last encoding block of the plurality of cascaded encoding blocks to obtain the encoded features of the audio signal.Here, the coding features are obtained by performing a convolution process on the residual features via the last coding block of multiple cascaded coding blocks to obtain convolution features, where the number of channels of the convolution features is greater than the number of channels of the residual features, and a pooling process on the convolution features to obtain coding features of the audio signal.

[0087] In step 13, a signal coding process is performed on the coded features of the audio signal to obtain an audio bitstream of the audio signal.

[0088] Here, in the field of digital signal processing, step 13 can be realized by performing a coding process based on the digital signal on the coding feature to obtain an audio bitstream of the audio signal.

[0089] In some embodiments, step 13 can be implemented by performing a quantization operation on the coded features to obtain index values ​​of the coded features, and performing an entropy coding operation on the index values ​​of the coded features to obtain an audio bitstream of the audio signal.

[0090] As shown in Fig. 7A, after performing audio encoding processing using the audio encoding method shown in Fig. 16A or 16B, that is, the audio signal x(n) passes through a third neural network 111 to obtain an encoding feature F(n), and then performs signal encoding (i.e., quantization encoding) on ​​the encoding feature F(n) to obtain an audio bitstream. The obtained audio bitstream is transmitted to the decoding side, and the received audio bitstream is decoded at the decoding side to obtain a synthesized audio signal x'(n). Referring to Fig. 17A, Fig. 17A is a flowchart of an audio decoding method provided in an embodiment of the present application, which realizes an audio decoding function. The following description will be made with reference to steps 21 to 23 shown in Fig. 17A.

[0091] In step 21, a signal decoding process is performed on the audio bitstream to obtain coding features corresponding to the audio bitstream.

[0092] Here, the audio bitstream is obtained by performing the above-described audio coding on the audio signal.

[0093] Note that signal decoding is the inverse process of signal encoding. Therefore, the values ​​generated in the decoding process are estimates of the values ​​in the encoding process. For example, the coding features generated in the decoding process are estimates of the coding features in the encoding process.

[0094] For example, performing a signal decoding process on an audio bitstream can be achieved by performing an entropy decoding process on the audio bitstream to obtain index values ​​corresponding to the audio bitstream, and performing an inverse quantization process on the index values ​​corresponding to the audio bitstream to obtain coding features corresponding to the audio bitstream, where the inverse quantization process is achieved by looking up a quantization table, which is a mapping table generated by quantization during the encoding process.

[0095] For example, a received audio bitstream is first subjected to entropy decoding and then referred to a quantization table (i.e., inverse quantization; the quantization table is a mapping table generated by quantization during the encoding process) to obtain coding features corresponding to the audio bitstream. Note that the process of decoding a received audio bitstream on the decoding side is the reverse process of the encoding process on the encoding side, so the values ​​generated during the decoding process are estimates of the values ​​in the encoding process. For example, the coding features generated in the decoding process are estimates of the coding features in the encoding process.

[0096] In step 22, decoding-side residual processing is performed on the coding features corresponding to the audio bitstream using at least one residual unit to obtain audio features corresponding to the audio bitstream.

[0097] Here, the audio features corresponding to the audio bitstream are estimated values ​​for the audio features on the encoding side.

[0098] According to the characteristics of the residual unit, the residual processing in step 22 is used to calculate the residual of the coded feature at the decoding side. For example, the residual of the coded feature is obtained by adding the coded feature and the output of the residual unit at the decoding side, that is, the coded feature is taken as the input of the residual unit, and after the coded feature is processed by the residual unit at the decoding side, the output of the residual unit is obtained, and by utilizing the characteristics of the skip connection of the residual unit, the input of the residual unit at the decoding side and the output of the residual unit are added to obtain the residual of the coded feature.

[0099] In some embodiments, referring to FIG. 17B, FIG. 17B is another flowchart of the audio decoding method provided in an embodiment of the present application, and FIG. 17B shows that step 22 of FIG. 17A can be realized by steps 221 to 222.

[0100] In step 221, a feature decoding process is performed on the coded features corresponding to the audio bitstream to obtain residual features corresponding to the audio bitstream.

[0101] For example, feature decoding is the reverse process of feature encoding, and performs feature decoding processing on the encoded features to obtain residual features (a type of estimate) corresponding to the audio bitstream. That is, the residual features corresponding to the audio bitstream are estimates of the residual features on the encoding side. Because the encoding and decoding processes are reverse processes, the residual features corresponding to the audio bitstream are not residual results obtained by residual calculation on the decoding side. After obtaining the residual features corresponding to the audio bitstream, residual calculation can be performed on the residual features corresponding to the audio bitstream. In an embodiment of the present application, a first neural network is called, and feature decoding processing is performed on the encoded features corresponding to the audio bitstream via the first neural network to obtain the residual features corresponding to the audio bitstream.

[0102] In some embodiments, step 221 can be implemented by performing a convolution operation on the coding features corresponding to the audio bitstream to obtain convolution features, where the number of channels of the convolution features is smaller than the number of channels of the coding features corresponding to the audio bitstream, and performing an upsampling operation on the convolution features to obtain residual features corresponding to the audio bitstream.

[0103] In the field of audio coding and decoding, upsampling operations are used to improve the resolution of feature maps (i.e., convolutional features) to more accurately reconstruct audio signals. Upsampling involves interpolation or other forms of upsampling techniques to generate higher-resolution feature maps, which helps to better restore the original details and characteristics of the audio signal during the decoding process. In audio decoding, neural network techniques such as convolution, pooling, and upsampling can be used to effectively extract useful features, reduce computational complexity, and more accurately reconstruct the original content of the audio signal. These techniques are of great significance in improving the performance and efficiency of audio decoding and help promote the development and application of audio coding and decoding technology.

[0104] Of course, before step 221, causal convolution is further performed on the coding features corresponding to the audio bitstream to obtain the coding features after the causal convolution, and step 22 is performed based on the coding features after the causal convolution. That is, feature decoding processing is performed on the coding features after the causal convolution to obtain the residual features corresponding to the audio bitstream.

[0105] In step 222, feature residual processing is performed on the residual features corresponding to the audio bitstream by at least one residual unit to obtain audio features corresponding to the audio bitstream.

[0106] Here, the feature residual processing in step 222 is used to calculate the residual of the residual feature corresponding to the audio bitstream at the decoding side, and determine the residual of the residual feature as the audio feature corresponding to the audio bitstream, where the residual feature corresponding to the audio bitstream is an estimate for the residual feature at the encoding side. Because the encoding process and the decoding process are inverse processes, the residual feature corresponding to the audio bitstream is not the residual result obtained by residual calculation at the decoding side. After obtaining the residual feature corresponding to the audio bitstream, the residual unit at the decoding side can perform residual calculation on the residual feature corresponding to the audio bitstream to obtain the audio feature corresponding to the audio bitstream.

[0107] Here, by performing residual processing on the residual features corresponding to the audio bitstream, comprehensive learning of the residual features can be ensured, while better utilizing the shallow layer information of the residual features and avoiding leakage of shallow layer feature information.

[0108] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 222 can be realized by performing one residual processing on the residual features by one residual unit to obtain audio features corresponding to the audio bitstream, where one residual processing of one residual unit is used to calculate one residual corresponding to the residual features at the decoding side.

[0109] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 222 can be realized by: performing residual processing on the residual features by a first residual unit of the plurality of cascaded residual units, where the single residual processing of the first residual unit is used to calculate a residual of the residual features once and determine the residual of the residual features as a residual result of the first residual unit; outputting the residual result output from the first residual unit to a subsequent cascaded residual unit, and continuously performing single residual processing and outputting the residual result by the subsequent cascaded residual unit, where the single residual processing of the subsequent cascaded residual unit is used to calculate a residual of the residual result input to the subsequent cascaded residual unit; and using the residual result output from the last residual unit as the audio feature corresponding to the audio bitstream.

[0110] In some embodiments, the processing process of performing one residual processing on a residual unit is as follows. The k-th residual unit among a plurality of cascaded residual units performs a convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit, and adds the convolution result of the k-th residual unit and the input of the k-th residual unit to obtain a residual result output from the k-th residual unit. Here, k is a positive integer that sequentially increases, 1≦k≦J, J is the number of residual units. When k is 1, the input of the k-th residual unit is a residual feature. When k is not 1, the input of the k-th residual unit is a residual feature that is the residual result output from the (k - 1)-th residual unit. That is, performing one residual processing on a residual feature can be realized by the first residual unit among a plurality of cascaded residual units performing a convolution process on the residual feature to obtain the convolution result of the first residual unit, and adding the convolution result of the first residual unit and the residual feature to obtain a residual result output from the first residual unit. Continuing to perform one residual processing and outputting a residual result by subsequent cascaded residual units can be realized by the j-th residual unit among a plurality of cascaded residual units performing a convolution process on the residual result output from the (j - 1)-th residual unit to obtain the convolution result of the j-th residual unit, adding the convolution result of the j-th residual unit and the residual result output from the (j - 1)-th residual unit to obtain a residual result output from the j-th residual unit, and outputting the residual result output from the j-th residual unit to the (j + 1)-th residual unit. Here, j is a positive integer that sequentially increases, 1<j<J, and J is the number of residual units.

[0111] Continuing from the above embodiments, each residual unit includes a dilated convolution operator. The k-th residual unit among a plurality of cascaded residual units performs a convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. This process can be realized by the k-th residual unit among a plurality of cascaded residual units performing a dilated convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. That is, through the dilated convolution operator included in the first residual unit, a dilated convolution process is performed on the residual features to obtain the dilated convolution result of the first residual unit. The j-th residual unit among a plurality of cascaded residual units performs a dilated convolution process on the residual result output from the (j - 1)-th residual unit through the dilated convolution operator included in the j-th residual unit to obtain the dilated convolution result of the j-th residual unit, where j is a positive integer that increases sequentially, 1 < j ≤ J, and J is the number of residual units.

[0112] Following the above embodiment, each residual unit further includes at least one causal convolution operator in addition to the dilation and convolution operator, and performs a convolution process on the input of the kth residual unit to obtain the convolution result of the kth residual unit. After that, the kth residual unit performs a causal convolution process on the obtained dilation and convolution result through at least one causal convolution operator included in the kth residual unit, and the obtained causal convolution result is used as the convolution result of the kth residual unit. That is, the first residual unit performs a causal convolution process on the dilation and convolution result of the first residual unit through at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result output from the first residual unit. A dilated convolution process is performed on the residual result output from the j-1th residual unit via a dilated convolution operator included in the jth residual unit to obtain the dilated convolution result of the jth residual unit. Then, a causal convolution process is performed on the dilated convolution result of the jth residual unit via at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit.

[0113] In some embodiments, when applying grouped convolution to a dilated convolution operator included in the residual unit, performing a dilated convolution process on the audio features can be achieved by: performing a grouping process on input channels of the residual features to obtain a plurality of first groups, each first group including first elements (i.e., first feature values) corresponding to at least two channels in the residual features; and performing a dilated convolution process on the first elements in each first group. When applying grouped convolution to a causal convolution operator included in the residual unit, performing a causal convolution process on the obtained dilated convolution results can be achieved by: performing a grouping process on input channels of the dilated convolution results to obtain a plurality of second groups, each second group including second elements (i.e., second feature values) corresponding to at least two channels in the dilated convolution result; and performing a causal convolution process on the second elements in each second group.

[0114] In some embodiments, the first neural network used for audio decoding includes a plurality of cascaded decoding blocks, each of which includes a feature decoding block and at least one residual unit; and step 22 can be realized by: performing cascaded feature decoding processing on encoded features corresponding to the audio bitstream via feature decoding blocks in the plurality of cascaded decoding blocks to obtain residual features corresponding to the audio bitstream; and correspondingly, performing residual processing on the residual features corresponding to the audio bitstream by at least one residual unit in the plurality of cascaded decoding blocks to obtain audio features corresponding to the audio bitstream, wherein the residual processing of the at least one residual unit in the plurality of cascaded decoding blocks calculates a residual of the residual features, and the calculated residual of the residual features is used to determine the audio feature corresponding to the audio bitstream.

[0115] In some embodiments, performing cascade-connected feature decoding processing on the encoded features corresponding to the audio bitstream through the feature decoding blocks in a plurality of cascade-connected decoding blocks to obtain the residual features corresponding to the audio bitstream can be achieved by performing one-time feature decoding processing on the encoded features corresponding to the audio bitstream through the feature decoding block in the first decoding block of the plurality of cascade-connected decoding blocks, outputting the decoding result output from the feature decoding block in the first decoding block to at least one residual unit in the first decoding block, performing one-time feature decoding processing on the residual result output from at least one residual unit in the (i - 1)-th decoding block through the feature decoding block in the i-th decoding block of the plurality of cascade-connected decoding blocks, outputting the decoding result output from the feature decoding block in the i-th decoding block to at least one residual unit in the i-th decoding block, and setting the decoding result output from the feature decoding block in the last decoding block as the residual features corresponding to the audio bitstream, where i is a positive integer that sequentially increases, 1 < i ≤ I, and I is the number of decoding blocks. Here, the decoding result output from the feature decoding block in the first decoding block is obtained by performing a convolution process on the encoded features corresponding to the audio bitstream through the feature decoding block in the first decoding block of the plurality of cascade-connected decoding blocks to obtain convolution features, where the number of channels of the convolution features is smaller than the number of channels of the encoded features, and performing an upsampling process on the convolution features.Here, the decoding result output from the feature decoding block in the i-th decoding block is obtained by performing a convolution process on the residual result output from at least one residual unit in the (i-1)-th decoding block via the feature decoding block in the i-th decoding block to obtain a convolution feature, where the number of channels of the convolution feature is smaller than the number of channels of the residual result output from the at least one residual unit, and a process of performing an upsampling process on the convolution feature to obtain a decoding result output from the feature decoding block in the i-th decoding block.

[0116] In some embodiments, performing residual processing on residual features corresponding to the audio bitstream by at least one residual unit in a plurality of cascaded decoding blocks to obtain audio features corresponding to the audio bitstream can be realized by performing residual processing on residual features corresponding to the audio bitstream by at least one residual unit in a last decoding block of the plurality of cascaded decoding blocks to obtain audio features corresponding to the audio bitstream, wherein the residual processing of the at least one residual unit in the last decoding block calculates a residual of the residual features, and the residual of the calculated residual features is used to determine the audio feature corresponding to the audio bitstream.

[0117] Following on from step 22 above, in step 2, a feature reconstruction process is performed on the audio features corresponding to the audio bitstream to obtain a synthesized audio signal corresponding to the audio bitstream.

[0118] Here, feature reconstruction is the reverse process of feature extraction, and the function of data decompression is realized by increasing the dimension of audio features through feature reconstruction processing.

[0119] In some embodiments, step 23 can be implemented by performing an upsampling operation on audio features corresponding to the audio bitstream to obtain upsampled features, and performing a causal convolution operation on the upsampled features to obtain a synthesized audio signal corresponding to the audio bitstream.

[0120] As described above, the audio encoding method provided in the embodiment of the present application can be implemented by various types of electronic devices. Referring to Fig. 4A, Fig. 4A is a first flowchart of the audio encoding method according to the embodiment of the present application, which realizes an audio encoding function. The following description will be made with reference to steps 101 to 103 shown in Fig. 4A.

[0121] As shown in Figure 4B, Figure 4B is a flowchart of an audio encoding method provided in an embodiment of the present application, and Figure 4B shows that Figure 4B further includes step 104 before step 101 of Figure 4A. In step 104, a subband decomposition process is performed on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal. Then, step 101 is performed on the low-frequency subband signals obtained in step 104.

[0122] Note that the embodiments of the present application do not limit the bands of the low-frequency subband signal and the high-frequency subband signal. That is, the low-frequency subband signal and the high-frequency subband signal obtained by decomposition may be two subband signals obtained by equally dividing the band of the audio signal, or may be two subband signals obtained by unequally dividing the band of the audio signal. For example, when the effective bandwidth of the audio signal x(n) is 0 to 16 kHz, the low-frequency subband signal x LB (n) and high frequency subband signal x HB The effective bandwidths of (n) are 0 to 8 kHz and 8 to 16 kHz, respectively, and the low-frequency subband signal x LB (n) and high frequency subband signal x HBThe effective bandwidths of (n) may be 0 to 6 kHz and 6 to 16 kHz, respectively. The embodiments of the present application do not limit the number of divided bands. That is, two subband signals may be obtained by dividing the audio signal band equally or unevenly, or more than two subband signals, for example, three, four, or more.

[0123] Here, the audio signal includes a low-frequency portion and a high-frequency portion, and the low-frequency signal (i.e., low-frequency subband signal) is obtained by separating the low-frequency complexity from the audio signal of a specific sampling rate using a filter based on the characteristics of the audio signal, and the high-frequency signal (i.e., high-frequency subband signal) is obtained by separating the low-frequency portion from the audio signal of a specific sampling rate. For example, if the effective bandwidth of the audio signal x(n) is 0-16 kHz, the effective bandwidth of the low-frequency signal is 0-8 kHz, and the effective bandwidth of the high-frequency signal x(n) is 0-16 kHz. HB The effective bandwidth of (n) may be 6 to 16 kHz. The embodiments of the present application do not limit the division of the band of the audio signal. For example, the audio signal can be divided equally or unevenly to obtain uniform low-frequency and high-frequency signals.

[0124] As an example of acquiring an audio signal, the encoding side acquires an audio signal (also called an input signal) by calling a microphone built into the encoding side's terminal equipment to collect the audio signal in response to an audio collection command triggered by the sending side (e.g., a web conference organizer, a broadcaster, an initiator of a voice call, etc.).

[0125] After obtaining the audio signal, the QMF analysis filter divides the audio signal into low-frequency subband signals x LB (n) and high frequency subband signal x HB Since the low-frequency subband signals have a greater effect on audio coding than the high-frequency subband signals, differentiated signal processing can be performed on the low-frequency subband signals and the high-frequency subband signals in the subsequent process.

[0126] In some embodiments, step 104 can be implemented by: performing a sampling process on the audio signal to obtain a sampling signal, where the sampling signal includes a plurality of sample points obtained by sampling; performing a low-pass filtering process on the sampling signal to obtain a low-pass filtered signal; performing a down-sampling process on the low-pass filtered signal to obtain a low-frequency sub-band signal of the audio signal; performing a high-pass filtering process on the sampling signal to obtain a high-pass filtered signal; and performing a down-sampling process on the high-pass filtered signal to obtain a high-frequency sub-band signal of the audio signal.

[0127] Here, the audio signal is a continuous analog signal, the sampling signal is a discrete digital signal, and the sampling point is a sampling value obtained by sampling the audio signal.

[0128] In the field of digital signal processing, downsampling is used to reduce the sampling rate of an audio signal to reduce the amount of data, reduce system complexity, or meet specific application requirements, where the downsampling factor of the downsampling may be a multiple of 2, such as 2, 4, 8, etc.

[0129] For example, if the audio signal is an input signal with a sampling rate of Fs=32000Hz, the audio signal is sampled to obtain a sampled signal x(n) containing 640 sample points. The analysis filter (2 channels) in the QMF filter bank is called to perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, high-pass filtering on the sampled signal to obtain a high-pass filtered signal, and downsampling on the low-pass filtered signal to obtain a low-frequency subband signal x(n) of the audio signal.LB (n), and then downsampling is performed on the high-pass filtered signal to obtain the high-frequency subband signal x of the audio signal. HB (n) to obtain the low frequency subband signal x LB (n) and high frequency subband signal x HB The effective bandwidths of (n) are 0 to 8 kHz and 8 to 16 kHz, respectively, and the low-frequency subband signal x LB (n) and high frequency subband signal x HB The number of sample points for (n) is 320.

[0130] A QMF filter bank is a filter pair that includes an analysis filter and a synthesis filter. The QMF analysis filter can decompose an input signal with a sampling rate of Fs into two-pass signals with a sampling rate of Fs / 2, representing a QMF low-pass signal and a QMF high-pass signal, respectively. The restored low-pass and high-pass signals on the decoding side are synthesized by the QMF synthesis filter, resulting in a reconstructed signal with a sampling rate of Fs that corresponds to the input signal.

[0131] In the field of digital signal processing, an audio signal is first filtered by a filter (e.g., a low-frequency filter, a high-pass filter) to remove high-frequency components and aliasing interference in the audio signal, so that the downsampled audio signal does not lose necessary information. Then, a downsampling process is performed to retain sampling points at regular intervals from the filtered audio signal, thereby reducing the sampling rate of the audio signal.

[0132] In step 101, a feature extraction process is performed on the low-frequency sub-band signals of the audio signal to obtain the low-frequency features of the low-frequency sub-band signals.

[0133] Step 101 is similar to step 11, but whereas the processing target of the feature extraction processing in step 11 is an audio signal, the processing target of the feature extraction processing in step 101 is a low-frequency subband signal.

[0134] Here, the embodiment of the present application invokes a third neural network (NN) based on the low-frequency subband signal, and extracts low-frequency features from the low-frequency subband signal through the third neural network, thereby enabling subsequent and continuous feature extraction based on the important low-frequency features. Note that the embodiment of the present application is not limited to the structure of the third NN, and the third NN may be a convolutional neural network, a deep neural network, etc.

[0135] In some embodiments, step 101 can be achieved by performing a causal convolution process on low-frequency subband signals of the audio signal to obtain causal convolution features, and performing a pooling process on the causal convolution features to obtain low-frequency features of the low-frequency subband signals.

[0136] For example, referring to the network configuration diagram of the third NN shown in Figure 11, the third NN includes one causal convolution layer and a preprocessing layer. First, a 16-channel causal convolution layer is invoked to expand the input tensor (i.e., low-frequency subband signal) into 16x320 causal convolution features, and then the 16x320 causal convolution features are preprocessed through a preprocessing layer. For example, after performing one convolution operation on the 16x320 causal convolution features, a pooling process with a coefficient of 2 is performed, and a parametric rectified linear unit (PReLU) is used as the activation function to generate a 16x160 tensor (i.e., low-frequency features).

[0137] In step 102, at least one residual unit is used to perform coding-side residual processing on the low-frequency features to obtain low-frequency coding features of the low-frequency sub-band signal.

[0138] Step 102 is similar to step 12, but the target of the residual processing in step 12 is audio features, whereas the target of the residual processing in step 102 is low-frequency features.

[0139] Here, by performing residual processing on the low-frequency features of the low-frequency subband signal, comprehensive learning of the low-frequency features can be ensured based on the characteristics of residual processing, while better utilizing the shallow layer information of the low-frequency features and avoiding the leakage of shallow layer feature information of the low-frequency features.

[0140] In some embodiments, referring to FIG. 4C, FIG. 4C is a flowchart of one of the audio encoding methods provided in the embodiments of the present application, and FIG. 4C shows that step 102 of FIG. 4A can be realized by steps 1021 to 1022.

[0141] In step 1021, feature residual processing is performed on the low-frequency features by at least one residual unit to obtain residual features of the low-frequency sub-band signal.

[0142] In some embodiments, if the at least one residual unit is one residual unit, step 1021 can be realized by performing one residual processing on the low-frequency features by the one residual unit to obtain residual features of the low-frequency subband signal.

[0143] In some embodiments, when at least one residual unit is a plurality of cascaded residual units, refer to FIG. 4D, which is a flowchart of an audio encoding method provided in an embodiment of the present application, and shows that step 1021 in FIG. 4C can be realized by steps 10211A to 10213A. In step 10211A, a first residual unit of the plurality of cascaded residual units performs one residual processing on the low-frequency features, and in 10212A, the residual result output from the first residual unit is output to the subsequent cascaded residual unit, and the subsequent cascaded residual unit continues to perform one residual processing and output the residual result, and in step 10213A, the residual result output from the last residual unit is used as the residual feature of the low-frequency subband signal.

[0144] As shown in FIG. 12A , when at least one residual unit used for the feature residual has four cascaded residual units, the first residual unit performs one residual processing on the low-frequency features and outputs the residual result output from the first residual unit to the second residual unit, the second residual unit performs one residual processing on the residual result output from the first residual unit and outputs the residual result output from the second residual unit to the third residual unit, the third residual unit performs one residual processing on the residual result output from the second residual unit and outputs the residual result output from the third residual unit to the fourth residual unit, and the fourth residual unit performs one residual processing on the residual result output from the third residual unit, to obtain a residual intermediate result.

[0145] In some embodiments, the processing process of the residual unit is as follows. The k-th residual unit of a plurality of cascaded residual units performs a convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit, and adds the convolution result of the k-th residual unit and the input of the k-th residual unit to obtain the residual result output from the k-th residual unit. Here, k is a positive integer that increases sequentially, 1≦k≦J, where J is the number of residual units. When k is 1, the input of the k-th residual unit is a low-frequency feature. When k is not 1, the input of the k-th residual unit is a residual feature that is the residual result output from the (k - 1)-th residual unit. That is, step 10211A can be realized by the first residual unit of a plurality of cascaded residual units performing a convolution process on the low-frequency feature to obtain the convolution result of the first residual unit, and adding the convolution result of the first residual unit and the low-frequency feature to obtain the residual result output from the first residual unit. Step 10212A can be realized by the j-th residual unit of a plurality of cascaded residual units performing a convolution process on the residual result output from the (j - 1)-th residual unit to obtain the convolution result of the j-th residual unit, adding the convolution result of the j-th residual unit and the residual result output from the (j - 1)-th residual unit to obtain the residual result output from the j-th residual unit, and outputting the residual result output from the j-th residual unit to the (j + 1)-th residual unit. Here, j is a positive integer that increases sequentially, 1<j<J, and J is the number of residual units.

[0146] Continuing from the above embodiments, each residual unit includes a dilated convolution operator. The k-th residual unit among a plurality of cascaded residual units performs a convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. This process can be realized by the k-th residual unit among a plurality of cascaded residual units performing a dilated convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. That is, through the dilated convolution operator included in the first residual unit, a dilated convolution process is performed on low-frequency features to obtain the dilated convolution result of the first residual unit. The j-th residual unit among a plurality of cascaded residual units performs a dilated convolution process on the residual result output from the (j - 1)-th residual unit through the dilated convolution operator included in the j-th residual unit to obtain the dilated convolution result of the j-th residual unit. Here, j is a positive integer that increases sequentially, 1 < j ≦ J, and J is the number of residual units. Each residual unit includes a dilated convolution operator with a specified dilation rate. Using dilated convolution operators with a progressive dilation rate is equivalent to extracting features input at different resolutions using different receptive fields, and a more comprehensive analysis of the data can be performed. After each residual unit is convolved by the dilated convolution operator with the dilation rate, it is added to the shallow features from the skip connection (i.e., the input of each residual unit). Thereby, by directly using the shallow feature information, the network can make full use of the shallow feature information during the learning process.

[0147] Following the above embodiment, each residual unit further includes at least one causal convolution operator in addition to the dilation and convolution operator, and performs a convolution process on the input of the kth residual unit to obtain the convolution result of the kth residual unit. After that, the kth residual unit performs a causal convolution process on the obtained dilation and convolution result through at least one causal convolution operator included in the kth residual unit, and the obtained causal convolution result is used as the convolution result of the kth residual unit. That is, the first residual unit performs a causal convolution process on the dilation and convolution result of the first residual unit through at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result output from the first residual unit. The jth residual unit performs a dilation and convolution process on the residual result output from the j-1th residual unit through a dilation and convolution operator included in the jth residual unit to obtain the dilation and convolution result of the jth residual unit. Then, the jth residual unit performs a causal convolution process on the dilation and convolution result of the jth residual unit through at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit. Note that each residual unit further includes at least one causal convolution operator, and continues to extract local information of the features input to the causal convolution operator through the causal convolution operator.

[0148] In neural network models, causal convolution is a special type of algorithm for processing time sequence data (audio signals are a type of time sequence data). It ensures that the output of a neural network depends only on the current and previous time steps, thereby maintaining temporal causality. In practical applications, causal convolution adjusts the size of the convolution kernel to ensure that the convolution kernel does not extend beyond the current time step. This effectively captures long-term dependencies within a time sequence while avoiding the problems of gradient vanishing or gradient explosion caused by future information. Causal convolution is particularly important in fields such as natural language processing, speech recognition, and time sequence prediction. This is because causal convolution follows the time order of the data, avoids the confusion of past information, and can effectively process and predict long-term sequence data. In tasks such as speech recognition and time sequence prediction, causal convolution has demonstrated excellent performance due to its ability to maintain time order.

[0149] In some embodiments, when applying grouped convolution to the dilated convolution operator included in the residual unit, performing the dilated convolution process on the low-frequency features can be achieved by performing a grouping process on input channels of the low-frequency features to obtain a plurality of first groups, each first group including a first element corresponding to at least two channels in the low-frequency features, and performing the dilated convolution process on the first element in each first group. When applying grouped convolution to the causal convolution operator included in the residual unit, performing the causal convolution process on the obtained dilated convolution result can be achieved by performing a grouping process on input channels of the dilated convolution result to obtain a plurality of second groups, each second group including a second element corresponding to at least two channels in the causal convolution result, and performing the causal convolution process on the second element in each second group.

[0150] Following step 1021, in step 1022, feature coding processing is performed on the residual feature to obtain low-frequency coded features of the low-frequency sub-band signal.

[0151] Here, a feature coding process is performed on the residual feature to obtain low-frequency coded features of the low-frequency sub-band signal, and then a signal coding process is performed based on the low-frequency coded features to obtain a low-frequency bitstream of the audio signal.

[0152] In some embodiments, step 1022 can be implemented by performing a convolution process on the residual features to obtain convolutional features, where the number of channels of the convolutional features is greater than the number of channels of the residual features, and performing a pooling process on the convolutional features to obtain low-frequency coding features of the low-frequency subband signals.

[0153] For example, a third neural network (NN) is invoked based on the low-frequency subband signal, and after processing by the residual unit processing in the third neural network, residual features are obtained. Then, convolution processing is performed on the residual features through a convolution layer in the third neural network to increase the number of channels of the residual features. Finally, pooling processing is performed on the convolutional features through a pooling layer in the third neural network to obtain low-frequency encoded features of the low-frequency subband signal. Of course, the third neural network may further include a causal convolution layer, which performs causal convolution on the low-frequency encoded features through the causal convolution layer to obtain low-frequency encoded features after the causal convolution. Then, signal encoding processing is performed on the low-frequency encoded features of the low-frequency subband signal after the causal convolution to obtain a low-frequency bitstream of the audio signal.

[0154] In some embodiments, the third neural network for audio coding includes a plurality of cascaded coding blocks, each coding block including at least one residual unit and a feature coding block, and step 1021 and step 1022 are realized by the plurality of cascaded coding blocks. Referring to FIG. 4E, FIG. 4E is a flowchart of an audio coding method provided in an embodiment of the present application, and FIG. 4E shows that step 1021 in FIG. 4C can be realized by step 10211B, and step 1022 can be realized by step 10221B. In step 10211B, residual processing is performed on the low-frequency features by at least one residual unit in the plurality of cascaded coding blocks to obtain residual features of the low-frequency sub-band signal, and in step 10221B, feature coding processing is performed on the residual features via feature coding blocks in the plurality of cascaded coding blocks to obtain low-frequency coding features of the low-frequency sub-band signal.

[0155] In some embodiments, step 10211B can be realized by performing residual processing on low-frequency features by at least one residual unit in the first encoded block of a plurality of cascaded encoded blocks, outputting the residual result output from at least one residual unit in the first encoded block to the feature encoding block in the first encoded block, performing residual processing on the encoded result output from the feature encoding block in the (i - 1)-th encoded block by at least one residual unit in the i-th encoded block of the plurality of cascaded encoded blocks, outputting the residual result output from at least one residual unit in the i-th encoded block to the feature encoding block in the i-th encoded block, and using the residual result output from at least one residual unit in the last encoded block as the residual feature of the low-frequency subband signal, where i is a positive integer that increases sequentially, 1 < i ≤ I, and I is the number of encoded blocks. Step 10221B can be realized by performing feature encoding processing on the residual feature through the feature encoding block in the last encoded block of the plurality of cascaded encoded blocks to obtain the low-frequency encoding feature of the low-frequency subband signal. Here, the low-frequency encoding feature is a process of performing convolution processing on the residual feature through the last encoded block of the plurality of cascaded encoded blocks to obtain a convolution feature, where the number of channels of the convolution feature is larger than the number of channels of the residual feature, and a process of performing pooling processing on the convolution feature to obtain the low-frequency encoding feature of the low-frequency subband signal.

[0156] For example, as shown in FIG. 11, after invoking a third neural network based on a low-frequency subband signal, low-frequency features (16x160 tensors obtained after preprocessing in FIG. 11) are obtained through the third neural network. The third neural network includes four cascaded coding blocks with different downsampling factors (Down_factors). Each coding block includes one residual block (including at least one residual unit), one convolutional layer, and one pooling layer. Each residual block includes four residual units (Residual Units) based on dilation convolution (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer is used to double the number of input channels, and the activation function may be PReLU, thereby ensuring data volume and avoiding data loss. The pooling layer is a pooling operation including the Down_factor, completing the downsampling and achieving data compression. The Down_factors of the four coding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four coding blocks is set to 32, 64, 128, and 256, respectively. After being processed by the four coding blocks, the input 16x160 tensor is converted into 32x80, 64x20, 128x5, and 256x1 tensors, respectively.For example, residual processing is performed on the low-frequency features (i.e., 16×160 tensor) through the residual block in the first coding block, and the residual result output from the residual block in the first coding block is output to the feature coding block in the first coding block. After processing by the feature coding block (including one convolution layer and one pooling layer) in the first coding block, the coding result of the feature coding block in the first coding block (i.e., 32×80 tensor) is obtained, and the coding result of the feature coding block in the first coding block (i.e., 32×80 tensor) is output to the second coding block, and the second coding block Residual processing is performed on the coding result (i.e., a 32x80 tensor) of the feature coding block in the first coding block via the residual block in the block, the residual result output from the residual block in the second coding block is output to the feature coding block in the second coding block, and after processing by the feature coding block in the second coding block (including one convolutional layer and one pooling layer), the coding result (i.e., a 64x20 tensor) of the feature coding block in the second coding block is obtained, and the coding result (i.e., a 64x20 tensor) of the feature coding block in the first coding block is output to the third coding block. The above processing is performed sequentially, and the output of the last coding block is used as the low-frequency coding feature.

[0157] Following step 102, in step 103, a signal coding process is performed on the low-frequency coding features of the low-frequency sub-band signal to obtain a low-frequency bitstream of the audio signal.

[0158] Step 103 is similar to step 13, but the processing target of the signal encoding process in step 13 is the encoded features, whereas the processing target of the signal encoding process in step 103 is the low-frequency encoded features.

[0159] Here, in the field of digital signal processing, step 103 can be realized by performing a digital signal-based coding process on the low-frequency coding features to obtain a low-frequency bitstream of the audio signal.

[0160] In some embodiments, step 103 can be implemented by performing a quantization process on the low-frequency coded features to obtain index values ​​of the low-frequency coded features, and performing an entropy coding process on the index values ​​of the low-frequency coded features to obtain a low-frequency bitstream of the audio signal.

[0161] For example, the low-frequency coding feature F of the low-frequency subband signal LB For (n), scalar quantization (quantizing each component separately) and entropy coding methods can be used. In addition, the embodiments of the present application are not limited to the combination of vector quantization (combining multiple adjacent components and jointly quantizing them as one vector) and entropy coding techniques, and the high-frequency bit stream and low-frequency bit stream obtained by encoding are transmitted to the decoding side, and the high-frequency bit stream and low-frequency bit stream are decoded on the decoding side.

[0162] Continuing with the description below, Fig. 4B shows that after step 104, steps 105 to 106 are included.

[0163] In step 105, high frequency analysis processing is performed on the high frequency subband signal to obtain high frequency coding features of the high frequency subband signal.

[0164] Since low-frequency subband signals have a greater impact on audio coding than high-frequency subband signals, differentiated signal processing is performed on the low-frequency subband signals and the high-frequency subband signals to reduce the feature dimension of the high-frequency features below that of the low-frequency features. For example, the feature dimension of the low-frequency features is 56, and the feature dimension of the high-frequency features is 8. Here, high-frequency analysis processing is used to reduce the dimension of the high-frequency subband signals and achieve the function of data compression. Here, the high-frequency coding features are features that represent the high-frequency subband signals, and the feature dimension of the high-frequency coding features is smaller than that of the high-frequency subband signals.

[0165] Referring to Fig. 4F, Fig. 4F is a flowchart of an audio encoding method provided in an embodiment of the present application, and Fig. 4F shows that step 105 of Fig. 4B can be realized by step 1051A. In step 1051A, a fourth neural network is invoked to perform feature extraction processing based on the fourth neural network on the high-frequency subband signal to obtain high-frequency encoding features of the high-frequency subband signal, where the number of channels of the fourth neural network model is smaller than the number of channels of the third neural network, and the third neural network is used to extract low-frequency encoding features from the low-frequency subband signal.

[0166] Step 1051A will be described below. Step 1051A can be realized by performing feature extraction processing on high frequency subband signals of an audio signal to obtain high frequency features of the high frequency subband signals, and performing coding-side residual processing on the high frequency features using at least one residual unit to obtain high frequency coding features.

[0167] Here, we introduce another fourth NN structure similar to the third NN to generate a low-dimensional feature vector (i.e., high-frequency encoded features of high-frequency subband signals). Because high-frequency subband signals are less important to quality than low-frequency subband signals, the structure of the fourth NN for high-frequency subband signals does not need to be as complex as the third NN.

[0168] Note that the processing process of residual processing in step 1051A is the same as the processing process of step 102. For example, using at least one residual unit to perform residual processing on high-frequency features to obtain high-frequency encoding features includes: using at least one residual unit to perform feature residual processing on high-frequency features to obtain residual features of high-frequency sub-band signals; and performing feature encoding processing on the residual features to obtain high-frequency encoding features of high-frequency sub-band signals.

[0169] Wherein, if the at least one residual unit is one residual unit, the one residual unit performs one residual processing on the high-frequency features to obtain the residual features of the high-frequency sub-band signal; if the at least one residual unit is a plurality of cascaded residual units, the first residual unit of the plurality of cascaded residual units performs one residual processing on the high-frequency features, and the residual result output from the first residual unit is output to the subsequent cascaded residual units, and the subsequent cascaded residual units continue to perform one residual processing and output the residual result, and the residual result output from the last residual unit is the residual feature of the high-frequency sub-band signal.

[0170] In some embodiments, the processing process of the residual unit is as follows: a kth residual unit of the multiple cascaded residual units performs a convolution process on the input of the kth residual unit to obtain a convolution result of the kth residual unit, and adds the convolution result of the kth residual unit and the input of the kth residual unit to obtain a residual result output from the kth residual unit, where k is an increasing positive integer, 1≦k≦J, and J is the number of residual units, and when k is 1, the input of the kth residual unit is a residual feature, and when k is not 1, the input of the kth residual unit is a residual feature that is the residual result output from the k−1th residual unit.

[0171] Continuing with the above embodiment, each residual unit includes a dilation and convolution operator, and the kth residual unit of the multiple cascaded residual units continues to perform a convolution process on the input of the kth residual unit to obtain a convolution result of the kth residual unit, which can be realized by performing a dilation and convolution process on the input of the kth residual unit of the multiple cascaded residual units to obtain a convolution result of the kth residual unit.

[0172] Following the above embodiment, each residual unit further includes at least one causal convolution operator in addition to the dilation convolution operator, and performs a convolution process on the input of the kth residual unit to obtain the convolution result of the kth residual unit. After that, the at least one causal convolution operator included in the kth residual unit performs a causal convolution process on the obtained dilation convolution result, and the obtained causal convolution result is used as the convolution result of the kth residual unit.

[0173] In some embodiments, when applying grouped convolution to the dilated convolution operator included in the residual unit, performing the dilated convolution process on the high frequency features can be achieved by performing a grouping process on input channels of the high frequency features to obtain a plurality of first groups, each first group including first elements corresponding to at least two channels in the high frequency features, and performing the dilated convolution process on the first elements in each first group. When applying grouped convolution to the causal convolution operator included in the residual unit, performing the causal convolution process on the obtained dilated convolution result can be achieved by performing a grouping process on input channels of the dilated convolution result to obtain a plurality of second groups, each second group including second elements corresponding to at least two channels in the dilated convolution result, and performing the causal convolution process on the second elements in each second group.

[0174] In some embodiments, performing a feature encoding process on the residual features to obtain high-frequency encoded features of the high-frequency subband signals can be achieved by performing a convolution process on the residual features to obtain convolutional features, where the number of channels of the convolutional features is greater than the number of channels of the residual features, and performing a pooling process on the convolutional features to obtain high-frequency encoded features of the high-frequency subband signals.

[0175] In some embodiments, the fourth neural network for audio coding includes a plurality of cascaded coding blocks, each coding block including at least one residual unit and a feature coding block, and step 1051A is realized by the plurality of cascaded coding blocks, and step 1051A can be realized by performing residual processing on high-frequency features by at least one residual unit in the plurality of cascaded coding blocks to obtain residual features of high-frequency subband signals, and performing feature coding processing on the residual features via feature coding blocks in the plurality of cascaded coding blocks to obtain high-frequency coding features of the high-frequency subband signals.

[0176] In some embodiments, at least one residual unit in a plurality of cascaded encoding blocks performs residual processing on high-frequency characteristics to obtain residual characteristics of a high-frequency sub-band signal. Specifically, at least one residual unit in the first encoding block of the plurality of cascaded encoding blocks performs residual processing on high-frequency characteristics, and outputs the residual result from at least one residual unit in the first encoding block to the feature encoding block in the first encoding block. At least one residual unit in the i-th encoding block of the plurality of cascaded encoding blocks performs residual processing on the encoded result output from the feature encoding block in the (i - 1)-th encoding block, and outputs the residual result from at least one residual unit in the i-th encoding block to the feature encoding block in the i-th encoding block. The residual result output from at least one residual unit in the last encoding block is used as the residual characteristics of the high-frequency sub-band signal. Here, i is a positive integer that sequentially increases, 1 < i ≤ I, and I is the number of encoding blocks. Through the feature encoding blocks in the plurality of cascaded encoding blocks, feature encoding processing is performed on the residual characteristics to obtain the high-frequency encoding characteristics of the high-frequency sub-band signal. This can be achieved by performing feature encoding processing on the residual characteristics through the feature encoding block in the last encoding block of the plurality of cascaded encoding blocks to obtain the high-frequency encoding characteristics of the high-frequency sub-band signal. Here, the high-frequency encoding characteristics are obtained by performing convolution processing on the residual characteristics through the last encoding block of the plurality of cascaded encoding blocks to obtain convolution characteristics, where the number of channels of the convolution characteristics is larger than the number of channels of the residual characteristics, and then performing pooling processing on the convolution characteristics to obtain the high-frequency encoding characteristics of the high-frequency sub-band signal.

[0177] Referring to Fig. 4F, Fig. 4F is a flowchart of an audio encoding method provided in an embodiment of the present application, and Fig. 4F shows that step 105 of Fig. 4B can be realized by step 1051B. In step 1051B, a band extension process is performed on the high-frequency subband signal to obtain high-frequency encoding features of the high-frequency subband signal, where the feature dimension of the high-frequency encoding features is lower than the feature dimension of the low-frequency encoding features.

[0178] For example, since high-frequency subband signals are less critical to quality than low-frequency subband signals, they can be compressed in a different way: by band extension (reconstructing a wideband speech signal from a band-limited narrowband speech signal), the high-frequency subband signals can be quickly compressed and the high-frequency coding features of the high-frequency subband signals can be extracted.

[0179] In some embodiments, step 1051B can be realized by performing a frequency domain transform process based on a plurality of sample points included in the high frequency subband signal to obtain transform coefficients corresponding to each of the plurality of sample points; dividing the transform coefficients corresponding to each of the plurality of sample points into a plurality of subbands; performing an averaging process on the transform coefficients included in each subband to obtain an average energy corresponding to the subband; and determining the subband spectral envelope corresponding to each of the plurality of subbands as a high frequency encoding feature of the high frequency subband signal.

[0180] Note that the frequency domain transformation method in the embodiments of the present application includes a modified discrete cosine transform (MDCT), a discrete cosine transform (DCT), a fast Fourier transform (FFT), etc., and the embodiments of the present application are not limited to the frequency domain transformation method. The averaging process in the embodiments of the present application includes an arithmetic mean and a geometric mean, and the embodiments of the present application are not limited to the averaging process method.

[0181] In some embodiments, performing a frequency domain transform process based on a plurality of sample points included in the high frequency subband signal to obtain transform coefficients corresponding to each of the plurality of sample points includes obtaining a reference high frequency subband signal of a reference audio signal, where the reference audio signal is an audio signal adjacent to the audio signal, and performing a discrete cosine transform process on the plurality of sample points included in the high frequency subband signal based on the plurality of sample points included in the reference high frequency subband signal and the plurality of sample points included in the high frequency subband signal to obtain transform coefficients corresponding to each of the plurality of sample points included in the high frequency subband signal.

[0182] In some embodiments, the process of performing the geometric mean operation on the transform coefficients in each subband is as follows: the sum of the squares of the transform coefficients corresponding to the sample points in each subband is determined, and the ratio of the sum of the squares to the number of sample points in the subband is determined as the average energy corresponding to each subband.

[0183] As an example, a high-frequency subband signal x containing 320 points HB A modified discrete cosine transform (MDCT) is invoked on (n) to generate 320 MDCT coefficients (i.e., transform coefficients corresponding to each of the multiple sample points included in the high-frequency subband signal). Specifically, in the case of 50% overlap, the high-frequency data of the (n+1)th frame (i.e., the reference audio signal) and the high-frequency data of the nth frame (i.e., the audio signal) are merged (combined), and a 640-point MDCT is calculated to obtain the first 320 MDCT coefficients.

[0184] The 320 MDCT coefficients are divided into N subbands (i.e., the transform coefficients corresponding to each of the sample points are divided into multiple subbands), where a subband is a group of adjacent MDCT coefficients. The 320 MDCT coefficients can be divided into eight subbands. For example, the 320 points can be evenly distributed, i.e., each subband contains the same number of points. Of course, the embodiment of the present application cannot divide the 320 points unevenly. For example, the low-frequency subbands contain fewer MDCT coefficients (higher frequency resolution), and the high-frequency subbands contain more MDCT coefficients (lower frequency resolution).

[0185] According to the Nyquist sampling theorem (which states that to restore the original signal from a sampled signal without distortion, the sampling frequency must be at least twice the highest frequency of the original signal. If the sampling frequency is less than twice the highest frequency of the spectrum, aliasing noise will occur in the signal spectrum, but if it is more than twice the highest frequency, aliasing noise will not occur in the spectrum), the 320 MDCT coefficients described above represent a spectrum from 8 to 16 kHz. However, in ultra-wideband voice communications, a spectrum up to 16 kHz is not necessarily required. For example, when setting a spectrum up to 14 kHz, only the first 240 MDCT coefficients need to be considered, and the number of subbands can be accordingly limited to six.

[0186] For each subband, the average energy of all MDCT coefficients in the current subband (i.e., averaging is performed on the transform coefficients included in each subband) is calculated as the subband spectral envelope (a spectral envelope is a smooth curve that passes through each major peak point of the spectrum). For example, if the MDCT coefficients included in the current subband are x(n), where n = 1, 2, ..., 40, the average energy Y = ((x(1)² + x(2)² + ... + x(40)²) / 40) is calculated by the geometric mean. If the 320 MDCT coefficients are divided into eight subbands, eight subband spectral envelopes are obtained, and the feature vectors F of the high-frequency subband signals from which these eight subband spectral envelopes were generated are HB (n), i.e., high frequency coding features.

[0187] Following on from step 105, in step 106, the high frequency encoded features are subjected to signal encoding processing to obtain a high frequency bitstream of the audio signal.

[0188] Here, in the field of digital signal processing, step 106 can be realized by performing a digital signal-based encoding process on the high-frequency encoded features to obtain a high-frequency bitstream of the audio signal.

[0189] In some embodiments, step 106 can be implemented by performing a quantization process on the high frequency encoded features to obtain index values ​​of the high frequency encoded features, and performing an entropy encoding process on the index values ​​of the high frequency encoded features to obtain a high frequency bitstream of the audio signal.

[0190] For example, the high-frequency coding feature F of the high-frequency subband signal LBFor (n), scalar quantization (quantizing each component individually) and entropy coding methods can be used. In addition, the embodiments of the present application are not limited to a combination of vector quantization (combining multiple adjacent components and jointly quantizing them as one vector) and entropy coding techniques, and the high-frequency bit stream and the high-frequency bit stream obtained by encoding are transmitted to the decoding side, and the high-frequency bit stream and the high-frequency bit stream are decoded on the decoding side.

[0191] As mentioned above, the audio decoding method provided in the embodiment of the present application can be implemented by various types of electronic devices. Referring to Figure 5A, Figure 5A is a flowchart of the audio decoding method provided in the embodiment of the present application, which realizes the audio decoding function. The audio decoding method and the above-mentioned audio encoding method are reverse processes, and will be described below with reference to the steps shown in Figure 5A.

[0192] In step 201, a signal decoding process is performed on the low-frequency bitstream to obtain low-frequency coding features corresponding to the low-frequency bitstream.

[0193] Here, the low-frequency bit stream is obtained by performing the above-mentioned audio coding on the low-frequency sub-band signal of the audio signal. Note that step 201 is similar to step 21, and only the processing target is different.

[0194] For example, after a low-frequency bitstream is encoded using the audio encoding method shown in FIG. 4A, the low-frequency bitstream obtained by encoding is transmitted to a decoding side. After receiving the low-frequency bitstream, the decoding side performs signal decoding processing on the low-frequency bitstream to obtain low-frequency coding features corresponding to the low-frequency bitstream.

[0195] Note that signal decoding is the reverse process of signal encoding.

[0196] For example, performing a signal decoding process on the low-frequency bitstream can be achieved by performing an entropy decoding process on the low-frequency bitstream to obtain index values ​​corresponding to the low-frequency bitstream, and performing an inverse quantization process on the index values ​​corresponding to the low-frequency bitstream to obtain low-frequency coding features corresponding to the low-frequency bitstream, where the inverse quantization process is achieved by looking up a quantization table, which is a mapping table generated by quantization during the encoding process.

[0197] As an example, the received bitstreams (high-frequency bitstream and low-frequency bitstream) are first entropy-decoded and then the quantization table is referenced (i.e., inverse quantization, the quantization table is a mapping table generated by quantization during the encoding process) to obtain an estimate of the low-frequency feature vector F′. LB (n), i.e., obtain low-frequency coding features corresponding to the low-frequency bitstream. Note that the process of decoding the received bitstream at the decoding side is the reverse process of the encoding process at the encoding side, so the values ​​generated during the decoding process are estimates for the values ​​in the encoding process. For example, the low-frequency coding features generated during the decoding process are estimates for the low-frequency coding features in the encoding process.

[0198] In step 202, decoding-side residual processing is performed on the low-frequency coding features corresponding to the low-frequency bitstream using at least one residual unit to obtain low-frequency features corresponding to the low-frequency bitstream.

[0199] Step 202 is similar to step 22, but the residual processing target in step 22 is the coded features, whereas the residual processing target in step 202 is the low-frequency coded features.

[0200] In some embodiments, referring to FIG. 5B, FIG. 5B is a flowchart of an audio decoding method provided in an embodiment of the present application, and FIG. 5B shows that step 202 of FIG. 5A can be realized by steps 2021 to 2022.

[0201] In step 2021, a feature decoding process is performed on the low-frequency encoded features corresponding to the low-frequency bitstream to obtain residual features corresponding to the low-frequency bitstream.

[0202] For example, feature decoding is the reverse process of feature encoding, and performs feature decoding processing on low-frequency encoded features to obtain residual features (a type of estimate) corresponding to the low-frequency bitstream. In the embodiment of the present application, a first neural network is invoked, and feature decoding processing is performed on the low-frequency encoded features corresponding to the low-frequency bitstream via the first neural network, thereby obtaining residual features corresponding to the low-frequency bitstream.

[0203] In some embodiments, step 2021 can be implemented by performing a convolution process on low-frequency encoded features corresponding to the low-frequency bitstream to obtain convolutional features, where the number of channels of the convolutional features is smaller than the number of channels of the low-frequency encoded features corresponding to the low-frequency bitstream, and performing an upsampling process on the convolutional features to obtain residual features corresponding to the low-frequency bitstream.

[0204] In the field of audio coding and decoding, upsampling operations are used to improve the resolution of feature maps (i.e., convolutional features) to more accurately reconstruct audio signals. Upsampling involves interpolation or other forms of upsampling techniques to generate higher-resolution feature maps, which helps to better restore the original details and characteristics of the audio signal during the decoding process. In audio decoding, neural network techniques such as convolution, pooling, and upsampling can be used to effectively extract useful features, reduce computational complexity, and more accurately reconstruct the original content of the audio signal. These techniques are of great significance in improving the performance and efficiency of audio decoding and help promote the development and application of audio coding and decoding technology.

[0205] Of course, before step 2021, causal convolution is further performed on the low-frequency coding features corresponding to the low-frequency bitstream to obtain the low-frequency coding features after the causal convolution, and step 202 is performed based on the low-frequency coding features after the causal convolution. That is, feature decoding processing is performed on the low-frequency coding features after the causal convolution to obtain the residual features corresponding to the low-frequency bitstream.

[0206] In step 2022, feature residual processing is performed on the residual features corresponding to the low-frequency bitstream by at least one residual unit to obtain low-frequency features corresponding to the low-frequency bitstream.

[0207] Here, by performing residual processing on the residual features corresponding to the low-frequency bitstream, comprehensive learning of the residual features can be ensured, while at the same time, shallow layer information of the residual features can be better utilized and leakage of shallow layer feature information can be avoided.

[0208] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 2022 can be realized by performing one residual processing on the residual features by one residual unit to obtain low-frequency features corresponding to the low-frequency bitstream.

[0209] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 2022 can be realized by: performing one residual processing on the residual features by a first residual unit of the plurality of cascaded residual units; outputting the residual result output from the first residual unit to a subsequent cascaded residual unit; continuing to perform one residual processing and output the residual result by the subsequent cascaded residual unit; and using the residual result output from the last residual unit as the low-frequency features corresponding to the low-frequency bitstream.

[0210] In some embodiments, the processing process of the residual unit is as follows. The k-th residual unit of a plurality of cascaded residual units performs a convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit, and adds the convolution result of the k-th residual unit and the input of the k-th residual unit to obtain the residual result output from the k-th residual unit. Here, k is a positive integer that sequentially increases, 1≦k≦J, J is the number of residual units. When k is 1, the input of the k-th residual unit is the residual feature. When k is not 1, the input of the k-th residual unit is the residual feature that is the residual result output from the (k - 1)-th residual unit. That is, performing one residual process on the residual feature can be realized by the first residual unit of a plurality of cascaded residual units performing a convolution process on the residual feature to obtain the convolution result of the first residual unit, and adding the convolution result of the first residual unit and the residual feature to obtain the residual result output from the first residual unit. The step of continuously performing one residual process and outputting the residual result by the subsequent cascaded residual units can be realized by the j-th residual unit of a plurality of cascaded residual units performing a convolution process on the residual result output from the (j - 1)-th residual unit to obtain the convolution result of the j-th residual unit, adding the convolution result of the j-th residual unit and the residual result output from the (j - 1)-th residual unit to obtain the residual result output from the j-th residual unit, and outputting the residual result output from the j-th residual unit to the (j + 1)-th residual unit. Here, j is a positive integer that sequentially increases, 1<j<J, and J is the number of residual units.

[0211] Continuing from the above embodiments, each residual unit includes a dilated convolution operator. The k-th residual unit among a plurality of cascaded residual units performs a convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. This process can be realized by the k-th residual unit among a plurality of cascaded residual units performing a dilated convolution process on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. That is, through the dilated convolution operator included in the first residual unit, a dilated convolution process is performed on the residual features to obtain the dilated convolution result of the first residual unit. The j-th residual unit among a plurality of cascaded residual units performs a dilated convolution process on the residual result output from the (j - 1)-th residual unit through the dilated convolution operator included in the j-th residual unit to obtain the dilated convolution result of the j-th residual unit, where j is a positive integer that increases sequentially, 1 < j ≦ J, and J is the number of residual units.

[0212] Following the above embodiment, each residual unit further includes at least one causal convolution operator in addition to the dilation and convolution operator, and performs a convolution process on the input of the kth residual unit to obtain the convolution result of the kth residual unit. After that, the kth residual unit performs a causal convolution process on the obtained dilation and convolution result through at least one causal convolution operator included in the kth residual unit, and the obtained causal convolution result is used as the convolution result of the kth residual unit. That is, the first residual unit performs a causal convolution process on the dilation and convolution result of the first residual unit through at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result output from the first residual unit. A dilated convolution process is performed on the residual result output from the j-1th residual unit via a dilated convolution operator included in the jth residual unit to obtain the dilated convolution result of the jth residual unit. Then, a causal convolution process is performed on the dilated convolution result of the jth residual unit via at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit.

[0213] In some embodiments, when applying grouped convolution to a dilated convolution operator included in the residual unit, performing a dilated convolution process on the low-frequency features can be achieved by: performing a grouping process on input channels of the residual features to obtain a plurality of first groups, each first group including a first element corresponding to at least two channels in the residual features; and performing a dilated convolution process on the first element in each first group. When applying grouped convolution to a causal convolution operator included in the residual unit, performing a causal convolution process on the obtained dilated convolution result can be achieved by: performing a grouping process on input channels of the dilated convolution result to obtain a plurality of first groups, each first group including a second element corresponding to at least two channels in the dilated convolution result; and performing a causal convolution process on the second element in each first group.

[0214] In some embodiments, the first neural network used for audio decoding includes a plurality of cascaded decoding blocks, each of which includes a feature decoding block and at least one residual unit. Referring to FIG. 5C , FIG. 5C is a flowchart of an audio decoding method provided in an embodiment of the present application. FIG. 5C illustrates that step 2021 in FIG. 5B can be implemented by step 20211A, and step 2022 can be implemented by step 20221A. In step 20211A, cascaded feature decoding processes are performed on low-frequency encoded features corresponding to the low-frequency bitstream via feature decoding blocks in the plurality of cascaded decoding blocks to obtain residual features corresponding to the low-frequency bitstream. Correspondingly, in step 20221A, residual processing is performed on the residual features corresponding to the low-frequency bitstream by at least one residual unit in the plurality of cascaded decoding blocks to obtain low-frequency features corresponding to the low-frequency bitstream.

[0215] In some embodiments, step 20211A performs a single feature decoding process on the low-frequency encoded features corresponding to the low-frequency bitstream through the feature decoding block in the first decoding block of a plurality of cascaded decoding blocks, and outputs the decoding result output from the feature decoding block in the first decoding block to at least one residual unit in the first decoding block; performs a single feature decoding process on the residual result output from at least one residual unit in the (i - 1)-th decoding block through the feature decoding block in the i-th decoding block of the plurality of cascaded decoding blocks, and outputs the decoding result output from the feature decoding block in the i-th decoding block to at least one residual unit in the i-th decoding block; and realizes the decoding result output from the feature decoding block in the last decoding block as the residual feature corresponding to the low-frequency bitstream, where i is a positive integer that sequentially increases, 1 < i ≤ I, and I is the number of decoding blocks. Here, the decoding result output from the feature decoding block in the first decoding block is obtained by performing a convolution process on the low-frequency encoded features corresponding to the low-frequency bitstream through the feature decoding block in the first decoding block of the plurality of cascaded decoding blocks to obtain convolution features, where the number of channels of the convolution features is smaller than the number of channels of the low-frequency encoded features, and performing an upsampling process on the convolution features to obtain the decoding result output from the feature decoding block in the first decoding block.Here, the decoding result output from the feature decoding block in the i-th decoding block is obtained by performing a convolution process on the residual result output from at least one residual unit in the (i-1)-th decoding block via the feature decoding block in the i-th decoding block to obtain a convolution feature, where the number of channels of the convolution feature is smaller than the number of channels of the residual result output from the at least one residual unit, and a process of performing an upsampling process on the convolution feature to obtain a decoding result output from the feature decoding block in the i-th decoding block.

[0216] In some embodiments, step 20221A can be realized by performing residual processing on residual features corresponding to the low-frequency bitstream by at least one residual unit in the last decoding block of multiple cascaded decoding blocks to obtain low-frequency features corresponding to the low-frequency bitstream.

[0217] For example, as shown in FIG. 14, the low-frequency decoding feature (F′ in FIG. 14) corresponding to the low-frequency bitstream is LBAfter invoking the first NN based on (n), i.e., a 256x1 tensor, low-frequency features (the output 16x160 tensor in Figure 14) are obtained through the first NN. The first NN includes four cascaded decoding blocks with different upsampling factors (Up_factors). Here, each decoding block includes one convolutional layer, one upsampling layer, and one residual block (including at least one residual unit), where the convolutional layer is used to halve the number of input channels and the activation function may be PReLU. The upsampling layer includes a specific Up_factor and is used to complete the upsampling. Each residual block includes four residual units (Residual Units) based on dilated convolution (the input and output feature dimensions of the residual units do not change). Here, the Up_factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels of the four decoding blocks are set to 128, 64, 32, and 16, respectively. After being processed by the four decoding blocks, the input 256×1 tensor is converted into 128×5, 64×20, 32×80, and 16×160 tensors, respectively.For example, the feature decoding block (including one convolutional layer and one upsampling layer) in the first decoding block performs one feature decoding process on the low-frequency encoded features (i.e., 256×1 tensor), and the decoding result output from the feature decoding block in the first decoding block is output to the residual block in the first decoding block. After processing by the residual block in the first decoding block, the residual result (i.e., 128×5 tensor) of the feature decoding block in the first decoding block is obtained, and the residual result (i.e., 128×5 tensor) of the residual block in the first decoding block is multiplied by 2. The residual result of the first decoding block (i.e., a 128x5 tensor) is output to the 3rd decoding block, and the residual result of the first decoding block (i.e., a 16x160 tensor) is output to the 4th decoding block. The residual result of the first decoding block (i.e., a 16x160 tensor) is output to the 5th decoding block. The residual result of the first decoding block (i.e., a 16x160 tensor) is output to the 3rd decoding block. The above process is repeated sequentially, and the output of the last decoding block is the low-frequency feature (i.e., a 16x160 tensor output from the 4th decoding block).

[0218] Following on from step 202 above, in step 20, a feature reconstruction process is performed on the low-frequency features corresponding to the low-frequency bitstream to obtain low-frequency sub-band signals corresponding to the low-frequency bitstream.

[0219] Step 203 is similar to step 23, but the processing target of the feature reconstruction processing in step 203 is low-frequency features, whereas the processing target of the feature reconstruction processing in step 23 is audio features.

[0220] Here, feature reconstruction is the reverse process of feature extraction, and the function of data decompression is realized by increasing the dimension of low-frequency features through feature reconstruction processing.

[0221] In some embodiments, step 203 can be implemented by performing an upsampling operation on low-frequency features corresponding to the low-frequency bitstream to obtain upsampled features, and performing a causal convolution operation on the upsampled features to obtain low-frequency subband signals corresponding to the low-frequency bitstream.

[0222] Referring to FIG. 5D, FIG. 5D is a flowchart of an audio decoding method provided in an embodiment of the present application, and FIG. 5D shows that FIG. 5A further includes step 204 to step 206.

[0223] In step 204, a signal decoding process is performed on the high frequency bitstream to obtain high frequency coding features corresponding to the high frequency bitstream.

[0224] Here, the high-frequency bitstream is obtained by performing audio coding on the high-frequency subband signals of the audio signal, where signal decoding and signal coding are inverse processes, for example, performing signal decoding processing on the high-frequency bitstream can be realized by performing entropy decoding processing on the high-frequency bitstream to obtain index values ​​corresponding to the high-frequency bitstream, and performing inverse quantization processing on the index values ​​corresponding to the high-frequency bitstream to obtain high-frequency coding features corresponding to the high-frequency bitstream.

[0225] In step 205, a high-frequency reconstruction process is performed on the high-frequency coding features corresponding to the high-frequency bitstream to obtain a high-frequency sub-band signal corresponding to the high-frequency bitstream.

[0226] Here, high-frequency reconstruction and high-frequency analysis are inverse processes. Note that the feature reconstruction in step 23 is different from the high-frequency reconstruction in step 205. The feature reconstruction in step 23 is the inverse process of the feature extraction in step 11, and is used to reconstruct a reconstructed audio signal based on audio features corresponding to the audio bitstream. Meanwhile, the high-frequency reconstruction in step 205 is used to reconstruct a high-frequency subband signal corresponding to the high-frequency bitstream based on high-frequency coding features corresponding to the high-frequency bitstream.

[0227] In some embodiments, step 205 can be implemented by invoking a second neural network model to perform feature reconstruction processing on the high-frequency encoded features to obtain high-frequency sub-band signals corresponding to the high-frequency bitstream.

[0228] For example, when the encoding side calls the fourth NN to perform feature extraction on the high-frequency subband signal to obtain the high-frequency encoding feature, the decoding side performs feature reconstruction processing on the second NN high-frequency encoding feature to obtain the corresponding high-frequency subband signal, where the structure of the fourth NN corresponds to the structure of the second NN.

[0229] In some embodiments, step 205 performs an inverse process of band extension on the high frequency coded features to obtain high frequency subband signals corresponding to the high frequency bitstream.

[0230] For example, when band extension is performed on high frequency subband signals on the encoding side to obtain high frequency features, the inverse process of band extension is performed on the high frequency features on the decoding side to obtain corresponding high frequency subband signals.

[0231] In some embodiments, performing an inverse bandwidth extension process on high-frequency encoding features and obtaining high-frequency subband signals corresponding to the high-frequency bitstream includes: performing a frequency domain transform process based on a plurality of sample points included in the low-frequency subband signal to obtain transform coefficients corresponding to each of the plurality of sample points; performing a spectral replication process on transform coefficients of the latter half of the transform coefficients corresponding to each of the plurality of sample points to obtain reference transform coefficients of the reference high-frequency subband signal; performing a gain process on the reference transform coefficients of the reference high-frequency subband signal based on a subband spectral envelope corresponding to the high-frequency features to obtain reference transform coefficients after the gain process; and performing an inverse frequency domain transform process on the reference transform coefficients after the gain process to obtain the corresponding high-frequency subband signal.

[0232] Note that the frequency domain transformation method in the embodiments of the present application includes a modified discrete cosine transform (MDCT), a discrete cosine transform (DCT), a fast Fourier transform (FFT), etc., and the embodiments of the present application are not limited to the frequency domain transformation method.

[0233] In some embodiments, performing gain processing on reference transform coefficients of a reference high frequency subband signal based on a subband spectral envelope corresponding to high frequency features and obtaining reference transform coefficients after gain processing includes: dividing the reference transform coefficients of the reference high frequency subband signal into a plurality of subbands based on the subband spectral envelope corresponding to the high frequency features; determining, for any subband among the plurality of subbands, a first average energy corresponding to the subband in the subband spectral envelope and a second average energy corresponding to the subband; determining a gain coefficient based on a ratio between the first average energy and the second average energy; and multiplying each reference transform coefficient included in the subband by the gain coefficient to obtain reference transform coefficients after gain processing.

[0234] As an example, first, the low-frequency subband signal x' generated on the decoding side is LB Similarly to the encoding side, 640-point MDCT transform is performed on (n) to generate 320-point MDCT coefficients (i.e., MDCT coefficients of the low-frequency part). That is, frequency domain transform processing is performed based on multiple sample points included in the low-frequency subband signal to obtain transform coefficients corresponding to each of the multiple sample points.

[0235] Next, x′ LB The 320 MDCT coefficients generated from (n) are replicated to generate MDCT coefficients of the high frequency part (i.e., reference transform coefficients of the reference high frequency subband signal). A basic feature of the reference speech signal is that there are many harmonics in the low frequency part and few harmonics in the high frequency part. Therefore, to avoid excessive harmonics being included in the MDCT spectrum of the artificially generated high frequency part by simple replication, the last 160 points of the 320 MDCT coefficients on which the low frequency subband signal depends are used as the master, and the spectrum is replicated twice to generate reference values ​​of the MDCT coefficients of the 320 reference high frequency subband signal (i.e., reference transform coefficients of the reference high frequency subband signal). That is, spectrum replication processing is performed on the latter half of the transform coefficients corresponding to each of the multiple sample points to obtain the reference transform coefficients of the reference high frequency subband signal.

[0236] Next, the eight subband spectral envelopes previously obtained (i.e., the eight subband spectral envelopes obtained by referencing the quantization table, i.e., the subband spectral envelopes corresponding to the high-frequency features) are retrieved. These eight subband spectral envelopes correspond to the eight high-frequency subbands. The reference values ​​of the MDCT coefficients of the generated 320-point reference high-frequency subband signal are divided into eight reference high-frequency subbands (i.e., the reference transform coefficients of the reference high-frequency subband signal are divided into multiple subbands). For each subband, a gain (multiplication in the frequency domain) is applied to the reference values ​​of the MDCT coefficients of the generated 320-point reference high-frequency subband signal based on one high-frequency subband and its corresponding reference high-frequency subband. For example, a gain coefficient is calculated based on the average energy of the high-frequency subband (i.e., the first average energy) and the average energy of the corresponding reference high-frequency subband (the second average energy). The MDCT coefficients corresponding to each point in the corresponding reference high-frequency subband are then multiplied by the gain coefficient. This ensures that the energy of the virtually generated high-frequency MDCT coefficients during decoding approaches the original coefficient energy from the encoding side, restoring the original high-frequency signal as much as possible.

[0237] For example, suppose the average energy of the reference high-frequency subband (i.e., the subband divided from the reference values ​​of the 320 MDCT coefficients of the generated high-frequency partial signal) is Y_L, and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectral envelope decoded based on the bitstream) is Y_H. Then, calculate one gain coefficient a = sqrt(Y_H / Y_L), where sqrt() is the square root calculation function used to calculate the square root of (Y_H / Y_L). After obtaining the gain coefficient a, multiply the MDCT coefficient of each point in the reference high-frequency subband by a directly. The average energy of the MDCT coefficients (virtually generated) after gain processing is very close to the original energy on the encoding side.

[0238] Finally, we call the inverse MDCT transform to obtain the estimates of the high-frequency subband signals x′ HB(n) (i.e., high-frequency subband signals corresponding to high-frequency features). MDCT inverse transformation is performed on the 320 MDCT coefficients after gain processing to generate 640-point estimates, and the first 320 valid estimates are defined as x' HB (n).

[0239] In step 206, a subband synthesis process is performed on the low-frequency subband signal and the high-frequency subband signal to obtain a reconstructed audio signal.

[0240] For example, subband synthesis processing is the reverse process of subband decomposition processing, and at the decoding side, subband synthesis processing is performed on low-frequency subband signals and high-frequency subband signals to restore the audio signal, where the reconstructed audio signal is an estimate of the restored audio signal.

[0241] If the received audio bitstream is a low-frequency bitstream in a full-frequency bitstream, the audio decoding method of the embodiment of the present application is performed on the low-frequency bitstream to obtain a low-frequency subband signal (an estimated value of the low-frequency subband signal in the subband decomposition at the encoding side). Of course, the full-frequency bitstream also includes a high-frequency bitstream, and audio decoding is performed on the high-frequency bitstream to obtain a high-frequency subband signal (an estimated value of the high-frequency subband signal in the subband decomposition at the encoding side). Subband synthesis processing is performed on the low-frequency subband signal and the high-frequency subband signal to obtain a reconstructed audio signal.

[0242] In some embodiments, step 206 can be implemented by performing an upsampling operation on the low-frequency subband signals to obtain a low-pass filtered signal, performing an upsampling operation on the high-frequency subband signals to obtain a high-frequency filtered signal, and performing a filtering synthesis operation on the low-pass filtered signal and the high-frequency filtered signal to obtain a reconstructed audio signal.

[0243] For example, after obtaining the low-frequency sub-band signal and the high-frequency sub-band signal, the QMF synthesis filter performs sub-band synthesis on the low-frequency sub-band signal and the high-frequency sub-band signal to restore the audio signal.

[0244] In the following, an exemplary application example of the present embodiment in a practical application scenario will be described.

[0245] The embodiments of the present application can be applied to various audio scenarios such as voice calls, instant messaging, etc. The following description will be given taking voice calls as an example.

[0246] In the related art, the principle of speech coding is roughly as follows: speech coding can directly code speech waveform samples sample by sample, or based on the principle of human speech, extract relevant low-dimensional features, code the features on the encoding side, and reconstruct the speech signal based on these parameters on the decoding side.

[0247] The above coding principles are all derived from audio signal modeling, i.e., compression methods based on signal processing. To improve coding quality while ensuring audio coding efficiency compared to compression methods based on signal processing, embodiments of the present application provide a low-complexity, low-bit-rate neural network audio compression method (i.e., audio encoding method and audio decoding method). That is, based on the characteristics of the audio signal, important parts (low-frequency subband signals) are processed using neural network (NN) technology, and a lower-dimensional feature vector is obtained from the input low-frequency subband signals. Here, residual blocks are used within the neural network to perform residual processing, allowing for more comprehensive analysis of the data and improving coding quality. Furthermore, an operation similar to "blocking" is used within the residual blocks, reducing algorithm complexity and improving coding efficiency.

[0248] The embodiments of the present application are applicable to the voice communication link shown in Figure 6C. Taking a VoIP (Voice over Internet Protocol) conference system as an example, the voice encoding and decoding technology according to the embodiments of the present application is deployed in the encoding and decoding units to solve the basic functions of voice compression. The encoder is located in the upstream client 601, and the decoder is located in the downstream client 602. The upstream client collects the voice, performs pre-processing and enhancement, encoding, etc., and transmits the encoded bitstream to the downstream client 602 via the network. The downstream client 602 decodes, enhances, etc., and plays the decoded voice.

[0249] Considering forward compatibility (i.e., compatibility between new and existing encoders), a transcoder must be deployed at the backend (i.e., server) of the system to solve the interconnection problem between new and existing encoders. For example, consider a case where the sender (upstream client) is a new NN encoder and the receiver (downstream client) is a Public Switched Telephone Network (PSTN) (G.722). The backend must run an NN decoder to generate an audio signal, and then call a G.722 encoder to generate a specific bitstream to realize the transcoding function. This allows the receiver to correctly decode based on the specific bitstream.

[0250] The low-complexity, low-bit-rate neural network audio compression method provided in the embodiment of the present application will be described below, taking into account both the high-frequency and low-frequency parts.

[0251] Hereinafter, a low-complexity, low-bit-rate neural network audio compression method (realized by an audio encoding method and an audio decoding method) provided in an embodiment of the present application will be described with reference to FIG. 7B.

[0252] The encoding side performs the following processing.

[0253] For the input audio signal x(n) of the nth frame, the analysis filter is used to generate the low-frequency subband signal x LB (n) and high frequency subband signal x HB Decompose into (n).

[0254] Low frequency subband signal x LB For (n), a third NN is called to generate a low-dimensional feature vector F LB (n) and obtain the feature vector F LB The dimension of (n) is smaller than the dimension of the low-frequency subband signal, so the amount of data is reduced. For example, for each frame x LB For (n), a neural network (encoding part) is invoked to generate a lower-dimensional feature vector F LB (n). The embodiments of the present application are not limited to other NN structures, such as an autoencoder, a fully-connected (FC) network, a long short-term memory (LSTM) network, and a convolutional neural network (CNN) + LSTM. Here, a residual block is used within the neural network to process the residual, allowing for a more comprehensive analysis of the data and improving the coding quality. An operation similar to "blocking" is also used within the residual block, reducing the complexity of the algorithm and improving the coding effect.

[0255] High frequency subband signal x HB For (n), the high frequency subband signal x HB (n) is a feature vector F HBFor example, in the bandwidth extension technology based on speech signal analysis, high-frequency subband signals can be generated with a bit rate of only 1 to 2 kbps, and the high-frequency subband signals can be generated using the same NN structure as the low-frequency subband signals or a simpler network (for example, the output feature vector is the low-frequency feature vector F LB (n) can also be used.

[0256] The feature vector corresponding to the subband signal (i.e., F LB (n) and F HB (n)) is subjected to vector quantization or scalar quantization, and the quantized index values ​​are subjected to entropy coding. The bit streams obtained after coding (low-frequency bit stream and high-frequency bit stream) are transmitted to the decoding side.

[0257] The decoding side performs the following processing.

[0258] The decoder decodes the received bitstreams (low-frequency bitstream and high-frequency bitstream) and obtains the estimated low-frequency feature vector F'. LB (n) and the high frequency feature vector estimate F′ HB (n) are obtained respectively.

[0259] For the low-frequency part, the estimated value F′ of the low-frequency feature vector LB (n), and the first NN is called to obtain the low-frequency subband signal estimate x′ LB (n) is obtained. Here, the first neural network uses a residual block to process the residual, which can perform a more comprehensive analysis of the data and improve the coding quality. In addition, the residual block uses an operation similar to "blocking", which can reduce the complexity of the algorithm and improve the coding effect.

[0260] For the high frequency part, the estimated value F′ of the high frequency feature vector HB (n) and call the high-frequency reconstruction to obtain the estimate x′ of the high-frequency subband signal. HBGenerate (n).

[0261] Finally, the QMF synthesis filter is invoked to generate the reconstructed synthetic speech signal x'(n).

[0262] The low-complexity, low-bit-rate neural network audio compression method provided in the embodiment of the present application will now be described in detail.

[0263] In some embodiments, an audio signal with a sampling rate of Fs=32000 Hz is taken as an example (note that the method provided in the embodiments of the present application is also applicable to scenes with other sampling rates, including but not limited to 8000 Hz, 32000 Hz, and 48000 Hz). At the same time, assume that the frame length is set to 20 ms. Therefore, when Fs=32000 Hz, each frame contains 640 sample points.

[0264] The encoding side and the decoding side will be described in detail below with reference to the flowchart shown in FIG. 7B.

[0265] The process of the coding side of the low frequency part and the high frequency part is as follows.

[0266] For an audio signal with a sampling rate of Fs=32000 Hz, the input signal of the nth frame contains 640 sample points and is denoted as input signal x(n).

[0267] In step 11, the QMF analysis filter is invoked to perform the signal decomposition.

[0268] Call the QMF analysis filter (2-channel QMF) to downsample and extract the low-frequency subband signal x LB (n) and high frequency subband signal x HB (n) can be obtained by using the low-frequency subband signal x LB (n) and high frequency subband signal x HBThe effective bandwidths of (n) are 0 to 8 kHz and 8 to 16 kHz, respectively, and the low-frequency subband signal x LB (n) and high frequency subband signal x HB The number of sample points for (n) is 320.

[0269] In step 12, the third NN is invoked based on the low frequency subband signal.

[0270] Low frequency subband signal x LB Based on (n), a third NN is called to generate a lower-dimensional feature vector F LB (n) is generated. Note that x LB The dimension of (n) is 320, and F LB The dimension of (n) is 56, and in terms of the amount of data, the third NN plays the role of "dimensionality reduction" and realizes the function of data compression. LB Not limited to the dimension of (n), x LB It may have other dimensions smaller than (n).

[0271] The data compression process by the third NN will be specifically described below with reference to the network configuration diagram of the third NN shown in FIG.

[0272] First, we invoke a 16-channel causal convolution to expand the input tensor (i.e., vector) into a 16x320 tensor.

[0273] Next, preprocessing is performed on the 16x320 tensor. For example, after performing one convolution operation on the 16x320 tensor, a pooling operation with a coefficient of 2 is performed, and the activation function may be PReLU, to generate a 16x160 tensor.

[0274] Next, four different downsampling factor (Down_factor) coding blocks are cascaded. Here, each coding block includes one residual block, one convolutional layer, and one pooling layer. Here, each residual block includes four residual units (Residual Units) based on dilation convolution (the input and output feature dimensions of the residual units do not change). The convolutional layer is used to double the number of input channels, and the activation function may be PReLU, thereby ensuring data volume and avoiding data loss. The pooling layer is a pooling operation including the Down_factor, which completes downsampling and achieves data compression. Here, the Down_factors of the four coding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four coding blocks are set to 32, 64, 128, and 256, respectively. After being processed by four coding blocks, the input 16x160 tensor is converted into tensors of 32x80, 64x20, 128x5, and 256x1, respectively. Here, the embodiment of the present application does not limit the number of coding blocks, and it may be any positive integer, such as 2, 3, 4, 5, etc.

[0275] Here, we will further explain the residual unit. A residual unit is a type of module within a deep neural network. By introducing skip connections into the neural network, the neural network is more easily optimized during training and problems such as gradient vanishing or gradient explosion can be avoided. The core idea is to perform residual learning on the input within the module. That is, input information is transmitted directly to the output via a direct path, bypassing some layers, allowing the network to better utilize shallow layer feature information during training. Figure 12A is a schematic diagram showing the configuration of a residual block used in the encoding block of a third neural network. This residual block includes four residual units based on dilation convolution. Each residual unit includes a dilation convolution block with a specified dilation rate. That is, each dilation convolution block includes a convolution operator with a specified dilation rate (e.g., dilation rate = 3). In the present embodiment, the use of four dilated convolution blocks with progressive dilation rates corresponds to using different sensitive fields to extract input features at different resolutions, enabling a more comprehensive analysis of the data. After residual processing by the four dilated convolution blocks with specified dilation rates, the result is added with the input from the skip connection to obtain the output result of the residual block, which is then output to the convolution layer connected to the residual block.

[0276] Here, as shown in FIG. 12B, an arbitrary residual unit in FIG. 12A will be further described. Any residual unit may include a dilated convolution with a specified dilation rate (used to expand the sensitive field) and use PReLU as the activation function. Furthermore, one or more causal convolutions (used to extract local information) may be cascaded and use PReLU as the activation function. The convolution kernel size of the dilated convolution with the specified dilation rate may be 3, 5, 7, 9, etc., and the convolution kernel size of the causal convolution may be 1, 3, etc. The embodiments of the present application do not limit the convolution kernel size of the dilated convolution with the specified dilation rate or the causal convolution. Furthermore, the causal convolution or dilated convolution in the embodiments of the present application may be implemented by other specific convolution units with similar or equivalent functions.

[0277] Furthermore, a grouped convolution algorithm is introduced to reduce the complexity of the residual unit algorithm. Grouped convolution divides input channels into multiple groups and performs convolution operations, associating only input channels and output channels within each group. Assume there are 16 input channels and 32 output channels. If the number of groups is 1, each input channel is associated with 32 output channels. If the number of groups is 2, the 16 input channels are first divided into two groups, 0 to 7 and 8 to 15, and within each group, the input channels are associated with output channels within the group. For example, input channels 0 to 7 in the first group are associated with output channels 0 to 15, and input channels 8 to 15 in the second group are associated with output channels 16 to 31. For example, the 0th output channel is associated only with the 0th to 7th input channels and not with the 8th to 15th input channels, and the 25th output channel is associated only with the 8th to 15th input channels and not with the 0th to 7th input channels. In this comparison, the introduction of grouped convolutions can avoid associating any input channel with all output channels, reducing the number of connections and complexity. Of course, the greater the number of groups, the smaller the correlation between the input and output channels, which also affects the coding effect, so a larger number of groups is not necessarily better. In the present embodiment, the dilated convolutions included in the corresponding four residual blocks in the four coding blocks can be configured using different numbers of groups, and specific group number configurations are shown in Table 1. [Table 1]

[0278] Finally, we use the same causal convolution as in the preprocessing to generate a 56-dimensional feature vector F LB (n) can be output.

[0279] In step 13, the high frequency subband signal x HBHigh frequency analysis is performed on (n). The purpose of high frequency analysis is to extract the high frequency subband signal x HB (n) to extract key information and create a lower-dimensional feature vector F HB (n). Here, the embodiment of the present application is HB Not limited to the dimension of (n), x LB (n) may be smaller than F HB The dimension of (n) is F LB It must be smaller than the dimension (n).

[0280] In some embodiments, referring to step 12, another fourth NN structure similar to the third NN can be introduced to generate a low-dimensional feature vector. Because the high-frequency subband signals are less important to quality than the low-frequency subband signals, the NN structure for the high-frequency subband signals does not need to be as complex as the third NN. The fourth NN structure for the high-frequency subband signals shown in FIG. 13 is similar to the third NN structure, but compared to the third NN structure, the fourth NN has a significantly reduced number of channels.

[0281] However, for high frequency subband signals, although the data amount of the high frequency subband signals can be significantly reduced by the fourth NN structure shown in Fig. 13, the model complexity of the fourth NN structure is still relatively complex. Therefore, the embodiment of the present application proposes another method for compressing high frequency subband signals, namely, band extension (restoring a wideband speech signal from a band-limited narrowband speech signal). The application of band extension in the embodiment of the present application will be specifically described below.

[0282] High frequency subband signal x containing 320 points HBA Modified Discrete Cosine Transform (MDCT) is invoked on (n) to generate 320-point MDCT coefficients. Specifically, in the case of 50% overlap, the high-frequency data of the (n+1)th frame and the high-frequency data of the nth frame are merged (combined), and a 640-point MDCT is calculated to obtain 320-point MDCT coefficients.

[0283] The 320 MDCT coefficients are divided into N subbands, where a subband is a group of adjacent MDCT coefficients, and the 320 MDCT coefficients can be divided into eight subbands. For example, the 320 points can be evenly distributed, i.e., each subband contains the same number of points. Of course, in the present embodiment, the 320 points cannot be divided unevenly. For example, the low-frequency subbands contain fewer MDCT coefficients (higher frequency resolution), and the high-frequency subbands contain more MDCT coefficients (lower frequency resolution).

[0284] According to the Nyquist sampling theorem (which states that to restore the original signal from a sampled signal without distortion, the sampling frequency must be at least twice the highest frequency of the original signal. If the sampling frequency is less than twice the highest frequency of the spectrum, aliasing noise will occur in the signal spectrum, but if it is more than twice the highest frequency, aliasing noise will not occur in the spectrum), the 320 MDCT coefficients described above represent a spectrum from 8 to 16 kHz. However, in ultra-wideband voice communications, a spectrum up to 16 kHz is not necessarily required. For example, when setting a spectrum up to 14 kHz, only the first 240 MDCT coefficients need to be considered, and the number of subbands can be accordingly limited to six.

[0285] For each subband, the average energy of all MDCT coefficients in the current subband is calculated as the subband spectral envelope (a spectral envelope is a smooth curve that passes through each major peak point of the spectrum). For example, if the MDCT coefficients included in the current subband are x(n), where n = 1, 2, ..., 40, then the average energy Y = ((x(1)² + x(2)² + ... + x(40)²) / 40). If the 320 MDCT coefficients are divided into eight subbands, eight subband spectral envelopes are obtained, and the feature vector F of the high-frequency subband signal from which these eight subband spectral envelopes were generated is HB (n).

[0286] In short, using either of the two methods above (NN structure and band extension), the 320-dimensional high-frequency subband signal can be output as an 8-dimensional feature vector, which allows high-frequency information to be expressed with a small amount of data, greatly improving coding efficiency.

[0287] In step 14, quantization coding is performed.

[0288] Feature vector F of the low-frequency subband signal LB (n) and the feature vector F of the high frequency subband signal HB For both (n), scalar quantization (quantizing each component individually) and entropy coding methods can be performed. Also, the embodiments of the present application do not limit the combination of vector quantization (combining multiple adjacent components and jointly quantizing them as one vector) and entropy coding techniques.

[0289] After quantization coding of the feature vectors, the corresponding bitstream can be generated. Experimental results show that high-quality compression of 32 kHz ultra-wideband signals can be achieved at bit rates of 6-10 kbps.

[0290] The process at the decoding side of the low frequency part and the high frequency part is as follows.

[0291] In step 21, quantization and decoding is performed.

[0292] Quantization decoding is the reverse process of quantization encoding. The received bitstream (including high-frequency and low-frequency bitstreams) is first subjected to entropy decoding, and then an estimate of the low-frequency feature vector F′ is obtained by consulting the quantization table. LB (n), and the estimate of the feature vector of the high-frequency bitstream F′ HB Get (n).

[0293] In step 22, the estimate F′ of the feature vector of the low-frequency bitstream is LB Based on (n), the first NN is called.

[0294] First, the estimated feature vector F′ of the low-frequency bitstream is LB Based on (n), the first neural network shown in FIG. 14 is called to obtain the estimated value x′ of the low-frequency subband signal. LB (n) is generated. Here, the first NN is similar to the third NN, such as causal convolution, and the structure of the post-processing is similar to the structure of the pre-processing in the third NN. The specific process of the first NN is as follows:

[0295] First, we call causal convolution on the input tensor F′ LB (n) can be expanded from a 56×1 to a 256×1 tensor.

[0296] Next, four decoding blocks with different upsampling factors (Up_factors) are cascaded. Each decoding block includes one convolutional layer, one upsampling module, and one residual block. Here, the convolutional layer is used to halve the number of input channels, the upsampling module includes a specific Up_factor and is used to complete the upsampling, and the residual block includes four residual units (Residual Units) based on dilated convolution. The Up_factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels of the four decoding blocks is set to 128, 64, 32, and 16, respectively. After being processed by the four decoding blocks, the 256x1 tensor is converted into 128x5, 64x20, 32x80, and 16x160 tensors, respectively. Note that the embodiment of the present application is not limited to the number of decoding blocks, and may be any positive integer, such as 2, 3, 4, or 5.

[0297] Here, for the upsampling module with a specific Up_factor, a repeat operation can be adopted to complete the upsampling operation by repeated padding, thus saving complexity.

[0298] Here, the configuration of the four residual units based on the dilated convolution on the decoding side is similar to the configuration of the residual unit on the encoding side, including but not limited to the internal structure of the residual unit, the convolution kernel size, the dilation rate, etc. The configuration of the number of groups used in the dilated convolution of the decoding block is as shown in Table 2. Here, the decoding block adopts a larger number of groups, i.e., 2, in order to associate more input channels with more output channels and improve the quality of speech reconstruction. [Table 2]

[0299] Next, post-processing is performed on the 16x160 tensor output from the cascaded decoding block. For example, a Repeat operation with a factor of 2 is performed on the 16x160 tensor output from the cascaded decoding block to complete upsampling, and then a convolution operation is performed using PReLU as the activation function to generate a 16x320 tensor. Finally, we invoke causal convolution to transform the input 16x320 tensor into a 1x320 tensor to reconstruct the low-frequency subband signal.

[0300] In step 23, the estimate F' of the feature vector of the high frequency subband signal is HB High frequency reconstruction is performed on (n).

[0301] Similar to the high frequency analysis on the encoding side, there are two methods for high frequency reconstruction in the present embodiment.

[0302] The first realization of high-frequency reconstruction is the second neural network shown in Fig. 15, which corresponds to the first realization of high-frequency analysis on the encoding side (corresponding to the fourth NN structure shown in Fig. 13). The estimates F' of the feature vectors of the high-frequency subband signals are HB Based on (n), a second neural network is invoked to obtain the estimate x′ of the high frequency subband signal. HB Generate (n).

[0303] Here, the structure of the second neural network is the same as that of the first realization of high frequency analysis such as causal convolution (Figure 13), and the structure of the post-processing is the same as that of the pre-processing in the first realization of high frequency analysis. Here, the decoding block structure is symmetrical with the coding block on the coding side, in that the coding block on the coding side performs dilation convolution and then completes downsampling by pooling, while the decoding block on the decoding side completes upsampling by pooling and then performs dilation convolution.

[0304] The second realization of high-frequency reconstruction corresponds to the second realization of high-frequency analysis on the encoding side (corresponding to the bandwidth extension technique). The eight amplitude spectrum subband spectral envelopes decoded from the high-frequency bitstream, i.e., the high-frequency feature vector estimates F′ are HB Perform the following operations on (n).

[0305] First, the estimated value x' of the low-frequency subband signal generated on the decoding side LB Similarly to the encoding side, 640-point MDCT transformation is performed on (n) to generate 320-point MDCT coefficients (that is, MDCT coefficients in the low-frequency portion).

[0306] Next, x′ LB The 320 MDCT coefficients generated from (n) are replicated to generate MDCT coefficients for the high frequency portion. A basic feature of the reference speech signal is that there are many harmonics in the low frequency portion and few harmonics in the high frequency portion. Therefore, to avoid excessive harmonics being included in the MDCT spectrum of the artificially generated high frequency portion due to simple replication, the last 160 points of the 320 MDCT coefficients on which the low frequency subband depends are used as the master, and the spectrum is replicated twice to generate reference values ​​for the MDCT coefficients of the 320 high frequency subband signal.

[0307] Next, the eight subband spectral envelopes previously obtained (i.e., the eight subband spectral envelopes obtained by referencing the quantization table) are retrieved. These eight subband spectral envelopes correspond to eight high-frequency subbands. The reference values ​​of the MDCT coefficients of the generated 320-point high-frequency subband signals are divided into eight reference high-frequency subbands. A gain (multiplication in the frequency domain) is applied to the reference values ​​of the MDCT coefficients of the generated 320-point high-frequency subband signals based on one high-frequency subband and its corresponding reference high-frequency subband. For example, a gain coefficient is calculated based on the average energy of the high-frequency subband and the average energy of the corresponding reference high-frequency subband, and the MDCT coefficients corresponding to each point in the corresponding reference high-frequency subband are multiplied by the gain coefficient. This ensures that the energy of the virtually generated high-frequency MDCT coefficients during decoding approaches the original coefficient energy at the encoding side.

[0308] For example, suppose the average energy of the reference high-frequency subband (i.e., the subband divided from the reference values ​​of the 320 MDCT coefficients of the generated high-frequency partial signal) is Y_L, and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectral envelope decoded based on the bitstream) is Y_H. Then, calculate one gain coefficient a = sqrt(Y_H / Y_L), where sqrt() is the square root calculation function used to calculate the square root of (Y_H / Y_L). After obtaining the gain coefficient a, multiply the MDCT coefficient of each point in the reference high-frequency subband by a directly. The average energy of the MDCT coefficients (virtually generated) after the gain process is very close to the original average energy of the encoding side, restoring the original high-frequency signal as much as possible.

[0309] Finally, we call the inverse MDCT transform to obtain the estimates of the high-frequency subband signals x′ HB The inverse MDCT transform is performed on the 320 MDCT coefficients after gain processing to generate a 640-point estimate, and the first 320 valid estimates are then used as x' HB (n).

[0310] In step 24, the synthesis filter is invoked.

[0311] At the decoding side, the estimated value x' of the low frequency subband signal is LB (n) and the estimate x' of the high frequency subband signal HB After obtaining (n), we can generate the 640-point reconstructed signal x′(n) simply by upsampling and invoking the QMF synthesis filter.

[0312] In the embodiment of the present application, the optimal parameters can be obtained by collecting data and jointly training the relevant networks on the encoding and decoding sides. Users only need to prepare data and set the corresponding network structure, and after the training is completed in the backend, the trained model can be used.

[0313] As described above, the neural network-based audio encoding / decoding method provided in the embodiments of the present application organically combines signal decomposition, signal processing technology, and deep neural networks, thereby ensuring audio quality while significantly improving encoding efficiency compared to signal processing methods within an acceptable complexity range.

[0314] The audio encoding method or audio decoding method provided in the embodiments of the present application has been described above in relation to exemplary applications and implementations of terminal devices provided in the embodiments of the present application. The embodiments of the present application further provide an audio encoding device and an audio decoding device. In actual applications, each functional module in the audio encoding device and the audio decoding device can be cooperatively implemented by hardware resources such as a processor, communication resources (e.g., corresponding to communication methods for various methods such as optical cable and cellular), and memory of electronic devices (e.g., terminal devices, servers, server clusters, etc.). Figure 3A shows an audio encoding device 555 stored in memory 550, and Figure 3B shows an audio decoding device 655 stored in memory 650. These may be software in the form of a program or plug-in, for example, application software designed in a programming language such as software C / C++ or Java, or a dedicated software module in a large-scale software system, an application program interface, a plug-in, a cloud service, etc. Examples of different implementations will be described below.

[0315] Here, the audio encoding device 555 includes a series of modules, including a feature extraction module 5551, an encoding module 5552, and a signal encoding module 5553. Next, a description will be given of how the modules in the audio encoding device 555 provided in the embodiment of the present application cooperate to realize audio encoding.

[0316] The feature extraction module 5551 is configured to perform feature extraction processing on the audio signal to obtain audio features of the audio signal, the encoding module 5552 is configured to perform encoding-side residual processing on the audio features using at least one residual unit to obtain encoding features of the audio signal, and the signal encoding module 5553 is configured to perform signal encoding processing on the encoding features of the audio signal to obtain an audio bitstream of the audio signal.

[0317] Here, the audio decoding device 655 includes a series of modules, including a signal decoding module 6551, a decoding module 6552, and a feature reconstruction module 6553. Next, a solution in which each module in the audio decoding device 655 provided in the embodiment of the present application cooperates to realize audio decoding will be described.

[0318] The signal decoding module 6551 is configured to perform signal decoding processing on an audio bitstream to obtain coding features corresponding to the audio bitstream, where the audio bitstream is obtained by audio encoding an audio signal; the decoding module 6552 is configured to perform decoding-side residual processing on the coding features using at least one residual unit to obtain audio features corresponding to the audio bitstream; and the feature reconstruction module 6553 is configured to perform feature reconstruction processing on the audio features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0319] In some embodiments, the decoding module 6552 is further configured to perform: performing a feature decoding process on the coded features corresponding to the audio bitstream to obtain residual features corresponding to the audio bitstream; and performing, by the at least one residual unit, feature residual processing on the residual features corresponding to the audio bitstream to obtain audio features corresponding to the audio bitstream.

[0320] In some embodiments, when at least one residual unit is a plurality of cascaded residual units, the decoding module 6552 is further configured to perform a single residual processing on the residual features by the first residual unit of the plurality of cascaded residual units; output the residual result output from the first residual unit to subsequent cascaded residual units, and continue to perform a single residual processing and output of the residual result by the subsequent cascaded residual units; and use the residual result output from the last residual unit as the audio features corresponding to the audio bit stream.

[0321] In some embodiments, the decoding module 6552 is further configured to perform a convolution process on the residual features by the first residual unit of the plurality of cascaded residual units to obtain the convolution result of the first residual unit; add the convolution result of the first residual unit and the residual features to obtain the residual result output from the first residual unit; perform a convolution process on the residual result output from the (j-1)-th residual unit by the j-th residual unit of the plurality of cascaded residual units to obtain the convolution result of the j-th residual unit; add the convolution result of the j-th residual unit and the residual result output from the (j-1)-th residual unit to obtain the residual result output from the j-th residual unit; and output the residual result output from the j-th residual unit to the (j + 1)-th residual unit, where j is a positive integer that sequentially increases, 1 < j < J, and J is the number of the residual units.

[0322] In some embodiments, each of the residual units includes a dilated convolution operator, and the decoding module 6552 is further configured to perform a dilated convolution process on the residual features via the dilated convolution operator included in the first residual unit.

[0323] In some embodiments, the decoding module 6552 is further configured to perform a grouping operation on the input channels of the residual features to obtain a plurality of first groups, each of the first groups including first elements corresponding to at least two channels in the residual features; and a dilation and convolution operation on the first elements in each of the first groups.

[0324] In some embodiments, each of the residual units further includes at least one causal convolution operator, and after performing a dilation and convolution process on the residual features through the dilation and convolution operator included in the first residual unit, the decoding module 6552 is further configured to perform a causal convolution process on the obtained first dilation and convolution result through the at least one causal convolution operator included in the first residual unit, and set the obtained causal convolution result as the convolution result of the first residual unit.

[0325] In some embodiments, the second residual module 6553 is further configured to perform a grouping process on the input channels of the dilated convolution result to obtain a plurality of second groups, each of the second groups including second elements corresponding to at least two channels in the dilated convolution result; and a causal convolution process on the second elements in each of the second groups.

[0326] In some embodiments, the first neural network used for audio decoding includes a plurality of cascaded decoding blocks, each of the decoding blocks including a feature decoding block and at least one residual unit, and the decoding module 6552 is further configured to perform: performing cascaded feature decoding processing on encoded features corresponding to the audio bitstream via feature decoding blocks in the plurality of cascaded decoding blocks to obtain residual features corresponding to the audio bitstream; and performing residual processing on the residual features corresponding to the audio bitstream by at least one residual unit in the plurality of cascaded decoding blocks to obtain audio features corresponding to the audio bitstream.

[0327] In some embodiments, the decoding module 6552 further performs a single feature decoding process on the encoded features corresponding to the audio bitstream through the feature decoding block in the first decoding block of the plurality of cascaded decoding blocks, and outputs the decoding result output from the feature decoding block in the first decoding block to at least one residual unit in the first decoding block; and performs a single feature decoding process on the residual result output from at least one residual unit in the (i-1)-th decoding block through the feature decoding block in the i-th decoding block of the plurality of cascaded decoding blocks, and outputs the decoding result output from the feature decoding block in the i-th decoding block to at least one residual unit in the i-th decoding block; and sets the decoding result output from the feature decoding block in the last decoding block as the residual features corresponding to the audio bitstream, where i is a positive integer that sequentially increases, 1 < i ≤ I, and I is the number of the decoding blocks; and performs a residual process on the residual features corresponding to the audio bitstream by at least one residual unit in the last decoding block of the plurality of cascaded decoding blocks to obtain the audio features corresponding to the audio bitstream.

[0328] In some embodiments, the decoding module 6552 further performs a convolution process on the encoded features corresponding to the audio bitstream by the feature decoding block in the first decoding block of the plurality of cascaded decoding blocks to obtain convolution features, where the number of channels of the convolution features is smaller than the number of channels of the encoded features; and performs an upsampling process on the convolution features to obtain the decoding result output from the feature decoding block in the first decoding block.

[0329] In some embodiments, the feature reconstruction 6553 is further configured to perform an upsampling operation on audio features corresponding to the audio bitstream to obtain upsampled features, and to perform a causal convolution operation on the upsampled features to obtain a reconstructed audio signal corresponding to the audio bitstream.

[0330] In some embodiments, the audio bitstream is a low-frequency bitstream within a full-frequency bitstream, the full-frequency bitstream including the low-frequency bitstream and a high-frequency bitstream, and the signal decoding module 6551 is further configured to perform the steps of: performing a signal decoding process on the high-frequency bitstream to obtain high-frequency encoding features corresponding to the high-frequency bitstream, the high-frequency bitstream being obtained by audio encoding high-frequency sub-band signals of the audio signal; performing a high-frequency reconstruction process on the high-frequency encoding features corresponding to the high-frequency bitstream to obtain high-frequency sub-band signals corresponding to the high-frequency bitstream; and performing a sub-band synthesis process on the reconstructed audio signal corresponding to the low-frequency bitstream and the high-frequency sub-band signals to obtain a reconstructed audio signal.

[0331] In some embodiments, the signal decoding module 6551 is further configured to perform the steps of: invoking a second neural network to perform a feature reconstruction process based on the second neural network on the high frequency encoding features to obtain a high frequency subband signal corresponding to the high frequency bitstream, wherein the number of channels of the second neural network model is smaller than the number of channels of the first neural network, and the first neural network is used to reconstruct the reconstructed audio signal from the audio bitstream; or performing an inverse process of band extension on the high frequency encoding features to obtain a high frequency subband signal corresponding to the high frequency bitstream.

[0332] An embodiment of the present application provides a computer program product including computer-executable instructions stored in a computer-readable storage medium, wherein a processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions to cause the electronic device to perform an audio encoding method or an audio decoding method according to an embodiment of the present application.

[0333] An embodiment of the present application provides a computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, cause the processor to perform an audio encoding method or an audio decoding method provided in an embodiment of the present application, for example, the audio encoding method shown in FIG. 4A.

[0334] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic memory, optical disk, or CD-ROM, or may be various electronic devices that include one or any combination of the above memories.

[0335] In some embodiments, computer-executable instructions (abbreviated as executable instructions) may take the form of a program, software, software module, script, or code, may be written in any type of programming language (including compiled or interpreted, declarative or procedural languages), and may be arranged in any form, including as an independent program, or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0336] As an example, the executable instructions do not necessarily correspond to a file in a file system, but may be stored as part of a file that holds other programs or data, for example, within one or more scripts within a HyperText Markup Language (HTML) document, within a single file dedicated to the program being discussed, or within multiple joint files (such as files that store one or more modules, subprograms, or code portions).

[0337] As an example, the executable instructions may be arranged to be executed on a single electronic device, may be arranged to be executed on multiple electronic devices located at a single location, or may be arranged to be executed on multiple electronic devices distributed across multiple locations and interconnected via a communications network.

[0338] It is to be understood that embodiments of the present application involve associated data such as user information, and where the embodiments of the present application are applicable to a particular product or technology, user permission or consent may be required, and the collection, use, and processing of associated data may be in compliance with the relevant laws, regulations, and standards of the relevant country or region.

[0339] The above is merely an example of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application shall be included in the scope of protection of the present application.

Claims

1. 1. An audio decoding method implemented by an electronic device, comprising: performing a signal decoding process on an audio bitstream to obtain coding features corresponding to the audio bitstream, the audio bitstream being obtained by audio encoding an audio signal; performing decoding-side residual processing on the coding features using at least one residual unit to obtain audio features corresponding to the audio bitstream; performing a feature reconstruction process on audio features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

2. the step of performing a decoding-side residual process on the coding features corresponding to the audio bitstream using the at least one residual unit to obtain audio features corresponding to the audio bitstream includes: performing a feature decoding process on the coding features corresponding to the audio bitstream to obtain residual features corresponding to the audio bitstream; performing feature residual processing on the residual features corresponding to the audio bitstream by the at least one residual unit to obtain audio features corresponding to the audio bitstream; 2. The audio decoding method of claim 1.

3. When the at least one residual unit is a plurality of cascaded residual units, performing feature residual processing on residual features corresponding to the audio bitstream by the at least one residual unit to obtain audio features corresponding to the audio bitstream may include: performing a single residual processing on the residual features by a first residual unit of the plurality of cascaded residual units; outputting the residual result output from the first residual unit to a subsequent cascaded residual unit, so that the subsequent cascaded residual unit continues to perform one round of residual processing and output the residual result; and determining the residual result output from the last residual unit as the audio feature corresponding to the audio bitstream.

3. An audio decoding method according to claim 2.

4. The step of performing a single residual processing on the residual features by a first residual unit of the plurality of cascaded residual units includes: a first residual unit of the plurality of cascaded residual units, performing a convolution process on the residual features to obtain a convolution result of the first residual unit; adding the convolution result of the first residual unit and the residual feature to obtain a residual result output from the first residual unit; and / or Continuing to perform one residual processing and output the residual result by the subsequent cascaded residual unit includes: by the j-th residual unit of the plurality of cascaded residual units, A process of performing a convolution process on the residual result output from the j-1th residual unit to obtain a convolution result of the jth residual unit; adding the convolution result of the j-th residual unit and the residual result output from the j-1-th residual unit to obtain the residual result output from the j-th residual unit; and outputting the residual result output from the j-th residual unit to the j+1-th residual unit, where j is a sequentially increasing positive integer, 1<j<J, and J is the number of the residual units.

4. An audio decoding method according to claim 3.

5. each said residual unit includes a dilation and convolution operator; The step of performing a convolution process on the residual features includes: performing a dilation and convolution process on the residual features via a dilation and convolution operator included in the first residual unit; 5. An audio decoding method according to claim 4.

6. The step of performing a dilation and convolution process on the residual features includes: performing a grouping process on the input channels of the residual features to obtain a plurality of first groups, each of the first groups including first elements corresponding to at least two channels in the residual features; performing a dilation and convolution process on the first elements in each of the first groups; 6. An audio decoding method according to claim 5.

7. Each of the residual units further includes at least one causal convolution operator; After performing a dilation and convolution process on the residual feature through a dilation and convolution operator included in the first residual unit, the audio decoding method includes: the method further includes a step of performing a causal convolution process on the obtained dilated convolution result through at least one causal convolution operator included in the first residual unit, and setting the obtained causal convolution result as a convolution result of the first residual unit.

6. An audio decoding method according to claim 5.

8. The step of performing a causal convolution process on the obtained dilated convolution result includes: performing a grouping process on the input channels of the dilated and convolved result to obtain a plurality of second groups, each of the second groups including second elements corresponding to at least two channels in the dilated and convolved result; performing a causal convolution process on the second elements in each of the second groups; 8. An audio decoding method according to claim 7.

9. The first neural network used for audio decoding includes a plurality of cascaded decoding blocks, each of which includes a feature decoding block and at least one residual unit; performing a feature decoding process on the coding features corresponding to the audio bitstream to obtain residual features corresponding to the audio bitstream, performing cascaded feature decoding processes on encoded features corresponding to the audio bitstream via feature decoding blocks in the plurality of cascaded decoding blocks to obtain residual features corresponding to the audio bitstream; and / or performing feature residual processing on the residual features corresponding to the audio bitstream to obtain audio features corresponding to the audio bitstream, performing residual processing on residual features corresponding to the audio bitstream by at least one residual unit in the plurality of cascaded decoding blocks to obtain audio features corresponding to the audio bitstream; 9. An audio decoding method according to claim 8.

10. performing cascaded feature decoding processes on encoded features corresponding to the audio bitstream via feature decoding blocks in the plurality of cascaded decoding blocks to obtain residual features corresponding to the audio bitstream, performing a feature decoding process on the coded features corresponding to the audio bitstream through a feature decoding block in a first decoding block of the plurality of cascaded decoding blocks, and outputting the decoding result output from the feature decoding block in the first decoding block to at least one residual unit in the first decoding block; performing a feature decoding process once on a residual result output from at least one residual unit in an (i-1)th decoding block through a feature decoding block in an i-th decoding block of the plurality of cascaded decoding blocks, and outputting the decoding result output from the feature decoding block in the i-th decoding block to at least one residual unit in the i-th decoding block; and determining a decoding result output from a feature decoding block in a last decoding block as a residual feature corresponding to the audio bitstream, where i is a positive integer that increases sequentially, 1<i≦I, and I is the number of the decoding blocks; and / or performing residual processing on residual features corresponding to the audio bitstream by at least one residual unit in the plurality of cascaded decoding blocks to obtain audio features corresponding to the audio bitstream, performing residual processing on residual features corresponding to the audio bitstream by at least one residual unit in a last decoding block of the plurality of cascaded decoding blocks to obtain audio features corresponding to the audio bitstream; 10. An audio decoding method according to claim 9.

11. performing a single feature decoding process on the coded features corresponding to the audio bitstream via a feature decoding block in a first decoding block of the plurality of cascaded decoding blocks, a feature decoding block in a first decoding block of the plurality of cascaded decoding blocks, performing a convolution process on the coding features corresponding to the audio bitstream to obtain convolution features, wherein the number of channels of the convolution features is smaller than the number of channels of the coding features; performing an upsampling process on the convolutional features to obtain a decoding result output from a feature decoding block in the first decoding block; 11. An audio decoding method according to claim 10.

12. performing a feature reconstruction process on audio features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream, performing an upsampling process on the audio features corresponding to the audio bitstream to obtain upsampling features; performing a causal convolution process on the upsampled features to obtain a reconstructed audio signal corresponding to the audio bitstream.

2. The audio decoding method of claim 1.

13. the audio bitstream is a low-frequency bitstream within a full-frequency bitstream, the full-frequency bitstream including the low-frequency bitstream and the high-frequency bitstream; The audio decoding method includes: performing a signal decoding process on the high-frequency bitstream to obtain high-frequency coding features corresponding to the high-frequency bitstream, the high-frequency bitstream being obtained by audio coding a high-frequency sub-band signal of the audio signal; performing a high-frequency reconstruction process on high-frequency coding features corresponding to the high-frequency bitstream to obtain high-frequency sub-band signals corresponding to the high-frequency bitstream; performing a subband synthesis process on the reconstructed audio signal corresponding to the low-frequency bitstream and the high-frequency subband signals to obtain a reconstructed audio signal; 13. An audio decoding method according to claim 12.

14. performing a high-frequency reconstruction process on high-frequency coding features corresponding to the high-frequency bitstream to obtain a high-frequency sub-band signal corresponding to the high-frequency bitstream, Invoking a second neural network to perform feature reconstruction processing based on the second neural network on the high-frequency encoded features to obtain high-frequency sub-band signals corresponding to the high-frequency bitstream, wherein the number of channels of the second neural network model is smaller than the number of channels of the first neural network, and the first neural network is used to reconstruct the reconstructed audio signal from the audio bitstream; or performing an inverse process of band extension on the high frequency encoded features to obtain a high frequency sub-band signal corresponding to the high frequency bitstream; 14. An audio decoding method according to claim 13.

15. An audio encoding method performed by an electronic device, comprising: performing a feature extraction process on the audio signal to obtain audio features of the audio signal; performing coding-side residual processing on the audio features using at least one residual unit to obtain coding features of the audio signal; performing a signal coding process on the coded features of the audio signal to obtain an audio bitstream of the audio signal.

16. An audio decoding device, comprising: a signal decoding module configured to perform a signal decoding process on an audio bitstream to obtain coding features corresponding to said audio bitstream, said audio bitstream being obtained by audio encoding an audio signal; a decoding module configured to perform decoding-side residual processing on the encoded features using at least one residual unit to obtain audio features corresponding to the audio bitstream; a feature reconstruction module configured to perform a feature reconstruction process on audio features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

17. An audio encoding device, comprising: a feature extraction module configured to perform a feature extraction process on an audio signal to obtain audio features of the audio signal; an encoding module configured to perform encoding-side residual processing on the audio features using at least one residual unit to obtain coding features of the audio signal; a signal coding module configured to perform a signal coding process on the coding features of the audio signal to obtain an audio bitstream of the audio signal.

18. An electronic device, a memory for storing computer-executable instructions; and a processor that, when executing computer-executable instructions stored in the memory, performs the audio decoding method of any one of claims 1 to 14 or the audio encoding method of claim 15.

19. A computer program causing a computer to execute the audio decoding method according to any one of claims 1 to 14 or the audio encoding method according to claim 15.

Images