Audio encoding method, device, electronic device, and program

Abstract

The present application provides an audio encoding method, an apparatus, a device, a storage medium, and a computer program product, the method including the steps of: performing first-level feature extraction on an audio signal to obtain first-level signal features; for an i-th level among N levels, splicing the audio signal and (i-1)-th level signal features to obtain spliced ​​features, and performing i-th level feature extraction on the spliced ​​features to obtain i-th level signal features, where N and i are integers greater than 1 and i is less than or equal to N; traversing i to obtain signal features for each level among the N levels, where a data dimension of the signal features is less than a data dimension of the audio signal; and encoding the first-level signal features and the signal features for each level among the N levels, respectively, to obtain a stream of audio signals at each level.

Description

[Technical Field]

[0001] The present application relates to the technical field of audio processing, and in particular to an audio encoding method, an audio decoding method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0002] This application is based on and claims priority from a Chinese patent application bearing application number 202210677636.4 and filed on June 15, 2022, the entire contents of which are hereby incorporated by reference into this application. [Background technology]

[0003] Audio coding and decoding technology is a core technology applied to communication services, including long-distance audio and video calls. Audio coding technology can be understood as transmitting as much voice information as possible using relatively limited network bandwidth resources. Audio coding is a type of signal source coding, whose purpose is to compress the data volume of the information the user wants to transmit as much as possible on the encoding side, remove redundant information, and simultaneously restore it to a lossless (or nearly lossless) state on the decoding side.

[0004] However, in the related art, the efficiency of audio encoding is low when ensuring the quality of audio encoding. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide an audio encoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the efficiency of audio encoding and ensure the quality of audio encoding. [Means for solving the problem]

[0006] The technical means of the embodiments of the present application are realized as follows.

[0007] An embodiment of the present application provides an audio encoding method, performing first-level feature extraction on the audio signal to obtain the first-level signal features; For an i-th level among the N levels, splicing the audio signal and signal features of the (i-1)-th level to obtain spliced ​​features, and performing i-th level feature extraction on the spliced ​​features to obtain signal features of the i-th level, where N and i are integers greater than 1, and i is equal to or less than N; traversing the i and obtaining signal features for each of the N levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; encoding the signal features of the first level and the signal features of each of the N levels to obtain a stream of the audio signal at each level.

[0008] An embodiment of the present application further provides an audio decoding method, comprising: receiving a stream corresponding to each of a plurality of levels obtained by encoding an audio signal; decoding the streams of each of the levels to obtain signal features of each of the levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; performing feature reconstruction on the signal features of each of the levels to obtain a level audio signal for each of the levels; and performing audio synthesis on the level audio signals of the plurality of levels to obtain the audio signal.

[0009] An embodiment of the present application further provides an audio encoding device, a first feature extraction module configured to perform first level feature extraction on the audio signal to obtain the first level signal features; a second feature extraction module configured to splice the audio signal and (i-1)-th level signal features for an i-th level among N levels to obtain spliced ​​features, and perform i-th level feature extraction on the spliced ​​features to obtain the i-th level signal features, where N and i are integers greater than 1, and i is less than or equal to N; a traversing module configured to traverse the i to obtain signal features for each of the N levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; an encoding module configured to encode the signal features of the first level and the signal features of each of the N levels, respectively, to obtain a stream of the audio signal at each level.

[0010] An embodiment of the present application further provides an audio decoding device, a receiving module configured to receive streams corresponding to a plurality of levels obtained by encoding an audio signal; a decoding module configured to decode the streams of each of the levels to obtain signal features of each of the levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; a feature reconstruction module configured to perform feature reconstruction on signal features of each of the levels to obtain a level audio signal for each of the levels; an audio synthesis module configured to perform audio synthesis on the level audio signals of the plurality of levels to obtain the audio signal.

[0011] An embodiment of the present application further provides an electronic device, a memory configured to store executable instructions; and a processor configured, when executing executable instructions stored in the memory, to implement the methods provided by the embodiments of the present application.

[0012] The embodiments of the present application further provide a computer-readable storage medium having executable instructions stored thereon, the executable instructions, when executed by a processor, implementing the methods provided by the embodiments of the present application.

[0013] The embodiments of the present application further provide a computer program product, which includes a computer program or instructions, which, when executed by a processor, implements the method provided by the embodiments of the present application. [Effects of the Invention]

[0014] The embodiments of the present application have the following beneficial effects:

[0015] By encoding an audio signal level by level, signal features for each level are obtained, and the data dimension of the signal features for each level is smaller than that of the audio signal, so the data dimension of the data processed in the audio encoding process is reduced and the encoding efficiency of the audio signal is improved. When extracting signal features for an audio signal level by level, the output of each level is used as the input of the next level, and each level is combined with the extracted signal features of the previous level to perform more accurate feature extraction for the audio signal. As the number of levels increases, the information loss of the audio signal in the feature extraction process can be minimized. In this way, the audio signal information contained in the multiple streams obtained by encoding the signal features extracted in this manner is closer to the original audio signal, so the information loss of the audio signal in the encoding process is reduced and the encoding quality of the audio encoding is ensured. [Brief explanation of the drawings]

[0016] [Figure 1]1 is a schematic diagram illustrating the architecture of an audio encoding system 100 according to an embodiment of the present application. [Figure 2] 1 is a structural schematic diagram of an electronic device 500 that implements an audio encoding method provided by an embodiment of the present application. [Figure 3] 1 is a flowchart of an audio encoding method provided by an embodiment of the present application; [Figure 4] 1 is a flowchart of an audio encoding method provided by an embodiment of the present application; [Figure 5] 1 is a flowchart of an audio encoding method provided by an embodiment of the present application; [Figure 6] 1 is a flowchart of an audio encoding method provided by an embodiment of the present application; [Figure 7] 1 is a flowchart of an audio encoding method provided by an embodiment of the present application; [Figure 8] 1 is a flowchart of an audio encoding method provided by an embodiment of the present application; [Figure 9] 1 is a flowchart of an audio encoding method provided by an embodiment of the present application; [Figure 10] 1 is a flowchart of an audio decoding method provided by an embodiment of the present application; [Figure 11] 1 is a flowchart of an audio decoding method provided by an embodiment of the present application; [Figure 12] FIG. 1 is a schematic diagram of spectrum comparison at different bit rates provided by an embodiment of the present application. [Figure 13] 1 is a flowchart of audio encoding and audio decoding provided by an embodiment of the present application; [Figure 14] 1 is a schematic diagram of an audio communication link provided by an embodiment of the present application; [Figure 15] FIG. 1 is a schematic diagram of a filter bank provided by an embodiment of the present application. [Figure 16A] FIG. 1 is a schematic diagram of a typical convolutional network provided by an embodiment of the present application. [Figure 16B] FIG. 1 is a schematic diagram of a dilated convolutional network provided by an embodiment of the present application. [Figure 17] FIG. 1 is a structural schematic diagram of a first-layer low-frequency analysis neural network model provided in an embodiment of the present application. [Figure 18] FIG. 1 is a structural schematic diagram of a second-layer low-frequency analysis neural network model provided in an embodiment of the present application. [Figure 19] FIG. 1 is a model schematic diagram of a first-layer low-frequency synthesis neural network model provided in an embodiment of the present application. [Figure 20] FIG. 2 is a structural schematic diagram of a second-layer low-frequency synthesis neural network model provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0017] In order to make the objectives, technical means, and advantages of the present application clearer, the present application will be described in detail below in combination with the drawings. However, the described embodiments should not be regarded as limitations on the present application, and all other embodiments obtained by those skilled in the art without any creative work fall within the scope of protection of the present application.

[0018] In the following description, references to "some embodiments" may describe a subset of all possible embodiments, but it should be understood that "some embodiments" may be the same or different subsets of all possible embodiments, and may be combined with each other where not inconsistent.

[0019] In the following description, the relative terms "first\second\third" are used merely to distinguish between similar objects and do not represent a particular order for the objects. It will be understood that "first\second\third" can be interchanged with the particular order or order where permitted, such that the embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terms used herein are for the purpose of describing the examples of the present application only and are not intended to limit the present application.

[0021] Before describing the embodiments of the present application in detail, nouns and terms related to the embodiments of the present application will be explained. The nouns and terms related to the embodiments of the present application can be applied to the following interpretations.

[0022] 1) The client is an application program that runs on a terminal and is used to provide various services, such as an instant messaging client or an audio playback client.

[0023] 2) Audio coding is a type of application that performs data compression on digital audio signals that contain voice.

[0024] 3) Quadrature Mirror Filters (QMF): A QMF filter bank is used to decompose a subband signal into multiple signals, thereby reducing the signal bandwidth, and each signal after decomposition is filtered by its own channel.

[0025] 4) Quantization refers to the process of approximating the continuous values ​​(or a large number of possible discrete values) of a signal to a limited number (or a relatively small number) of discrete values, and includes vector quantization, scalar quantization, etc.

[0026] 5) For vector quantization, some scalar data are organized into one vector, the vector space is divided into some sub-regions, one representative vector is found in each sub-region, and the vectors that fall into the sub-region during quantization are replaced with the corresponding representative vector, i.e., quantized to the representative vector.

[0027] 6) For scalar quantization, the entire dynamic range is divided into several small intervals, each of which has a representative value, and when quantizing, the signal value that falls into a small interval is replaced with the corresponding representative value, i.e., the signal value is quantized to the representative value.

[0028] 7) Entropy coding is a coding method that does not lose any information during the coding process according to the entropy principle. The general information entropy is the average information amount of the signal source, and common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0029] 8) A neural network (NN) is an algorithmic mathematical model that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. Such networks rely on the complexity of the system to achieve their information processing goals by coordinating the interconnections between a large number of internal nodes.

[0030] 9) Deep learning (DL) is a new research direction in the field of machine learning (ML). Deep learning involves learning the inherent rules and representation levels of sample data, and the information acquired during this learning process is extremely useful for interpreting data such as text, images, and sounds. The ultimate goal is to enable machines to have human-like analytical learning capabilities and recognize data such as text, images, and sounds.

[0031] Embodiments of the present application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the efficiency of audio encoding and ensure the quality of audio encoding.

[0032] The following describes an implementation scenario of an audio encoding method provided by an embodiment of the present application. Referring to Fig. 1, Fig. 1 is a schematic architecture diagram of an audio encoding system 100 provided by an embodiment of the present application. To support an exemplary application, terminals (terminals 400-1 and 400-2 are shown as examples) are connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of both, and may use wireless or wired links to transmit data. Here, terminal 400-1 is a transmitter of an audio signal, and terminal 400-2 is a receiver of the audio signal.

[0033] In a process in which terminal 400-1 transmits an audio signal to terminal 400-2 (for example, in a process in which terminal 400-1 and terminal 400-2 are performing a remote call based on a set client), terminal 400-1 performs first-level feature extraction on the audio signal to obtain first-level signal features, and for an i-th level among N levels, splices the audio signal with the (i-1)th-level signal features to obtain a spliced ​​feature. Terminal 400-1 also performs i-th-level feature extraction on the spliced ​​feature to obtain an i-level signal feature. N and i are integers greater than 1, and i is less than or equal to N. Terminal 400-1 traverses i to obtain signal features for each of the N levels. The data dimension of the signal features is smaller than the data dimension of the audio signal, and terminal 400-1 is configured to encode the first-level signal features and the signal features for each of the N levels, respectively, obtain an audio signal stream for each level, and transmit the audio signal stream for each level to server 200.

[0034] The server 200 is configured to receive streams corresponding to each of a plurality of levels obtained by encoding an audio signal by the terminal 400-1, and to transmit the streams corresponding to each of the plurality of levels to the terminal 400-2.

[0035] The terminal 400-2 receives streams corresponding to the multiple levels obtained by encoding the audio signal transmitted by the server 200, decodes the streams for each level, and obtains signal features for each level. The data dimension of the signal features is smaller than the data dimension of the audio signal, and the terminal 400-2 is configured to perform feature reconstruction on the signal features for each level, obtain level audio signals for each level, and perform audio synthesis on the level audio signals for the multiple levels to obtain an audio signal.

[0036] In some embodiments, the audio encoding method provided by the embodiments of the present application can be implemented by various electronic devices, for example, by a terminal alone, a server alone, or a terminal and a server working together. For example, the terminal independently executes the audio encoding method provided by the embodiments of the present application, or the terminal sends an encoding request for an audio signal to a server. The server executes the audio encoding method provided by the embodiments of the present application in response to the received encoding request. The embodiments of the present application can be applied to various scenarios, including, but not limited to, cloud technology, artificial intelligence, smart transportation, and driving assistance.

[0037] In some embodiments, the electronic device performing the audio encoding provided by the embodiments of the present application may be various types of terminal devices or servers. Here, the server (e.g., server 200) may be an independent physical server, a server cluster consisting of multiple physical servers, or a distributed system. The terminal (e.g., terminal 400) may be, but is not limited to, a smartphone, a tablet, a laptop, a desktop computer, an intelligent voice interaction device (e.g., a smart speaker), a smart home appliance (e.g., a smart TV), a smart watch, an in-vehicle terminal, etc. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application do not impose any limitations thereon.

[0038] In some embodiments, the audio encoding method provided by the embodiments of the present application can be realized with the help of cloud technology. Cloud technology refers to a type of hosting technology that unifies a set of resources, such as hardware, software, and networks, in a wide area network or a local area network to realize data calculation, storage, processing, and sharing. Cloud technology is a collective term for network technology, information technology, integration technology, management platform technology, and application technology based on the application of cloud computing business model, forming a resource pool that can be used flexibly and conveniently according to needs. Cloud computing technology will become an important support. Background services of technical network systems require large amounts of calculation and storage resources. For example, the server (e.g., server 200) may be a cloud server that provides basic cloud computing services, such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0039] In some embodiments, a terminal or a server can implement the audio encoding method provided by the embodiments of the present application by running a computer program, for example, the computer program may be a native program or software module in an operating system, a native application program (APP, Application) that needs to be installed in an operating system to operate, an applet that runs simply by downloading it to a browser environment, or an applet that can be incorporated into any APP. In summary, the computer program may be any type of application program, module, or plug-in.

[0040] In some embodiments, multiple servers may form a blockchain, where the servers are nodes in the blockchain, and information connections may exist between individual nodes in the blockchain, and information may be transmitted between the nodes via the information connections. Here, data related to the audio encoding method provided by the embodiments of the present application (e.g., audio signal streams at each level, neural network models used for feature extraction) may be stored in the blockchain.

[0041] The following describes an electronic device that implements an audio encoding method provided by an embodiment of the present application. Referring to FIG. 2, FIG. 2 is a structural diagram of an electronic device 500 that implements an audio encoding method provided by an embodiment of the present application. Taking the electronic device 500 as the terminal (e.g., the terminal 400-1) shown in FIG. 1 as an example, the electronic device 500 that implements the audio encoding method provided by an embodiment of the present application includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The components in the electronic device 500 are coupled together by a bus system 540. It can be understood that the bus system 540 is used to realize communication between these components. In addition to including a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses in FIG. 2 are marked as the bus system 540.

[0042] The processor 510 may be a type of integrated circuit chip, having a signal processing capability, and may be, for example, a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. Here, the general-purpose processor may be a microprocessor, or any other conventional processor, etc.

[0043] The memory 550 may be removable, non-removable, or a combination thereof. The memory 550 may optionally include one or more storage devices physically separate from the processor 510. The memory 550 may include volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Here, the non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in the embodiments herein is intended to include any suitable type of memory.

[0044] In some embodiments, memory 550 may store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as illustratively described below.

[0045] The operating system 551 includes system programs configured to process various basic system services and perform hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to realize various basic services and process hardware-based tasks.

[0046] The network communication module 552 is configured to reach other computing devices via one or more (wired or wireless) network interfaces 520, example network interfaces 520 include Bluetooth, Wireless Fidelity (WiFi), Universal Serial Bus (USB), etc.

[0047] In some embodiments, the audio encoding device provided by the embodiments of the present application may be implemented using a software approach. Figure 2 shows an audio encoding device 553 stored in a memory 550, which may be software in the form of a program, plug-in, etc., and includes software modules: a first feature extraction module 5531, a second feature extraction module 5532, a traversal module 5533, and an encoding module 5534. These modules are logical and can therefore be arbitrarily combined or divided according to the functions to be implemented. The functions of each module will be described below.

[0048] An audio encoding method provided by an embodiment of the present application will be described below. In some embodiments, the audio encoding method provided by the embodiment of the present application can be implemented by various electronic devices, for example, by a terminal alone, a server alone, or a terminal and a server working together. Taking the implementation by a terminal as an example, as shown in FIG. 3, FIG. 3 is a flowchart of the audio encoding method provided by the embodiment of the present application, and the audio encoding method provided by the embodiment of the present application includes steps 101 to 104.

[0049] Step 101: The terminal performs first-level feature extraction on the audio signal to obtain first-level signal features.

[0050] In practical applications, the audio signal may be a voice signal during a call (e.g., a network call, a telephone call), a voice message (e.g., a voice message sent in an instant messaging client), music being played, audio, etc. When transmitting an audio signal, the audio signal needs to be coded, so that a sender of the audio signal can transmit a stream obtained by coding, and a receiver of the stream can obtain the audio signal by decoding the received stream. Next, the coding process of the audio signal will be described. In the embodiments of the present application, a level-by-level coding scheme is adopted to code the audio signal, and the level-by-level coding scheme is realized by performing multiple levels of coding on the audio signal. The coding process of each level will be described below. First, for the first level, a terminal performs first-level feature extraction on the audio signal, and obtains the signal features of the audio signal extracted by the first level, i.e., the first-level signal features.

[0051] In some embodiments, an audio signal includes a low-frequency subband signal and a high-frequency subband signal, and when processing the audio signal (e.g., feature extraction, encoding), the low-frequency subband signal and the high-frequency subband signal included in the audio signal can be processed respectively. Based on this, please refer to FIG. 4, which is a flowchart of an audio encoding method provided by an embodiment of the present application, and FIG. 4 shows that step 101 in FIG. 3 can be realized by steps 201 to 203. Step 201: Perform subband decomposition on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal. Step 202: Perform first-level feature extraction on the low-frequency subband signals to obtain first-level low-frequency signal features, and perform first-level feature extraction on the high-frequency subband signals to obtain first-level high-frequency signal features. Step 203: The low-frequency signal features and the high-frequency signal features are used as first-level signal features.

[0052] It should be noted that in step 201, in the process of performing feature extraction on the audio signal at a first level, the terminal first performs subband decomposition on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal, and then performs feature extraction on the low-frequency subband signals and high-frequency subband signals, respectively. In some embodiments, refer to FIG. 5, which is a flowchart of an audio encoding method provided by an embodiment of the present application, and FIG. 5 shows that step 201 of FIG. 4 can be realized by steps 2011 to 2013. Step 2011: Sample the audio signal according to a first sampling frequency to obtain a sampled signal. Step 2012: Low-pass filter the sampled signal to obtain the low-pass filtered signal, and down-sample the low-pass filtered signal to obtain a low-frequency subband signal of a second sampling frequency. Step 2013: High-pass filter the sampled signal to obtain the high-pass filtered signal, and down-sample the high-pass filtered signal to obtain a high-frequency subband signal of a second sampling frequency, where the second sampling frequency is lower than the first sampling frequency.

[0053] In step 2011, the audio signal is sampled according to a first sampling frequency to obtain a sampled signal, which may be preset. In practical applications, the audio signal is a continuous analog signal, and sampling the audio signal at the first sampling frequency obtains a discrete digital signal, i.e., a sampled signal, which includes a plurality of sample points (i.e., sampling values) obtained by sampling the audio signal.

[0054] In step 2012, the sampled signal is low-pass filtered to obtain a low-pass filtered signal, and the low-pass filtered signal is down-sampled to obtain a low-frequency sub-band signal with a second sampling frequency. In step 2013, the sampled signal is high-pass filtered to obtain a high-pass filtered signal, and the high-pass filtered signal is down-sampled to obtain a high-frequency sub-band signal with a second sampling frequency. In steps 202 and 203, the low-pass filtering and high-pass filtering can be achieved by a QMF analysis filter. In practice, the second sampling frequency can be half the first sampling frequency, thereby obtaining low-frequency sub-band signals and high-frequency sub-band signals with the same frequency.

[0055] In step 202, after obtaining the low-frequency subband signal and the high-frequency subband signal of the audio signal, first-level feature extraction is performed on the low-frequency subband signal of the audio signal to obtain first-level low-frequency signal features, and first-level feature extraction is performed on the high-frequency subband signal to obtain first-level high-frequency signal features. In step 203, the low-frequency signal features and the high-frequency signal features are taken as first-level signal features.

[0056] In some embodiments, refer to FIG. 6, which is a flowchart of an audio encoding method provided by an embodiment of the present application. FIG. 6 shows that step 101 in FIG. 3 can be further realized by steps 301 to 304. Step 301: Perform a first convolution process on the audio signal to obtain first-level convolution features. Step 302: Perform a first pooling process on the convolution features to obtain first-level pooled features. Step 303: Perform a first downsampling process on the pooled features to obtain first-level downsampled features. Step 304: Perform a second convolution process on the downsampled features to obtain first-level signal features.

[0057] It is necessary to explain that in step 301, a first convolution process can be performed on the audio signal. In practical application, the first convolution process can be performed by invoking a causal convolution process with a preset number of channels (for example, 24 channels) to obtain a first-level convolution feature.

[0058] In step 302, a first pooling process is performed on the convolution features obtained in step 301. In practical application, the first pooling process can be performed by presetting a pooling factor (e.g., 2), and then performing the first pooling process on the convolution features based on the pooling factor to obtain first-level pooled features.

[0059] In step 303, a first downsampling is performed on the pooling features obtained in step 302. In practical applications, a downsampling factor can be preset, and downsampling is performed based on the downsampling factor. The first downsampling can be achieved by one coding layer or multiple coding layers. In some embodiments, the first downsampling is achieved by M cascaded coding layers. Accordingly, refer to FIG. 7, which is a flowchart of an audio coding method provided by an embodiment of the present application. FIG. 7 shows that step 303 in FIG. 6 can be further achieved by steps 3031 to 3033. Step 3031: The first coding layer of the M cascaded coding layers performs a first downsampling on the pooling features to obtain a downsampling result of the first coding layer. Step 3032: The j-th coding layer of the M cascaded coding layers performs a first downsampling on the downsampling result of the (j-1)-th coding layer to obtain a downsampling result of the j-th coding layer. where M and j are integers greater than 1, and j is less than or equal to M. Step 3033: Traverse j to obtain the downsampling result of the Mth coding layer, and take the downsampling result of the Mth coding layer as the first-level downsampling feature.

[0060] It should be noted that in steps 3031 to 3033, the downsampling factors of each coding layer may be the same or different, and in practical applications, the downsampling factor corresponds to the pooling factor and performs the function of downward sampling.

[0061] In step 304, a second convolution process can be performed on the downsampled features. In practical applications, the second convolution process can be performed by invoking causal convolution with a preset number of channels, thereby obtaining first-level signal features.

[0062] 6 can be implemented by calling a first neural network model, which includes a first convolutional layer, a pooling layer, a downsampling layer, and a second convolutional layer. Thus, the first convolutional layer is called to perform a first convolutional process on the audio signal to obtain first-level convolutional features, the pooling layer is called to perform a first pooling process on the convolutional features to obtain first-level pooled features, the downsampling layer is called to perform a first downsampling on the pooled features to obtain first-level downsampling features, and the second convolutional layer is called to perform a second convolutional process on the downsampling features to obtain first-level signal features.

[0063] It should be noted that when performing first-level feature extraction on an audio signal, first-level feature extraction can also be performed on each of the low-frequency subband signals and high-frequency subband signals of the audio signal through steps 301 to 304 shown in Fig. 6 (i.e., step 202 shown in Fig. 4). That is, a first convolution process is performed on the low-frequency subband signals of the audio signal to obtain first-level first convolution features, a first pooling process is performed on the first convolution features to obtain first-level first pooled features, a first downsampling process is performed on the first pooled features to obtain first-level first downsampled features, and a second convolution process is performed on the first downsampled features to obtain first-level low-frequency signal features. A first convolution process is performed on a high-frequency subband signal of the audio signal to obtain a first-level second convolution feature, a first pooling process is performed on the second convolution feature to obtain a first-level second pooled feature, a first downsampling process is performed on the second pooled feature to obtain a first-level second downsampling feature, and a second convolution process is performed on the second downsampling feature to obtain a first-level high-frequency signal feature.

[0064] Step 102: For the i-th level among the N levels, splice the audio signal and the (i-1)-th level signal features to obtain spliced ​​features, and perform i-th level feature extraction on the spliced ​​features to obtain i-th level signal features.

[0065] Here, N and i are integers greater than 1, and i is less than or equal to N.

[0066] After performing first-level feature extraction on the audio signal, feature extraction of remaining levels can be performed on the audio signal. In an embodiment of the present application, the remaining levels include N levels, and for an i-th level among the N levels, the audio signal and signal features of the (i-1)th level are spliced ​​together to obtain spliced ​​features, and i-level feature extraction is performed on the spliced ​​features to obtain i-level signal features. For example, for a second level, the audio signal and signal features of the first level are spliced ​​together to obtain spliced ​​features, and second-level feature extraction is performed on the spliced ​​features to obtain second-level signal features; for a third level, the audio signal and signal features of the second level are spliced ​​together to obtain spliced ​​features, and third-level feature extraction is performed on the spliced ​​features to obtain third-level signal features; and for a fourth level, the audio signal and signal features of the third level are spliced ​​together to obtain spliced ​​features, and fourth-level feature extraction is performed on the spliced ​​features to obtain fourth-level signal features.

[0067] In some embodiments, the audio signal includes a low-frequency subband signal and a high-frequency subband signal, and when processing the audio signal (e.g., feature extraction, encoding), the low-frequency subband signal and the high-frequency subband signal included in the audio signal can be processed, respectively. Based on this, for the i-th level among the N levels, subband decomposition can be further performed on the audio signal to obtain the low-frequency subband signal and the high-frequency subband signal of the audio signal. For the subband decomposition process, see steps 2011 to 2013 above. In this way, for the i-th level among the N levels, the data output by performing feature extraction includes the low-frequency signal features of the i-th level and the high-frequency signal features of the i-th level.

[0068] 8, which is a flowchart of an audio encoding method according to an embodiment of the present application. FIG. 8 shows that step 102 in FIG. 3 can be realized by steps 401 to 403. Step 401: splice a low-frequency subband signal of the audio signal with a low-frequency signal feature at an (i-1)th level to obtain a first splicing feature, and perform i-th level feature extraction on the first splicing feature to obtain an i-th level low-frequency signal feature. Step 402: splice a high-frequency subband signal of the audio signal with a high-frequency signal feature at an (i-1)th level to obtain a second splicing feature, and perform i-th level feature extraction on the second splicing feature to obtain an i-th level high-frequency signal feature. Step 403: The i-th level low-frequency signal feature and the i-th level high-frequency signal feature are taken as the i-th level signal feature.

[0069] It is necessary to explain that in step 401, after obtaining low-frequency subband signals and high-frequency subband signals of the audio signal, the low-frequency subband signals of the audio signal are spliced ​​with the low-frequency signal features extracted at the (i-1)th level to obtain first spliced ​​features, and then the first spliced ​​features are subjected to i-th level feature extraction to obtain i-th level low-frequency signal features. Similarly, in step 402, the high-frequency subband signals of the audio signal are spliced ​​with the high-frequency signal features extracted at the (i-1)th level to obtain second spliced ​​features, and then the second spliced ​​features are subjected to i-th level feature extraction to obtain i-th level high-frequency signal features. Thus, in step 403, the i-th level low-frequency signal features and the i-th level high-frequency signal features are taken as i-th level signal features.

[0070] In some embodiments, refer to FIG. 9, which is a flowchart of an audio encoding method provided by an embodiment of the present application. FIG. 9 shows that step 102 in FIG. 3 can be further realized by steps 501 to 504. Step 501: Perform a third convolution process on the spliced ​​features to obtain convolutional features of the i-th level. Step 502: Perform a second pooling process on the convolutional features to obtain pooled features of the i-th level. Step 503: Perform a second downsampling process on the pooled features to obtain downsampled features of the i-th level. Step 504: Perform a fourth convolution process on the downsampled features to obtain signal features of the i-th level.

[0071] It is necessary to explain that in step 501, a third convolution process can be performed on the spliced ​​feature (obtained by splicing the audio signal and the (i-1)th level signal feature). In practical application, the third convolution process can be performed by invoking a causal convolution with a preset number of channels, thereby obtaining the i-th level convolution feature.

[0072] In step 502, a second pooling process is performed on the convolutional features obtained in step 501. In practical application, the second pooling process can be performed by presetting a pooling factor, and then performing the second pooling process on the convolutional features according to the pooling factor to obtain the i-th level pooled features.

[0073] In step 503, a second downsampling is performed on the pooling features obtained in step 502. In practical applications, a downsampling factor can be preset, and downsampling is performed based on the downsampling factor. The second downsampling can be achieved by one coding layer or multiple coding layers. In some embodiments, the second downsampling can be achieved by X cascaded coding layers. Correspondingly, step 503 in FIG. 9 can be further implemented by steps 5031 to 5033. Step 5031: A first coding layer among the X cascaded coding layers performs a second downsampling on the pooling features to obtain a downsampling result of the first coding layer. Step 5032: A gth coding layer among the X cascaded coding layers performs a second downsampling on the downsampling result of the (g-1)th coding layer to obtain a downsampling result of the gth coding layer. Here, X and g are integers greater than 1, and g is less than or equal to X. Step 5033: Traverse g to obtain the downsampling result of the Xth coding layer, and take the downsampling result of the Xth coding layer as the i-th level downsampling feature.

[0074] It should be noted that in steps 5031 to 5033, the downsampling factors of each coding layer may be the same or different, and in practical applications, the downsampling factor corresponds to the pooling factor and performs the function of downward sampling.

[0075] In step 504, the downsampled features can be subjected to a fourth convolution process, which in practical application can be performed by invoking a causal convolution with a preset number of channels, thereby obtaining the i-th level signal features.

[0076] In practical application, steps 501 to 504 shown in Figure 9 can be implemented by invoking a second neural network model, which includes a third convolutional layer, a pooling layer, a downsampling layer, and a fourth convolutional layer. Thus, the third convolutional layer is invoked to perform a third convolutional process on the spliced ​​features to obtain the i-th level convolutional features, the pooling layer is invoked to perform a second pooling process on the convolutional features to obtain the i-th level pooled features, the downsampling layer is invoked to perform a second downsampling on the pooled features to obtain the i-th level downsampled features, and the fourth convolutional layer is invoked to perform a fourth convolutional process on the downsampled features to obtain the i-th level signal features. In practice, the feature dimensions of the signal features output by the second neural network may be smaller than the feature dimensions of the signal features input by the first neural network.

[0077] It should be noted that when performing the i-th level feature extraction, the i-th level feature extraction can also be performed on each of the low-frequency subband signals and the high-frequency subband signals of the audio signal through steps 501 to 504 shown in Fig. 9. That is, for the i-th level, a third convolution process is performed on the low-frequency splicing features (obtained by splicing the low-frequency subband signals and the (i-1)-th level low-frequency signal features) to obtain the i-th level convolution features, a second pooling process is performed on the convolution features to obtain the i-th level pooled features, a second downsampling process is performed on the pooled features to obtain the i-th level downsampling features, and a fourth convolution process is performed on the downsampling features to obtain the i-th level low-frequency signal features. For the i-th level, a third convolution process is performed on the high-frequency spliced ​​features (obtained by splicing the high-frequency subband signals and the high-frequency signal features of the (i-1)-th level) to obtain the i-th level convolved features, a second pooling process is performed on the convolved features to obtain the i-th level pooled features, a second downsampling process is performed on the pooled features to obtain the i-th level downsampled features, and a fourth convolution process is performed on the downsampled features to obtain the i-th level high-frequency signal features.

[0078] Step 103: Traverse i to obtain signal features for each level among the N levels.

[0079] Here, the data dimension of the signal features is smaller than the data dimension of the audio signal.

[0080] In step 102, the feature extraction process for the i-th level is described. In practical application, it is necessary to traverse i to obtain the signal features of each of the N levels. In the present embodiment, the data dimension of the signal features output by each level is smaller than the data dimension of the audio signal. In this way, the data dimension of the data related to the audio coding process can be reduced, and the coding efficiency of the audio coding can be improved.

[0081] Step 104: Encode the signal features of the first level and the signal features of each of the N levels to obtain a stream of audio signals at each level.

[0082] In practical application, after obtaining the signal features of the first level and each of the N levels, the signal features of the first level and each of the N levels are coded respectively, thereby obtaining an audio signal stream at each level, which can be transmitted to a receiving end of the audio signal, so that the receiving end decodes the audio signal as a decoder.

[0083] It should be noted that the signal feature output from the i-th level among the N levels can be understood as the residual signal feature between the signal feature output from the (i-1)-th level and the original audio signal. In this way, the signal feature of the extracted audio signal not only includes the signal feature of the extracted audio signal at the first level, but also includes the extracted residual signal features of each of the N levels. This makes the signal feature of the extracted audio signal more comprehensive and accurate, and reduces the loss of audio signal information in the feature extraction process. As a result, when the signal feature of the first level and the signal feature of each of the N levels are encoded, the quality of the encoded stream is higher, and the included audio signal information is closer to the original audio signal, thereby improving the encoding quality of the audio coding.

[0084] 3 can be realized by steps 104a1 to 104a2. Step 104a1: Quantization is performed on the signal features of the first level and on the signal features of each of the N levels, to obtain quantization results for the signal features of each level. Step 104a2: Entropy coding is performed on the quantization results for the signal features of each level, to obtain a stream of audio signals at each level.

[0085] It should be noted that in step 104a1, a quantization table can be preset, which includes a correspondence between signal features and quantization values. When performing quantization processing, the preset quantization table can be consulted to obtain corresponding quantization values ​​for the signal features of the first level and each of the N levels, and the obtained quantization values ​​are used as the quantization results. In step 104a2, entropy coding is performed on the quantization results of the signal features of each level to obtain audio signal streams at each level.

[0086] In practical applications, the audio signal includes a low-frequency subband signal and a high-frequency subband signal. In this case, the signal features output by each level accordingly include a low-frequency signal feature and a high-frequency signal feature. Based on this, when the signal features include a low-frequency signal feature and a high-frequency signal feature, in some embodiments, step 104 shown in FIG. 3 can be further implemented by steps 104b1 to 104b3. Step 104b1: Encode the low-frequency signal feature of the first level and the low-frequency signal feature of each of the N levels, respectively, to obtain a low-frequency stream of the audio signal at each level. Step 104b2: Encode the high-frequency signal feature of the first level and the high-frequency signal feature of each of the N levels, respectively, to obtain a high-frequency stream of the audio signal at each level. Step 104b3: Assume that the low-frequency stream and the high-frequency stream of the audio signal at each level are audio signal streams at the corresponding levels.

[0087] It should be noted that the encoding process of low-frequency signal features in step 104b1 may be implemented by adopting steps similar to steps 104a1 to 104a2. That is, quantization processes are performed on the low-frequency signal features of the first level and on the low-frequency signal features of each of the N levels, to obtain the quantization results of the low-frequency signal features of each level, and entropy coding is performed on the quantization results of the low-frequency signal features of each level, to obtain a low-frequency stream of the audio signal at each level. The encoding process of high-frequency signal features in step 104b2 may be implemented by adopting steps similar to steps 104a1 to 104a2. That is, quantization processes are performed on the high-frequency signal features of the first level and on the high-frequency signal features of each of the N levels, to obtain the quantization results of the high-frequency signal features of each level, and entropy coding is performed on the quantization results of the high-frequency signal features of each level, to obtain a high-frequency stream of the audio signal at each level.

[0088] In practical applications, audio signals include low-frequency subband signals and high-frequency subband signals. In this case, the signal features output by each level accordingly include low-frequency signal features and high-frequency signal features. Based on this, when the signal features include low-frequency signal features and high-frequency signal features, in some embodiments, step 104 shown in FIG. 3 can further be realized by steps 104c1 to 104c3. Step 104c1: Encode the low-frequency signal features of the first level according to a first encoding bit rate to obtain a first stream of the first level, and encode the high-frequency signal features of the first level according to a second encoding bit rate to obtain a second stream of the first level. Step 104c2: Perform the following processing on the signal features of each of the N levels: Encode the signal features of each of the levels according to a third encoding bit rate of the level to obtain a second stream of each level. Step 104c3: The second stream of the first level and the second stream of each of the N levels are regarded as audio signal streams at each level.

[0089] It should be noted that the first encoding bit rate is greater than the second encoding bit rate, and the second encoding bit rate is greater than the third encoding bit rate of any one of the N levels, and there is a positive correlation between the encoding bit rate of a level and the decoding quality indicator of the stream of the corresponding level. In step 104c2, a corresponding third encoding bit rate can be set for each of the N levels. The third encoding bit rates of each of the N levels may be the same, partially the same and partially different, or even completely different. Here, there is a positive correlation between the encoding bit rate of a level and the decoding quality indicator of the stream of the corresponding level. That is, the higher the encoding bit rate, the higher the (value of) the decoding quality indicator of the resulting stream. The first level contains the most audio signal features, and therefore the first encoding bit rate adopted by the first level low-frequency signal features is the largest, thereby ensuring the coding effect of the audio signal. At the same time, the high-frequency signal features of the first level are coded at a second coding bit rate that is smaller than the first coding bit rate, and the signal features of each of the N levels are coded at a third coding bit rate that is lower than the second coding bit rate, thereby increasing more features of the audio signal (including high-frequency signal features and residual signal features) while rationally allocating the coding bit rate of each level, thereby improving the coding efficiency of the audio signal.

[0090] In some embodiments, after the terminal obtains the audio signal streams at each level, it can further perform the following processing for each level: set a corresponding level transmission priority for the audio signal streams at the level, where the level transmission priority is negatively correlated with the level number of the level, and the level transmission priority is positively correlated with the decoding quality indicator of the stream at the corresponding level.

[0091] It should be noted that the level transmission priority of a level is used to characterize the transmission priority of the stream of that level. There is a negative correlation between the level transmission priority and the level number of a level, that is, the larger the level number, the lower the corresponding level transmission priority. For example, the level transmission priority of the first level (the level number is 1) is higher than the level transmission priority of the second level (the level number is 2). Based on this, when transmitting streams of each level to the decoding side, streams of the corresponding level can be transmitted according to the set level transmission priority. In practical application, when transmitting audio signal streams at multiple levels to the decoding side, streams of some levels or streams of all levels can be transmitted, and when transmitting streams of some levels, streams of the corresponding level can be transmitted according to the set level transmission priority.

[0092] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features, and the audio signal stream at each level includes a low-frequency stream obtained by encoding based on the low-frequency signal features and a high-frequency stream obtained by encoding based on the high-frequency signal features, and after obtaining the audio signal stream at each level, the terminal can further perform the following processing for each level respectively: set a first transmission priority for the low-frequency stream at level * and set a second transmission priority for the high-frequency stream at level *, where the first transmission priority is higher than the second transmission priority, and the second transmission priority at level (i-1) is lower than the first transmission priority at level i, and there is a positive correlation between the transmission priority of a stream and the decoding quality indicator of the corresponding stream.

[0093] It should be noted that there is a positive correlation between the transmission priority of a stream and the decoding quality indicator of the corresponding stream, and the data dimension of the high-frequency stream is smaller than that of the low-frequency stream. Therefore, the original information of the audio signal contained in the low-frequency stream at each level is greater than the original information of the audio signal contained in the high-frequency stream. That is, to ensure the decoding quality indicator of the low-frequency stream is higher than the decoding quality of the high-frequency stream, for each level, a first transmission priority can be set for the low-frequency stream at each level, and a second transmission priority can be set for the high-frequency stream at each level. The first transmission priority is higher than the second transmission priority. At the same time, the second transmission priority of the (i-1)th level can be set lower than the first transmission priority of the i-th level. That is, for each level, the transmission priority of the low-frequency stream is higher than the transmission priority of the high-frequency stream. In this way, the low-frequency stream at each level can be prioritized for transmission, and for multiple levels, the transmission priority of the low-frequency stream at the i-th level is higher than the transmission priority of the high-frequency stream at the (i-1)th level. This way, all low-frequency streams at multiple levels can be prioritized for transmission.

[0094] By applying the above embodiment of the present application, level-by-level coding of an audio signal can be realized by first performing first-level feature extraction on the audio signal to obtain first-level signal features, then splicing the audio signal and the (i-1)th-level signal features for the i-th level (i is an integer greater than 1 and is equal to or less than N) among N levels (N is an integer greater than 1) to obtain spliced ​​features, and then performing i-th-level feature extraction on the spliced ​​features to obtain i-level signal features, and then traversing i to obtain signal features for each of the N levels, and finally encoding the first-level signal features and the N-level signal features, respectively, to obtain an audio signal stream at each level.

[0095] By encoding an audio signal level by level, signal features of each level are obtained, and the data dimension of the signal features of each level is smaller than the data dimension of the audio signal, so the data dimension of the data processed in the audio encoding process is reduced, and the coding efficiency of the audio signal is improved.

[0096] When extracting signal features of an audio signal at each level, the output of each level is used as the input of the next level, and each level is combined with the extracted signal features of the previous level to perform more accurate feature extraction on the audio signal, and as the number of levels increases, the loss of audio signal information in the feature extraction process can be minimized. In this way, the audio signal information contained in the multiple streams obtained by encoding the signal features extracted in this manner is closer to the original audio signal, and the loss of audio signal information in the encoding process is reduced, ensuring the encoding quality of the audio encoding.

[0097] An audio decoding method provided by an embodiment of the present application will be described below. In some embodiments, the audio decoding method provided by the embodiment of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal alone, a server alone, or a terminal and a server working together. Taking the implementation by a terminal as an example, as shown in Figure 10, Figure 10 is a flowchart of the audio decoding method provided by the embodiment of the present application, and the audio decoding method provided by the embodiment of the present application includes the following steps 601 to 604.

[0098] Step 601: A terminal receives streams corresponding to a plurality of levels obtained by encoding an audio signal.

[0099] Here, the terminal, acting as the decoding side, receives streams corresponding to each of a plurality of levels obtained by encoding an audio signal.

[0100] Step 602: The streams at each level are decoded to obtain the signal features at each level.

[0101] Here, the data dimension of the signal features is smaller than the data dimension of the audio signal.

[0102] In some embodiments, the terminal decodes the streams of each level in the following manner to obtain the signal features of each level: For each level, the terminal performs the following processing: Entropy decoding is performed on the stream of the level to obtain the quantized value of the stream, and inverse quantization is performed on the quantized value of the stream to obtain the signal features of the level.

[0103] In practical application, the following processing can be performed for each level of stream: perform entropy decoding on the stream of that level to obtain the quantized value of the stream, and then perform inverse quantization on the quantized value of the stream according to the quantization table used in encoding the audio signal to obtain the stream, that is, query the quantized value of the stream and the corresponding signal features through the quantization table, thereby obtaining the signal features of that level.

[0104] In practical application, the received streams for each level may include a low-frequency stream and a high-frequency stream, where the low-frequency stream is obtained by encoding based on the low-frequency signal characteristics of the audio signal, and the high-frequency stream is obtained by encoding based on the high-frequency signal characteristics of the audio signal. Thus, when decoding the streams for each level, the low-frequency stream and the high-frequency stream for each level may be decoded separately. Here, the decoding process for the high-frequency stream and the low-frequency stream is similar to the decoding process for the stream. That is, the following process is performed for the low-frequency stream for each level: entropy decoding is performed for the low-frequency stream for that level to obtain the quantized value of the low-frequency stream, and inverse quantization is performed for the quantized value of the low-frequency stream to obtain the low-frequency signal characteristics for that level. The following process is performed for the high-frequency stream for each level: entropy decoding is performed for the high-frequency stream for that level to obtain the quantized value of the high-frequency stream, and inverse quantization is performed for the quantized value of the high-frequency stream to obtain the high-frequency signal characteristics for that level.

[0105] Step 603: Feature reconstruction is performed on the signal features of each level to obtain level audio signals of each level.

[0106] In practical applications, after obtaining signal features of each level through decoding, feature reconstruction is performed on the signal features of each level to obtain level audio signals of each level. In some embodiments, the terminal performs feature reconstruction on the signal features of each level in the following manner to obtain level audio signals of each level. The following processing is performed on the signal features of each level: perform a first convolution processing on the signal features to obtain level convolution features, upsample the convolution features to obtain level upsampling features, perform a pooling processing on the upsampling features, obtain level pooling features, and perform a second convolution processing on the pooling features to obtain level audio signals.

[0107] In practical application, the following processes are performed on the signal features of each level: First, a first convolution process is performed on the signal features, where the first convolution process is performed by invoking a causal convolution with a preset number of channels, thereby obtaining the convolution features of that level; Next, the convolution features are upsampled, where an upsampling factor is preset, and upsampling is performed based on the upsampling factor, thereby obtaining the upsampled features of that level; Then, a pooling process is performed on the upsampled features, where a pooling factor is preset, and further pooling is performed on the upsampled features based on the pooling factor, thereby obtaining the pooled features of that level; Finally, a second convolution process is performed on the pooled features, where the second convolution process is performed by invoking a causal convolution with a preset number of channels, thereby obtaining the level audio signal of that level.

[0108] The upsampling may be achieved by one decoding layer or multiple decoding layers. When the upsampling can be achieved by L (L>1) cascaded decoding layers, the terminal can upsample the convolutional features to obtain the upsampling feature of the level in the following manner: upsample the pooling feature through the first decoding layer of the L cascaded decoding layers to obtain the upsampling result of the first decoding layer; upsample the first upsampling result of the (k-1)th decoding layer through the kth decoding layer of the L cascaded decoding layers to obtain the upsampling result of the kth decoding layer, where L and k are integers greater than 1, and k is less than or equal to L; traverse k to obtain the upsampling result of the Lth decoding layer, and use the upsampling result of the Lth decoding layer as the upsampling feature of the level.

[0109] It should be noted that the upsampling factors of the individual decoding layers may be the same or different.

[0110] Step 604: Audio synthesis is performed on the level audio signals of the plurality of levels to obtain an audio signal.

[0111] In practical applications, after obtaining the level audio signals of each level, audio synthesis is performed on the level audio signals of multiple levels to obtain an audio signal.

[0112] In some embodiments, the stream includes a low-frequency stream and a high-frequency stream, and step 602 shown in FIG. 10 can be realized by the following steps: decode the low-frequency stream at each level to obtain the low-frequency signal feature at each level, and decode the high-frequency stream at each level to obtain the high-frequency signal feature at each level. Correspondingly, step 603 shown in FIG. 10 can be realized by the following steps: step 6031: perform feature reconstruction on the low-frequency signal feature at each level to obtain the level low-frequency subband signal at each level, and perform feature reconstruction on the high-frequency signal feature at each level to obtain the level high-frequency subband signal at each level. step 6032: use the level low-frequency subband signal and the level high-frequency subband signal as a level audio signal. Correspondingly, step 604 shown in FIG. 10 can be realized by the following steps: step 6041: add the level low-frequency subband signals at multiple levels to obtain a low-frequency subband signal, and add the level high-frequency subband signals at multiple levels to obtain a high-frequency subband signal. Step 6042: The low-frequency sub-band signal and the high-frequency sub-band signal are combined to obtain an audio signal.

[0113] In some embodiments, step 6042 can be realized by the following steps: Step 60421: Upsample the low-frequency subband signal to obtain a low-pass filtered signal; Step 60422: Upsample the high-frequency subband signal to obtain a high-frequency filtered signal; Step 60423: Perform filtering synthesis on the low-pass filtered signal and the high-frequency filtered signal to obtain an audio signal. It should be noted that in step 60423, synthesis processing can be performed by a QMF synthesis filter to obtain an audio signal.

[0114] Based on this, when the stream includes a low-frequency stream and a high-frequency stream, refer to Fig. 11, which is a flowchart of an audio decoding method provided by an embodiment of the present application. The audio decoding method provided by the embodiment of the present application includes step 701 of receiving low-frequency streams and high-frequency streams corresponding to each of a plurality of levels obtained by encoding an audio signal, step 702a of decoding the low-frequency streams at each level to obtain low-frequency signal features at each level, step 702b of decoding the high-frequency streams at each level to obtain high-frequency signal features at each level, step 703a of performing feature reconstruction on the low-frequency signal features at each level to obtain level-low-frequency sub-band signals at each level, and step 703b of performing feature reconstruction on the high-frequency signal features at each level to obtain level-high-frequency sub-band signals at each level. the step 703b of adding the plurality of levels of low-frequency subband signals to obtain a low-frequency subband signal, the step 704a of adding the plurality of levels of high-frequency subband signals to obtain a high-frequency subband signal, the step 705a of upsampling the low-frequency subband signal to obtain a low-pass filtered signal, the step 705b of upsampling the high-frequency subband signal to obtain a high-frequency filtered signal, and the step 706 of performing filtering synthesis on the low-pass filtered signal and the high-frequency filtered signal to obtain an audio signal.

[0115] It is necessary to explain that the feature reconstruction process of the high-frequency signal features and the low-frequency signal features can refer to the feature reconstruction process of the signal features in step 603. That is, for the high-frequency signal features of each level, the following processes are respectively performed: a first convolution process is performed on the high-frequency signal features to obtain high-frequency convolution features of the level, up-sampling the high-frequency convolution features, obtaining high-frequency up-sampled features of the level, performing a pooling process on the high-frequency up-sampled features, obtaining high-frequency pooled features of the level, and performing a second convolution process on the high-frequency pooled features to obtain a high-frequency level audio signal of the level; and for the low-frequency signal features of each level, the following processes are respectively performed: a first convolution process is performed on the low-frequency signal features to obtain low-frequency convolution features of the level, up-sampling the low-frequency convolution features, obtaining low-frequency up-sampled features of the level, performing a pooling process on the low-frequency up-sampled features, obtaining low-frequency pooled features of the level, and performing a second convolution process on the low-frequency pooled features to obtain a low-frequency level audio signal of the level.

[0116] In the above embodiment of the present application, a plurality of levels of streams are decoded respectively to obtain signal features of each level, and feature reconstruction is performed on the signal features of each level to obtain level audio signals of each level, and audio synthesis is performed on the level audio signals of the plurality of levels to obtain an audio signal. Since the data dimension of the signal features in the streams is smaller than the data dimension of the audio signal, it is smaller than the data dimension of the stream obtained by directly encoding the original audio signal in the related art, so that the data dimension of the data processed in the audio decoding process is reduced and the decoding efficiency of the audio signal is improved.

[0117] The following will describe an exemplary application of the embodiments of the present application in one practical application scenario.

[0118] Audio encoding and decoding technology transmits as much audio information as possible using relatively small network bandwidth resources. Audio codecs can achieve compression rates of over 10x, meaning that 10MB of audio data can be transmitted using only 1MB of compressed audio data. This significantly reduces the bandwidth required for transmitting information. To ensure smooth communication in communications systems, industry-wide standards for audio encoding and decoding are deployed. These include standards from international and domestic standards organizations such as the International Telecommunication Union Telecommunication Standardization Sector (ITU-T), the 3rd Generation Partnership Project (3GPP), the Internet Engineering Task Force (IETF), the Audio Video Coding Standard (AVS), and the China Communications Standards Association (CCSA), as well as standards such as G.711, G.722, AMR series, EVS, and OPUS. Figure 12 shows a schematic diagram of spectrum comparison at different bit rates to illustrate the relationship between compression bit rate and quality. Curve 1201 is the spectrum curve of the original speech, i.e., the uncompressed signal. Curve 1202 is the spectrum curve of the OPUS encoder at a bit rate of 20 kbps, and curve 1203 is the spectrum curve of the OPUS encoder at a bit rate of 6 kbps. As can be seen from Figure 12, as the encoding bit rate increases, the compressed signal becomes closer to the original signal.

[0119] Conventional audio coding can be divided into two types: time-domain coding and frequency-domain coding. Both are compression methods based on signal processing. 1) Time-domain coding, such as waveform speech coding, directly encodes the waveform of a speech signal. This type of coding method has the advantage of high quality encoded speech but low coding efficiency. In particular, speech signals can be coded using parameters, and the coding side only needs to extract the corresponding parameters of the speech signal to be transmitted. However, parameter coding has the advantage of extremely high coding efficiency but very low quality restored speech. 2) Frequency-domain coding converts an audio signal into the frequency domain, extracts frequency-domain coefficients, and then codes the frequency-domain coefficients. However, it also has poor coding efficiency. Thus, compression methods based on signal processing cannot ensure coding quality while improving coding efficiency.

[0120] Based on this, embodiments of the present application provide audio encoding methods and audio decoding methods, thereby improving encoding efficiency while ensuring encoding quality. In embodiments of the present application, even in low bitrate sections, different encoding methods can be freely selected according to the encoding content and network bandwidth conditions, and encoding efficiency can be improved when complexity and encoding quality are acceptable. Referring to Figure 13, Figure 13 is a flowchart of audio encoding and audio decoding provided by embodiments of the present application. Here, the number of levels is taken as two layers as an example (the present application does not limit the iterative operation of a third layer or higher levels). The audio encoding method provided by embodiments of the present application includes:

[0121] (1) Subband decomposition is performed on an audio signal to obtain a low-frequency subband signal and a high-frequency subband signal. In actual implementation, the audio signal is sampled according to a first sampling frequency to obtain a sampled signal, and then subband decomposition is performed on the sampled signal to obtain subband signals having frequencies lower than the first sampling frequency, including a low-frequency subband signal and a high-frequency subband signal. For example, an analysis filter (e.g., a QMF filter) is used to decompose the audio signal of the nth frame [Equation 1] into a low-frequency subband signal [Equation 2] and a high-frequency subband signal [Equation 3].

[0122]

number

number

number

[0123] (2) Analyze the low-frequency subband signal based on the first-layer low-frequency analysis neural network to obtain the first-layer low-frequency signal features. For example, for the low-frequency subband signal [Equation 4], invoke the first-layer low-frequency analysis neural network to obtain the first-layer low-dimensional low-frequency signal features [Equation 5]. It should be noted that the dimension of the signal features is smaller than the dimension of the low-frequency subband signal (thereby reducing the amount of data), and the neural network includes, but is not limited to, dilated CNN, autoencoder, full-connection, LSTM, CNN+LSTM, etc.

[0124]

number

number

[0125] (3) Analyze the high-frequency subband signal based on the first-layer high-frequency analysis neural network to obtain the high-frequency signal features of the first layer. For example, for the high-frequency subband signal [Equation 6], invoke the first-layer high-frequency analysis neural network to obtain the low-dimensional first-layer high-frequency signal features [Equation 7].

[0126]

number

number

[0127] (4) Analyze the low-frequency subband signals and the low-frequency signal features of the first layer based on the low-frequency analysis neural network of the second layer to obtain the low-frequency signal features of the second layer (i.e., the low-frequency residual signal features of the second layer). For example, by combining [Number 8] and [Number 9], the low-frequency analysis neural network of the second layer is called to obtain the low-dimensional low-frequency signal features of the second layer [Number 10].

[0128]

number

number

number

[0129] (5) Based on the second-layer high-frequency analysis neural network, the high-frequency subband signals and the first-layer high-frequency signal features are analyzed to obtain the second-layer high-frequency signal features (i.e., the second-layer high-frequency residual signal features). For example, [Equation 11] and [Equation 12] are combined to call the second-layer high-frequency analysis neural network to obtain the low-dimensional second-layer high-frequency signal features [Equation 13].

[0130]

number

number

number

[0131] (6) The quantization and coding part quantizes and codes the signal features of two layers (including the low-frequency signal features of the first layer, the high-frequency signal features of the first layer, the low-frequency signal features of the second layer, and the high-frequency signal features of the second layer) to obtain audio signal streams for each layer, and assigns a transmission priority to each layer stream. For example, it can be inferred that the first layer is transmitted with a higher priority, the second layer is transmitted with a lower priority, and so on.

[0132] In practical applications, the decoding side may receive only one layer of stream, and can use a "single-layer decoding" method for decoding, as shown in FIG. 13. Based on this, the audio decoding method provided in the embodiments of the present application includes the following: (1) Decode the received one-layer stream to obtain low-frequency signal features and high-frequency signal features of the layer; (2) Analyze the low-frequency signal features based on the low-frequency synthesis neural network of the first layer to obtain an estimated value of the low-frequency subband signal. For example, invoke the low-frequency synthesis neural network of the first layer based on the quantized value of the low-frequency signal feature [Equation 14] to generate an estimated value of the low-frequency subband signal [Equation 15]; (3) Analyze the high-frequency signal features based on the high-frequency synthesis neural network of the first layer to obtain an estimated value of the high-frequency subband signal. For example, invoke the high-frequency synthesis neural network of the first layer based on the quantized value of the high-frequency signal feature [Equation 16] to generate an estimated value of the high-frequency subband signal [Equation 17]. (4) Based on the estimated low-frequency subband signal [Equation 18] and the estimated high-frequency subband signal [Equation 19], synthesis filtering is performed by the synthesis filter, and finally a reconstructed audio signal at the original sampling frequency [Equation 20] is obtained, thereby completing the decoding process.

[0133]

number

number

number

number

number

number

number

[0134] In practical application, the decoding side may receive both of the two-layer streams, and can use the "two-layer decoding" method for decoding, as shown in Figure 13. Based on this, the audio decoding method provided in the embodiments of this application includes:

[0135] (1) The received stream of each layer is decoded to obtain the low-frequency signal features and high-frequency signal features of each layer.

[0136] (2) Analyze the low-frequency signal features of the first layer based on the low-frequency synthesis neural network of the first layer to obtain the estimated values ​​of the low-frequency subband signals of the first layer. For example, invoke the low-frequency synthesis neural network of the first layer based on the quantized values ​​of the low-frequency signal features of the first layer [Equation 21] to generate the estimated values ​​of the low-frequency subband signals of the first layer [Equation 22].

[0137]

number

number

[0138] (3) Analyze the high-frequency signal features of the first layer based on the high-frequency synthesis neural network of the first layer to obtain the estimated values ​​of the high-frequency subband signals of the first layer. For example, invoke the high-frequency synthesis neural network of the first layer based on the quantized values ​​of the high-frequency signal features of the first layer [Equation 23] to generate the estimated values ​​of the high-frequency subband signals of the first layer [Equation 24].

[0139]

number

number

[0140] (4) Analyze the low-frequency signal features of the second layer based on the low-frequency synthesis neural network to obtain an estimate of the low-frequency subband residual signal of the second layer. For example, invoke the low-frequency synthesis neural network of the second layer based on the quantized value of the low-frequency signal features of the second layer [Equation 25] to generate an estimate of the low-frequency subband residual signal [Equation 26].

[0141]

number

number

[0142] (5) Analyze the high-frequency signal features of the second layer based on the high-frequency synthesis neural network to obtain the estimated value of the high-frequency subband residual signal of the second layer. For example, invoke the high-frequency synthesis neural network of the second layer based on the quantized value of the high-frequency signal features of the second layer [Equation 27] to generate the estimated value of the high-frequency subband residual signal [Equation 28].

[0143]

number

number

[0144] (6) Using the low-frequency part, the estimated value of the low-frequency subband signal of the first layer and the estimated value of the low-frequency subband residual signal are summed to obtain the estimated value of the low-frequency subband signal. For example, the sum of [Equation 29] and [Equation 30] is obtained to obtain the estimated value of the low-frequency subband signal.

[0145]

number

number

[0146] (7) The high-frequency part is used to sum the estimated value of the high-frequency subband signal of the first layer and the estimated value of the high-frequency subband residual signal to obtain the estimated value of the high-frequency subband signal. For example, [Equation 31] is summed with [Equation 32] to obtain the estimated value of the high-quality high-frequency subband signal.

[0147]

number

number

[0148] (8) Based on the estimated low-frequency subband signal and the estimated high-frequency subband signal, synthesis filtering is performed by a synthesis filter, and finally a reconstructed audio signal at the original sampling frequency [Equation 33] is obtained, thereby completing the decoding process.

[0149]

number

[0150] The embodiments of the present application can be applied to various audio scenarios, such as long-distance voice communication. Taking long-distance voice communication as an example, refer to FIG. 14, which is a schematic diagram of an audio communication link provided by an embodiment of the present application. Here, a Voice over Internet Protocol (VoIP) conference system is taken as an example. The audio encoding and decoding technology of the embodiment of the present application is implemented in the encoding and decoding parts to solve the basic function of audio compression. The encoder is implemented in the uplink client 1401, and the decoder is implemented in the downlink client 1402. The uplink client collects audio and performs pre-processing, enhancement, encoding, etc. The encoded stream is transmitted to the downlink client 1402 via a network, where it is decoded, enhanced, etc., and then the downlink client 1402 plays the decoded audio.

[0151] Considering forward compatibility (i.e., the new encoder and the existing encoder are compatible), a transcoder needs to be deployed in the background of the system (i.e., the server) to solve the interconnection and intercommunication problem between the new encoder and the existing encoder. For example, if the sender (uplink client) is a new NN encoder, the receiver (downlink client) is a Public Switched Telephone Network (PSTN) decoder (e.g., a G.722 decoder). Therefore, after receiving the stream sent by the sender, the server must first run an NN decoder to generate an audio signal so that the receiver can accurately decode it, and then call the G.722 encoder to generate a specific stream. The same transcoding scenario will not be further described.

[0152] Before introducing the audio encoding method and audio decoding method provided in the embodiments of the present application in detail, the QMF filter bank and the dilated convolution network will be first introduced below.

[0153] A QMF filter bank is a filter pair that includes analysis and synthesis. For a QMF analysis filter, an input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, which represent a QMF low-pass signal and a QMF high-pass signal, respectively. The spectral responses of the low-pass part [Equation 34] and high-pass part [Equation 35] of the QMF filter are shown in Figure 15. Based on the relevant theoretical knowledge of the QMF analysis filter bank, the correlation between the coefficients of the low-pass filtering and high-pass filtering can be easily described, which is shown in Equation (1). [Number 36]

[0154]

number

number

number

[0155] Here, [Equation 37] represents the coefficients of low-pass filtering, and [Equation 38] represents the coefficients of high-pass filtering.

[0156]

number

number

[0157] Similarly, according to the QMF related theory, based on the QMF analysis filter banks [Equation 39] and [Equation 40], the QMF synthesis filter bank can be written, as shown in equation (2) [Equation 41].

[0158]

number

number

number

[0159] Here, [Equation 42] represents the restored low-pass signal, and [Equation 43] represents the restored high-pass signal.

[0160]

number

number

[0161] When the low-pass and high-pass signals restored by the decoder are synthesized by a QMF synthesis filter bank, a reconstructed signal with a sampling rate Fs corresponding to the input signal can be restored.

[0162] Referring to FIGS. 16A and 16B, FIG. 16A is a schematic diagram of a conventional convolutional network provided in an embodiment of the present application, and FIG. 16B is a schematic diagram of a dilated convolutional network provided in an embodiment of the present application. Compared to a conventional convolutional network, a dilated convolution can increase the receptive field while maintaining the dimensions of the feature map unchanged, and can also avoid errors caused by upsampling and downsampling. The convolution kernel sizes shown in FIGS. 16A and 16B are both 3×3, but the receptive field 901 of the conventional convolution shown in FIG. 16A is only 3, while the receptive field 902 of the dilated convolution shown in FIG. 16B reaches 5. That is, for a convolution kernel with dimensions of 3×3, the receptive field of the conventional convolution shown in FIG. 16A is 3, the dilation rate (the number of points spaced apart in the convolution kernel) is 1, and the receptive field of the dilated convolution shown in FIG. 16B is 5, and the dilation rate is 2.

[0163] The convolution kernel can also move on a plane similar to that shown in FIG. 16A or 16B, where the concept of stride rate (step size) is used. For example, each time the convolution kernel moves by one grid, the corresponding shift rate is 1. In addition, there is also the concept of the number of convolution channels, that is, the number of parameters corresponding to the convolution kernel used to perform the convolution analysis. Theoretically, the more channels there are, the more comprehensive and accurate the analysis of the signal will be, but the higher the number of channels, the greater the complexity. For example, for a 1×320 tensor, a 24-channel convolution operation can be used, resulting in a 24×320 tensor. It should be noted that the size of the dilation convolution kernel (e.g., for a voice signal, the size of the convolution kernel can be set to 1×3), the dilation rate, the shift rate, and the number of channels can be defined according to the needs of the actual application, but this is not limited to the embodiments of the present application.

[0164] In the following, an audio signal with Fs=32000Hz is taken as an example (the embodiments of the present application can also be applied to scenes with other sampling frequencies, including but not limited to 8000Hz, 16000Hz, 48000Hz, etc.), where the frame length is set to 20ms, and for Fs=32000Hz, one frame contains 640 sample points.

[0165] Next, with continued reference to Figure 13, the audio encoding method and audio decoding method provided by the embodiments of the present application will be described in detail respectively. Here, the audio encoding method provided by the embodiments of the present application includes: First step: generating an input signal.

[0166] Here, the 640 sample points of the n-th frame are written as [Equation 44].

[0167]

number

[0168] Second step: QMF subband signal decomposition.

[0169] Here, a QMF analysis filter (e.g., a two-channel QMF filter) is called to perform filtering, and the filtered signal obtained by filtering is downsampled to obtain two subband signals, namely, a low-frequency subband signal [Equation 45] and a high-frequency subband signal [Equation 46]. Here, the effective bandwidth of the low-frequency subband signal [Equation 47] is 0-8 kHz, the effective bandwidth of the high-frequency subband signal [Equation 48] is 8-16 kHz, and the number of sample points per frame is 320.

[0170]

number

number

number

number

[0171] Third step: Low frequency analysis of the first layer.

[0172] Here, the purpose of invoking the first-layer low-frequency analysis neural network is to generate a first-layer low-frequency signal feature [Equation 50] with lower dimensions based on the low-frequency subband signal [Equation 49]. In this example, the data dimension of [Equation 51] is 320, and the data dimension of [Equation 52] is 64. In view of the data volume, it is clear that the effect of "dimensionality reduction" is achieved after passing through the first-layer low-frequency analysis neural network, which can be understood as data compression. For example, refer to FIG. 17, which is a structural diagram of the first-layer low-frequency analysis neural network provided by an embodiment of the present application. The processing process for the low-frequency subband signal [Equation 53] includes the following:

[0173]

number

number

number

number

number

[0174] (1) Call one 24-channel causal convolution and expand the input tensor (i.e., [Number 54]) into a 24*320 tensor.

[0175]

number

[0176] (2) Preprocessing is performed on a 24*320 tensor. In practical applications, a pooling operation with a pooling factor of 2 can be performed, and the activation function can be ReLU, thereby generating a 24*160 tensor.

[0177] (3) Cascading coding blocks with three different down-sampling factors (Down_factor). Taking a coding block (Down_factor=4) as an example, one or more dilation convolutions can be performed. The size of each convolution kernel is fixed at 1*3, and the stride rate is 1. In addition, the dilation rate of the one or more dilation convolutions can be set according to needs, for example, 3. Of course, the embodiment of the present application does not restrict different dilation rates to be set for different dilation convolutions. Next, the Down_factors of the three coding blocks are set to 4, 5, and 8, respectively, which is equivalent to setting pooling factors of different sizes, thereby achieving the down-sampling effect. Finally, the number of channels of the three coding blocks is set to 48, 96, and 192, respectively. Therefore, the three cascaded coding blocks sequentially convert a 24*160 tensor into 48*40, 96*8, and 192*1 tensors, respectively.

[0178] (4) For the 192*1 tensor, a causal convolution similar to the preprocessing is performed to output a single 64-dimensional feature vector, i.e., the low-frequency signal feature of the first layer [Equation 55].

[0179]

number

[0180] Fourth step: High frequency analysis of the first layer.

[0181] Here, the purpose of invoking the first-layer high-frequency analysis neural network is to generate a first-layer high-frequency signal feature [Equation 57] with lower dimensions based on the high-frequency subband signal [Equation 56]. In this example, the structure of the first-layer high-frequency analysis neural network can be consistent with the first-layer low-frequency analysis neural network. That is, the input data dimension (i.e., [Equation 58]) is 320 dimensions, and the output data dimension (i.e., [Equation 59]) is 64 dimensions. Considering that the importance of high-frequency subband signals is relatively lower than that of low-frequency subband signals, the output dimension can be appropriately reduced. In this way, the complexity of the first-layer high-frequency analysis neural network can be reduced, but this example does not limit it.

[0182]

number

number

number

number

[0183] Fifth step: Low-frequency analysis of the second layer.

[0184] Here, the purpose of invoking the low-frequency analysis neural network in the second layer is to obtain lower-dimensional low-frequency signal features [Equation 62] in the second layer based on the low-frequency subband signals [Equation 60] and the low-frequency signal features [Equation 61] in the first layer. The low-frequency signal features in the second layer reflect the residual of the audio signal reconstructed at the decoding side by the output of the low-frequency analysis neural network in the first layer relative to the original audio signal. Therefore, at the decoding side, the residual signal of the low-frequency subband signal is predicted according to [Equation 63], and is summed with the estimated value of the low-frequency subband signal predicted by the output of the low-frequency analysis neural network in the first layer to obtain a more accurate estimate of the low-frequency subband signal.

[0185]

number

number

number

number

[0186] The second-layer low-frequency analysis neural network has a similar structure to the first-layer low-frequency analysis neural network. Referring to FIG. 18, FIG. 18 is a structural diagram of the second-layer low-frequency analysis neural network provided in the embodiment of the present application. Here, the main differences from the first-layer low-frequency analysis neural network include the following: (1) The input of the second-layer low-frequency analysis neural network, in addition to including the low-frequency subband signal [Equation 64], also includes the output [Equation 65] of the first-layer low-frequency analysis neural network. The two variables [Equation 66] and [Equation 67] can be spliced ​​into a 384-dimensional spliced ​​feature. (2) Considering that the second-layer low-frequency analysis processes the residual signal, the dimension of the output [Equation 68] of the second-layer low-frequency analysis neural network is set to 28.

[0187]

number

number

number

number

number

[0188] Sixth step: High frequency analysis of the second layer.

[0189] Here, the purpose of invoking the high-frequency analysis neural network in the second layer is to obtain the high-frequency signal features [Equation 71] of the second layer with lower dimensions based on the high-frequency subband signals [Equation 69] and the high-frequency signal features [Equation 70] of the first layer. The structure of the high-frequency analysis neural network in the second layer may be the same as that of the low-frequency analysis neural network in the second layer. That is, the input data dimension (the spliced ​​features of [Equation 72] and [Equation 73]) is 384 dimensions, and the output data dimension ([Equation 74]) is 28 dimensions.

[0190]

number

number

number

number

number

number

[0191] Seventh step: quantization coding.

[0192] The signal features output from the second layer are quantized by consulting a preset quantization table, and the quantized result is then coded. The quantization can be scalar quantization (each component is quantized independently), and the coding can be entropy coding. The embodiments of the present application do not limit the technical combination of vector quantization (adjacent components are combined into a vector and jointly quantized) and entropy coding.

[0193] In practice, the low-frequency signal features

[75] in the first layer are 64-dimensional features that can be encoded using 8 kbps, with an average bit rate of 2.5 bits per parameter per frame. The high-frequency signal features

[76] in the first layer are 64-dimensional features that can be encoded using 6 kbps, with an average bit rate of 1.875 bits per parameter per frame. Therefore, the total encoding speed for the first layer is 14 kbps.

[0194]

number

number

[0195] In practice, the low-frequency signal features

[77] in the second layer are 28-dimensional features, which can be encoded using 3.5 kbps, with an average bit rate of 2.5 bits per parameter per frame. The high-frequency signal features

[78] in the second layer are 28-dimensional features, which can be encoded using 3.5 kbps, with an average bit rate of 2.5 bits per parameter per frame. Therefore, the total encoding time for the second layer is 7 kbps.

[0196]

number

number

[0197] Based on this, different feature vectors can be progressively coded by a level-by-level coding scheme. According to different application scenarios, the embodiments of the present application do not limit the bit rate distribution of other schemes, and for example, further coding of the third layer or higher layers can be iteratively introduced. After quantization coding, streams can be generated, and different transmission strategies can be adopted for streams of different layers to ensure transmission with different priorities, for example, forward error correction (FEC) can be adopted to improve transmission quality through redundant transmission, and the redundancy multiples of different layers are not the same, for example, the redundancy multiple of the first layer can be set slightly higher.

[0198] Taking the example that all the layer streams are received by the decoding side and decoded correctly, the audio encoding method provided in the embodiment of the present application includes:

[0199] First step: Decryption.

[0200] Here, decoding is the reverse process of encoding. The received stream is analyzed and a quantization table is looked up to obtain estimates of low-frequency signal features and high-frequency signal features. For example, in the first layer, quantized values ​​of 64-dimensional signal features of the low-frequency subband signal [Equation 79] and 64-dimensional signal features of the high-frequency subband signal [Equation 80] are obtained, and in the second layer, quantized values ​​of 28-dimensional signal features of the low-frequency subband signal [Equation 81] and 28-dimensional signal features of the high-frequency subband signal [Equation 82] are obtained.

[0201]

number

number

number

number

[0202] Second step: Low frequency synthesis of the first layer.

[0203] Here, the purpose of invoking the first-layer low-frequency synthesis neural network is to generate an estimated value [Equation 84] of the first-layer low-frequency subband signal based on the quantized value [Equation 83] of the low-frequency feature vector. For example, refer to FIG. 19, which is a schematic diagram of a model of a first-layer low-frequency synthesis neural network provided by an embodiment of the present application. Here, the processing process of the first-layer low-frequency synthesis neural network is similar to that of the first-layer low-frequency analysis neural network, such as causal convolution. The post-processing structure of the first-layer low-frequency synthesis neural network is similar to the pre-processing structure of the first-layer low-frequency analysis neural network, and the structure of the decoding block is symmetrical to that of the encoding block. For the encoding block on the encoding side, first, dilation convolution is performed, followed by pooling to complete downward sampling. For the decoding block on the decoding side, first, pooling is performed to complete upward sampling, followed by dilation convolution.

[0204]

number

number

[0205] Third step: High frequency synthesis of the first layer.

[0206] Here, the structure of the first-layer high-frequency synthesis neural network is the same as the structure of the first-layer low-frequency synthesis neural network, and an estimate of the first high-frequency subband signal [Equation 86] can be obtained according to the quantized value of the low-frequency signal feature of the first layer [Equation 85].

[0207]

number

number

[0208] Fourth step: Second layer low frequency synthesis.

[0209] Here, the purpose of invoking the low-frequency synthesis neural network of the second layer is to generate an estimated value of the low-frequency sub-band residual signal [Number 88] based on the quantized value of the low-frequency signal feature of the second layer [Number 87]. Referring to Figure 20, Figure 20 is a structural schematic diagram of the low-frequency synthesis neural network of the second layer provided by the embodiment of the present application, the structure of the low-frequency synthesis neural network of the second layer is similar to the structure of the low-frequency synthesis neural network of the first layer, except that the input data dimension is 28 dimensions.

[0210]

number

number

[0211] Fifth step: Second layer high frequency synthesis.

[0212] Here, the structure of the low-frequency synthesis neural network in the second layer is the same as that of the low-frequency synthesis neural network in the second layer, and an estimate of the high-frequency subband residual signal [Equation 90] can be generated based on the quantized value of the low-frequency signal feature in the second layer [Equation 89].

[0213]

number

number

[0214] Sixth step: synthesis filtering.

[0215] Based on the previous steps, the decoding side obtains the low-frequency subband signal estimate [Equation 91] and the high-frequency subband signal [Equation 92], and the low-frequency subband residual signal estimate [Equation 93] and the high-frequency subband residual signal estimate [Equation 94]. Add [Equation 95] and [Equation 96] to generate a highly accurate low-frequency subband signal estimate, and add [Equation 97] and [Equation 98] to generate a highly accurate high-frequency subband signal estimate. Finally, upsampling is performed on the low-frequency subband signal estimate and the high-frequency subband signal estimate, and a QMF synthesis filter is invoked to perform synthesis filtering on the upsampling result. In this case, a 640-point reconstructed audio signal [Equation 99] is generated.

[0216]

number

number

number

number

number

number

number

number

number

[0217] In the embodiment of the present application, by collecting data, the related neural networks of the encoding side and the decoding side can be jointly trained to obtain optimal parameters, and the trained network model can be put into use. In the embodiment of the present application, only a specific example of network input, network structure, and network output is disclosed, but those skilled in the art can modify the above configuration according to their needs.

[0218] By applying the above-described embodiments of the present application, a technical solution for low-bitrate audio encoding and decoding based on signal processing and deep learning networks can be achieved. By organically combining signal decomposition and related signal processing technologies with deep neural networks, coding efficiency is significantly improved compared to related technologies, and when complexity is acceptable, coding quality is also improved. Depending on different coding content and bandwidth conditions, the encoding side selects different transmission strategies for each level to transmit the stream. The decoding side receives the lower-layer stream and outputs an audio signal of acceptable quality, and if it also receives other upper-layer streams, it can output high-quality audio.

[0219] As can be understood, in the embodiments of the present application, with respect to related data such as user information (e.g., audio signals transmitted by a user), when the embodiments of the present application are used in a product or technology, user permission or consent must be obtained, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0220] The following will continue to describe an exemplary structure of the audio encoding device 553 provided by the embodiments of the present application, which is implemented as a software module. In some embodiments, as shown in FIG. 2, the software modules in the audio encoding device 553 stored in the memory 550 may include a first feature extraction module 5531, a second feature extraction module 5532, a traversal module 5533, and an encoding module 5534.

[0221] The first feature extraction module 5531 is configured to perform first-level feature extraction on the audio signal to obtain the first-level signal features, and the second feature extraction module 5532 is configured to splice the audio signal and (i-1)th-level signal features for an i-th level out of N levels to obtain spliced ​​features, and to perform i-level feature extraction on the spliced ​​features to obtain the i-th level signal features. N and i are integers greater than 1, and i is less than or equal to N. The traversal module 5533 is configured to traverse i and obtain signal features for each of the N levels. A data dimension of the signal features is smaller than a data dimension of the audio signal, and the encoding module 5534 is configured to encode the first-level signal features and the signal features for each of the N levels, respectively, to obtain a stream of the audio signal at each level.

[0222] In some embodiments, the first feature extraction module 5531 is further configured to perform subband decomposition on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal, perform first-level feature extraction on the low-frequency subband signals to obtain the first-level low-frequency signal features, and perform first-level feature extraction on the high-frequency subband signals to obtain the first-level high-frequency signal features, and to use the low-frequency signal features and the high-frequency signal features as the first-level signal features.

[0223] In some embodiments, the first feature extraction module 5531 is further configured to sample the audio signal according to a first sampling frequency, to obtain a sampled signal, low-pass filter the sampled signal, to obtain a low-pass filtered signal, and down-sample the low-pass filtered signal, to obtain the low-frequency sub-band signal at a second sampling frequency, high-pass filter the sampled signal, to obtain a high-pass filtered signal, and down-sample the high-pass filtered signal, to obtain the high-frequency sub-band signal at a second sampling frequency, where the second sampling frequency is lower than the first sampling frequency.

[0224] In some embodiments, the second feature extraction module 5532 is further configured to: splice a low-frequency subband signal of the audio signal with an (i-1)th-level low-frequency signal feature to obtain a first spliced ​​feature; perform an i-th level feature extraction on the first spliced ​​feature to obtain the i-th level low-frequency signal feature; splice a high-frequency subband signal of the audio signal with an (i-1)th-level high-frequency signal feature to obtain a second spliced ​​feature; perform an i-th level feature extraction on the second spliced ​​feature to obtain the i-th level high-frequency signal feature; and use the i-th level low-frequency signal feature and the i-th level high-frequency signal feature as the i-th level signal feature.

[0225] In some embodiments, the first feature extraction module 5531 is further configured to perform a first convolution operation on the audio signal to obtain the first-level convolution features, perform a first pooling operation on the convolution features to obtain the first-level pooled features, perform a first downsampling operation on the pooled features to obtain the first-level downsampled features, and perform a second convolution operation on the downsampled features to obtain the first-level signal features.

[0226] In some embodiments, the first downsampling is realized by M cascaded encoding layers, and the first feature extraction module 5531 is further configured to: perform a first downsampling on the pooling feature by a first encoding layer of the M cascaded encoding layers to obtain a downsampling result of the first encoding layer; and perform a first downsampling on the downsampling result of the (j-1)th encoding layer by a j-th encoding layer of the M cascaded encoding layers to obtain a downsampling result of the j-th encoding layer, where M and j are integers greater than 1 and j is less than or equal to M; traverse j to obtain the downsampling result of the M-th encoding layer, and use the downsampling result of the M-th encoding layer as the first-level downsampling feature.

[0227] In some embodiments, the second feature extraction module 5532 is further configured to perform a third convolution operation on the spliced ​​features to obtain the i-th level convolution features, perform a second pooling operation on the convolution features to obtain the i-th level pooled features, perform a second downsampling operation on the pooled features to obtain the i-th level downsampled features, and perform a fourth convolution operation on the downsampled features to obtain the i-th level signal features.

[0228] In some embodiments, the encoding module 5534 is further configured to perform a quantization process on the signal features of the first level and the signal features of each of the N levels, respectively, to obtain quantization results of the signal features of each level, and to perform entropy encoding on the quantization results of the signal features of each level, to obtain the stream of the audio signal at each level.

[0229] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features. The encoding module 5534 is further configured to: encode the low-frequency signal features of the first level and the low-frequency signal features of each of the N levels, respectively, to obtain a low-frequency stream of the audio signal at each level; encode the high-frequency signal features of the first level and the high-frequency signal features of each of the N levels, respectively, to obtain a high-frequency stream of the audio signal at each level; and configure the low-frequency stream and high-frequency stream of the audio signal at each level as the stream of the audio signal at a corresponding level.

[0230] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features. The encoding module 5534 is further configured to: encode the low-frequency signal features of the first level according to a first encoding bit rate to obtain a first stream of the first level, and encode the high-frequency signal features of the first level according to a second encoding bit rate to obtain a second stream of the first level, and perform the following processing on the signal features of each of the N levels: encode the signal features of each of the levels according to a third encoding bit rate of the level to obtain a second stream of each of the levels, and the second stream of the first level and the second stream of each of the N levels constitute the stream of the audio signal at each level, where the first encoding bit rate is greater than the second encoding bit rate, and the second encoding bit rate is greater than a third encoding bit rate of any one of the N levels, and the encoding bit rates of the levels are positively correlated with the decoding quality indicator of the stream of the corresponding level.

[0231] In some embodiments, the encoding module 5534 is further configured to, for each of the levels, respectively: set a corresponding level transmission priority for the stream of the audio signal at the level, where the level transmission priority is negatively correlated with the level number of the level, and the level transmission priority is positively correlated with the decoding quality indicator of the stream at the corresponding level.

[0232] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features, and the audio signal stream at each level includes a low-frequency stream obtained by encoding based on the low-frequency signal features and a high-frequency stream obtained by encoding based on the high-frequency signal features. The encoding module 5534 is further configured to perform the following processing for each of the levels: set a first transmission priority for the low-frequency stream at that level and set a second transmission priority for the high-frequency stream at that level, where the first transmission priority is higher than the second transmission priority, and the second transmission priority at the (i-1)th level is lower than the first transmission priority at the i-th level, and the transmission priority of the stream is positively correlated with the decoding quality indicator of the corresponding stream.

[0233] By applying the above embodiment of the present application, level-by-level coding of an audio signal can be realized by first performing a first-level feature extraction on the audio signal to obtain first-level signal features, then splicing the audio signal and the (i-1)th-level signal features for the i-th level (i is an integer greater than 1 and is equal to or less than N) among N levels (N is an integer greater than 1) to obtain spliced ​​features, and then performing an i-th-level feature extraction on the spliced ​​features to obtain i-level signal features, and then traversing i to obtain signal features for each of the N levels, and finally encoding the first-level signal features and the N-level signal features, respectively, to obtain an audio signal stream at each level.

[0234] First, the data dimension of the extracted signal features is smaller than that of the audio signal, thus reducing the data dimension of the data processed in the audio coding process and improving the coding efficiency of the audio signal.

[0235] Second, when extracting signal features of an audio signal at each level, the output of each level is used as the input of the next level. This allows each level to be combined with the extracted signal features of the previous level to perform more accurate feature extraction on the audio signal, minimizing information loss of the audio signal during the feature extraction process as the number of levels increases. In this way, the audio signal information contained in the multiple streams obtained by encoding the signal features extracted in this manner is closer to the original audio signal, reducing information loss of the audio signal during the encoding process and ensuring the encoding quality of the audio encoding.

[0236] The following describes an audio decoding device provided by an embodiment of the present application, which includes: a receiving module configured to receive streams corresponding to a plurality of levels obtained by encoding an audio signal; a decoding module configured to decode the streams of each of the levels to obtain signal features of each of the levels, where a data dimension of the signal features is smaller than a data dimension of the audio signal; a feature reconstruction module configured to perform feature reconstruction on the signal features of each of the levels to obtain level audio signals of each of the levels; and an audio synthesis module configured to perform audio synthesis on the level audio signals of the plurality of levels to obtain the audio signals.

[0237] In some embodiments, the streams include a low-frequency stream and a high-frequency stream. The decoding module is further configured to respectively decode the low-frequency streams at each of the levels to obtain low-frequency signal features at each of the levels, and to respectively decode the high-frequency streams at each of the levels to obtain high-frequency signal features at each of the levels. Correspondingly, the feature reconstruction module is further configured to respectively perform feature reconstruction on the low-frequency signal features at each of the levels to obtain level low-frequency subband signals at each of the levels, and to respectively perform feature reconstruction on the high-frequency signal features at each of the levels to obtain level high-frequency subband signals at each of the levels, thereby combining the level low-frequency subband signals and the level high-frequency subband signals into a level audio signal at each of the levels. Correspondingly, the audio synthesis module is further configured to add a plurality of level low-frequency subband signals at each of the levels to obtain a low-frequency subband signal, and add a plurality of level high-frequency subband signals at each of the levels to obtain a high-frequency subband signal, and combine the low-frequency subband signal and the high-frequency subband signal to obtain the audio signal.

[0238] In some embodiments, the audio synthesis module is further configured to upsample the low-frequency subband signals to obtain low-pass filtered signals, upsample the high-frequency subband signals to obtain high-frequency filtered signals, and perform filtering synthesis on the low-pass filtered signals and the high-frequency filtered signals to obtain the audio signal.

[0239] In some embodiments, the feature reconstruction module is further configured to perform the following operations on signal features of each of the levels, respectively: perform a first convolution operation on the signal features to obtain convolution features of the level, upsample the convolution features to obtain upsampled features of the level, perform a pooling operation on the upsampled features to obtain pooled features of the level, and perform a second convolution operation on the pooled features to obtain a level audio signal of the level.

[0240] In some embodiments, the upsampling is realized by L cascaded decoding layers, and the feature reconstruction module is further configured to: upsample the pooling feature by a first decoding layer of the L cascaded decoding layers to obtain an upsampling result of the first decoding layer; upsample the first upsampling result of a (k-1)th decoding layer by a k-th decoding layer of the L cascaded decoding layers to obtain an upsampling result of the k-th decoding layer, where L and the k are integers greater than 1 and the k is less than or equal to L; traverse the k to obtain the upsampling result of the L-th decoding layer, and use the upsampling result of the L-th decoding layer as the upsampling feature of the level.

[0241] In some embodiments, the decoding module is further configured to perform the following operations for each of the levels: perform entropy decoding on the stream of the level to obtain quantized values ​​of the stream, and perform inverse quantization on the quantized values ​​of the stream to obtain signal features of the level.

[0242] In the above embodiment of the present application, the streams of multiple levels are decoded respectively to obtain signal features of each level, and feature reconstruction is performed on the signal features of each level to obtain level audio signals of each level, and audio synthesis is performed on the level audio signals of multiple levels to obtain an audio signal. Since the data dimension of the signal features is smaller than the data dimension of the audio signal, the data dimension of the data processed in the audio decoding process is reduced, and the decoding efficiency of the audio signal is improved.

[0243] The present embodiment further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the method provided by the present embodiment.

[0244] An embodiment of the present application further provides a computer-readable storage medium having executable instructions stored therein, which, when executed by a processor, causes the processor to perform a method provided by an embodiment of the present application.

[0245] In some embodiments, the computer-readable storage medium may be a memory such as Read-Only Memory (ROM), Random Access Memory (RAM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory, magnetic surface memory, optical disk, or CD-ROM, or may be any device including one or any combination of the above memories.

[0246] In some embodiments, the executable instructions may take the form of a program, software, software module, script, or code, written according to any type of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed according to any type, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0247] By way of example, the executable instructions may, but do not necessarily, correspond to a file in a file system and may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program under consideration, or in multiple collective files (e.g., files that store one or more modules, subprograms, or code portions).

[0248] As an example, the executable instructions may be executed on one computing device, or may be executed on multiple computing devices located at one location, or may be deployed to be executed on multiple computing devices distributed across multiple locations and interconnected by a communications network.

[0249] The above content is merely an embodiment of the present application and is not used to limit the scope of protection of the present application. All modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. 1. An audio encoding method performed by an electronic device, the method comprising: performing first level feature extraction on the audio signal to obtain the first level signal features; For an i-th level among the N levels, splicing the audio signal and signal features of the (i-1)-th level to obtain spliced ​​features, and performing i-th level feature extraction on the spliced ​​features to obtain signal features of the i-th level, where N and i are integers greater than 1, and i is equal to or less than N; traversing the i and obtaining signal features for each of the N levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; encoding the signal features of the first level and each of the N levels to obtain the audio signal stream at each level; performing first-level feature extraction on the audio signal to obtain the first-level signal features, performing a first convolution process on the audio signal to obtain the first-level convolution features; performing a first pooling process on the convolutional features to obtain the first-level pooled features; performing a first down-sampling on the pooled features to obtain the first-level down-sampled features; performing a second convolution process on the downsampled features to obtain the first level signal features.

2. An audio encoding method performed by an electronic device, the method comprising: performing first level feature extraction on the audio signal to obtain the first level signal features; For an i-th level among the N levels, splicing the audio signal and signal features of the (i-1)-th level to obtain spliced ​​features, and performing i-th level feature extraction on the spliced ​​features to obtain signal features of the i-th level, where N and i are integers greater than 1, and i is equal to or less than N; traversing the i and obtaining signal features for each of the N levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; encoding the signal features of the first level and each of the N levels to obtain the audio signal stream at each level; performing an i-th level feature extraction on the spliced ​​features to obtain the i-th level signal features, performing a third convolution process on the spliced ​​features to obtain the i-th level convolution features; performing a second pooling process on the convolutional features to obtain the i-level pooled features; performing a second down-sampling on the pooled features to obtain the i-level down-sampled features; performing a fourth convolution process on the downsampled features to obtain the i-level signal features.

3. An audio encoding method performed by an electronic device, the method comprising: performing first level feature extraction on the audio signal to obtain the first level signal features; For an i-th level among the N levels, splicing the audio signal and signal features of the (i-1)-th level to obtain spliced ​​features, and performing i-th level feature extraction on the spliced ​​features to obtain signal features of the i-th level, where N and i are integers greater than 1, and i is equal to or less than N; traversing the i and obtaining signal features for each of the N levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; encoding the signal features of the first level and each of the N levels to obtain the audio signal stream at each level; After the step of encoding the signal features of the first level and the signal features of each of the N levels to obtain the stream of the audio signal at each level, the method further comprises: further comprising the step of: setting, for each of the levels, a level transmission priority corresponding to the stream of the audio signal at each of the levels; An audio encoding method, wherein there is a negative correlation between the level transmission priority and the level number of each of the levels, and there is a positive correlation between the level transmission priority and a decoding quality indicator of the stream of each of the levels.

4. An audio encoding method performed by an electronic device, the method comprising: performing first level feature extraction on the audio signal to obtain the first level signal features; For an i-th level among the N levels, splicing the audio signal and signal features of the (i-1)-th level to obtain spliced ​​features, and performing i-th level feature extraction on the spliced ​​features to obtain signal features of the i-th level, where N and i are integers greater than 1, and i is equal to or less than N; traversing the i and obtaining signal features for each of the N levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; encoding the signal features of the first level and each of the N levels to obtain the audio signal stream at each level; The signal features include low-frequency signal features and high-frequency signal features, and the audio signal stream at each level includes a low-frequency stream obtained by encoding based on the low-frequency signal features and a high-frequency stream obtained by encoding based on the high-frequency signal features, and the method includes: and further comprising a step of performing, for each of the levels, a process of setting a first transmission priority to a low-frequency stream of the level and a second transmission priority to a high-frequency stream of the level; an audio encoding method, wherein the first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th level is lower than the first transmission priority of the i-th level, and there is a positive correlation between the transmission priority of the stream and the decoding quality index of the corresponding stream.

5. performing first-level feature extraction on the audio signal to obtain the first-level signal features, performing a subband decomposition on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal; performing a first level of feature extraction on the low frequency subband signals to obtain the first level of low frequency signal features, and performing a first level of feature extraction on the high frequency subband signals to obtain the first level of high frequency signal features; and determining the low frequency signal features and the high frequency signal features as the first level signal features.

6. performing subband decomposition on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal, sampling the audio signal according to a first sampling frequency to obtain a sampled signal; low-pass filtering the sampled signal to obtain a low-pass filtered signal, and down-sampling the low-pass filtered signal to obtain the low-frequency sub-band signal at a second sampling frequency; high-pass filtering the sampled signal to obtain a high-pass filtered signal, and down-sampling the high-pass filtered signal to obtain the high-frequency sub-band signal at a second sampling frequency; The method of claim 5 , wherein the second sampling frequency is less than the first sampling frequency.

7. The step of splicing the audio signal and (i-1)-th level signal features for an i-th level among the N levels to obtain spliced ​​features, and performing i-th level feature extraction on the spliced ​​features to obtain the i-th level signal features includes: splicing a low-frequency subband signal of the audio signal with an (i-1)-th level low-frequency signal feature to obtain a first spliced ​​feature, and performing an i-th level feature extraction on the first spliced ​​feature to obtain the i-th level low-frequency signal feature; splicing a high-frequency subband signal of the audio signal with a high-frequency signal feature of an (i-1)th level to obtain a second spliced ​​feature, and performing an i-th level feature extraction on the second spliced ​​feature to obtain the i-th level high-frequency signal feature; and determining the low-frequency signal features of the i-th level and the high-frequency signal features of the i-th level as the signal features of the i-th level.

8. the first downsampling is achieved by M cascaded coding layers; performing a first downsampling on the pooled features to obtain the first-level downsampled features, performing a first down-sampling on the pooling features by a first coding layer among the M cascaded coding layers to obtain a down-sampling result of the first coding layer; a step of performing a first downsampling on the downsampling results of the (j-1)th coding layers by a jth coding layer among the M cascaded coding layers to obtain a downsampling result of the jth coding layer, where M and j are integers greater than 1, and j is equal to or less than M; traversing the j to obtain a downsampling result of an M-th coding layer, and determining the downsampling result of the M-th coding layer as the first-level downsampling feature.

9. The step of encoding the signal features of the first level and the signal features of each of the N levels to obtain the audio signal stream at each level comprises: performing a quantization process on the signal features of the first level and the signal features of each of the N levels, respectively, to obtain quantized results of the signal features of each level; and performing entropy coding on the quantization results of the signal features at each level to obtain a stream of the audio signal at each level.

10. 1. An audio decoding method performed by an electronic device, the method comprising: receiving a stream corresponding to each of a plurality of levels obtained by encoding an audio signal according to the method of claim 5; decoding the streams of each of the levels to obtain signal features of each of the levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; performing feature reconstruction on the signal features of each of the levels to obtain a level audio signal for each of the levels; and performing audio synthesis on the level audio signals of the plurality of levels to obtain the audio signal.

11. the streams include a low frequency stream and a high frequency stream; The step of decoding the streams of each of the levels and obtaining signal features of each of the levels includes: decoding the low-frequency streams at each of the levels to obtain low-frequency signal features at each of the levels, and decoding the high-frequency streams at each of the levels to obtain high-frequency signal features at each of the levels; The step of performing feature reconstruction on the signal features of each of the levels to obtain a level audio signal for each of the levels includes: performing feature reconstruction on the low-frequency signal features of each of the levels to obtain a level low-frequency sub-band signal of each of the levels, and performing feature reconstruction on the high-frequency signal features of each of the levels to obtain a level high-frequency sub-band signal of each of the levels; and leveling the low-frequency subband signal and the high-frequency subband signal to a level audio signal of the level, The step of performing audio synthesis on the level audio signals of the plurality of levels to obtain the audio signal includes: adding a plurality of said level low frequency subband signals to obtain a low frequency subband signal, and adding a plurality of said level high frequency subband signals to obtain a high frequency subband signal; and combining the low-frequency subband signal and the high-frequency subband signal to obtain the audio signal.

12. The step of combining the low frequency subband signal and the high frequency subband signal to obtain the audio signal includes: upsampling the low frequency subband signal to obtain a low pass filtered signal; upsampling the high frequency subband signal to obtain a high frequency filtered signal; and performing a filtering synthesis on the low-pass filtered signal and the high-frequency filtered signal to obtain the audio signal.

13. The step of performing feature reconstruction on the signal features of each of the levels to obtain a level audio signal for each of the levels includes: For each of the signal features at said level, performing a first convolution process on the signal features to obtain convolution features of the level; upsampling the convolved features to obtain upsampled features for the level; performing a pooling process on the upsampled features to obtain pooled features of the level; and performing a second convolution process on the pooled features to obtain a level audio signal of the level.

14. The upsampling is achieved by L cascaded decoding layers, The process of upsampling the convolved features to obtain upsampled features of the level includes: upsampling the pooling features by a first decoding layer among the L cascaded decoding layers to obtain an upsampling result of the first decoding layer; A step of upsampling a first upsampling result of the (k−1)th decoding layer by a kth decoding layer among the L cascaded decoding layers to obtain an upsampling result of the kth decoding layer, L and k are integers greater than 1, and k is equal to or less than L; traversing the k to obtain an upsampling result of the L-th decoding layer, and setting the upsampling result of the L-th decoding layer as the upsampling feature of the level.

15. The step of decoding the streams of each of the levels and obtaining signal features of each of the levels includes: For each of the above levels, performing entropy decoding on the stream of levels to obtain quantized values ​​of the stream; and performing an inverse quantization process on the quantized values ​​of the stream to obtain signal characteristics of the level.

16. 1. An audio encoding device, comprising: a first feature extraction module configured to perform first level feature extraction on the audio signal to obtain the first level signal features; a second feature extraction module configured to, for an i-th level among N levels, splice the audio signal and signal features of an (i-1)-th level to obtain spliced ​​features, and perform i-th level feature extraction on the spliced ​​features to obtain signal features of the i-th level, wherein N and i are integers greater than 1, and i is less than or equal to N; a traversing module configured to traverse the i to obtain signal features for each of the N levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; an encoding module configured to encode the signal features of the first level and the signal features of each of the N levels to obtain a stream of the audio signal at each level, and further comprising: An audio coding device configured to perform the method according to any one of claims 1 to 4.

17. 1. An audio decoding device, comprising: a receiving module configured to receive streams corresponding to a plurality of levels obtained by encoding an audio signal; a decoding module configured to decode the streams of each of the levels to obtain signal features of each of the levels, wherein a data dimension of the signal features is smaller than a data dimension of the audio signal; a feature reconstruction module configured to perform feature reconstruction on signal features of each of the levels to obtain a level audio signal for each of the levels; an audio synthesis module configured to perform audio synthesis on the level audio signals of the plurality of levels to obtain the audio signal; and An audio decoding device configured to perform the method of claim 10.

18. An electronic device, the electronic device comprising: a memory configured to store executable instructions; a processor configured to implement the method of any one of claims 1 to 4 when executing executable instructions stored in said memory.

19. An electronic device, comprising: a memory configured to store executable instructions; a processor configured to implement the method of claim 10 when executing executable instructions stored in the memory.

20. A computer program which, when executed by a processor, causes the computer program to implement the method according to any one of claims 1 to 4.

21. A computer program that, when executed by a processor, causes the method of claim 10 to be realized.

Images