Audio coding method and device, equipment, storage medium and program product

Abstract

The invention provides an audio coding method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: performing first-level feature extraction processing on an audio signal to obtain a first-level signal feature; for the ith hierarchy in the N hierarchies, splicing processing is carried out on the audio signals and the signal features of the (i-1) th hierarchy to obtain spliced features, feature extraction processing of the ith hierarchy is carried out on the spliced features to obtain the signal features of the ith hierarchy, N and i are integers larger than 1, and i is smaller than or equal to N; the i is traversed, the signal feature of each level in the N levels is obtained, and the data dimension of the signal feature is smaller than the data dimension of the audio signal; respectively encoding the signal feature of the first level and the signal feature of each level in the N levels to obtain a code stream of the audio signal at each level; according to the invention, the audio coding efficiency can be improved and the audio coding quality can be ensured.

Description

[0001] This application is a divisional application of the patent application with application number 202210677636.4, application date June 15, 2022, and invention name “audio encoding method, device, equipment, storage medium and program product”. Technical Field

[0002] The present application relates to the field of audio processing technology, and in particular to an audio encoding method, an audio decoding method, an apparatus, a device, a storage medium and a computer program product. Background Art

[0003] Audio coding technology is a core technology used in communication services including remote audio and video calls. Audio coding technology can be understood as using less network bandwidth resources to transmit as much voice information as possible. Audio coding is a kind of source coding. The purpose of source coding is to compress the amount of data that the user wants to transmit as much as possible at the encoding end, remove the redundancy in the information, and restore it losslessly (or nearly losslessly) at the decoding end.

[0004] However, there is no effective solution in the related technology on how to effectively improve the efficiency of audio encoding while ensuring the quality of audio encoding. Summary of the invention

[0005] The embodiments of the present application provide an audio encoding method, apparatus, device, storage medium and computer program product, which can improve audio encoding efficiency and ensure audio encoding quality.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application provides an audio encoding method, including:

[0008] Performing a first-level feature extraction process on the audio signal to obtain the signal features of the first level;

[0009] For the i-th level among the N levels, the audio signal and the signal features of the (i-1)-th level are concatenated to obtain concatenated features, and

[0010] Performing feature extraction processing of the i-th level on the splicing features to obtain signal features of the i-th level, wherein N and i are integers greater than 1, and i is less than or equal to N;

[0011] Traversing the i to obtain signal features of each level in the N levels, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal;

[0012] The signal features of the first layer and the signal features of each layer in the N layers are respectively encoded to obtain the code stream of the audio signal at each layer.

[0013] The present application also provides an audio decoding method, including:

[0014] Receiving code streams corresponding to multiple layers obtained by encoding the audio signal;

[0015] Decoding the bitstreams of each layer respectively to obtain signal features of each layer, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal;

[0016] Reconstructing the signal features of each of the layers respectively to obtain the hierarchical audio signals of each of the layers;

[0017] Audio synthesis is performed on the hierarchical audio signals of the multiple levels to obtain the audio signal.

[0018] The present application also provides an audio encoding device, including:

[0019] A first feature extraction module, used to perform a first-level feature extraction process on the audio signal to obtain the signal features of the first level;

[0020] a second feature extraction module, configured to perform a splicing process on the audio signal and a signal feature of the (i-1)th level for the i-th level among the N levels to obtain a splicing feature, and perform a feature extraction process of the i-th level on the splicing feature to obtain the signal feature of the i-th level, wherein N and i are integers greater than 1, and i is less than or equal to N;

[0021] A traversal module, used for traversing the i to obtain signal features of each level in the N levels, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal;

[0022] The encoding module is used to encode the signal characteristics of the first layer and the signal characteristics of each layer in the N layers, respectively, to obtain the code stream of the audio signal at each layer.

[0023] In the above scheme, the first feature extraction module is also used to perform sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal; perform a first-level feature extraction processing on the low-frequency sub-band signal to obtain the low-frequency signal features of the first level, and perform a first-level feature extraction processing on the high-frequency sub-band signal to obtain the high-frequency signal features of the first level; and use the low-frequency signal features and the high-frequency signal features as the signal features of the first level.

[0024] In the above scheme, the first feature extraction module is also used to sample the audio signal according to a first sampling frequency to obtain a sampled signal; perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, and down-sample the low-pass filtered signal to obtain the low-frequency sub-band signal of a second sampling frequency; perform high-pass filtering on the sampled signal to obtain a high-pass filtered signal, and down-sample the high-pass filtered signal to obtain the high-frequency sub-band signal of the second sampling frequency; wherein the second sampling frequency is less than the first sampling frequency.

[0025] In the above scheme, the second feature extraction module is also used to perform splicing processing on the low-frequency subband signal of the audio signal and the low-frequency signal feature of the (i-1)th level to obtain a first splicing feature, and perform the feature extraction processing of the i-th level on the first splicing feature to obtain the low-frequency signal feature of the i-th level; perform splicing processing on the high-frequency subband signal of the audio signal and the high-frequency signal feature of the (i-1)th level to obtain a second splicing feature, and perform the feature extraction processing of the i-th level on the second splicing feature to obtain the high-frequency signal feature of the i-th level; use the low-frequency signal feature of the i-th level and the high-frequency signal feature of the i-th level as the signal feature of the i-th level.

[0026] In the above scheme, the first feature extraction module is also used to perform a first convolution processing on the audio signal to obtain the convolution features of the first level; perform a first pooling processing on the convolution features to obtain the pooling features of the first level; perform a first downsampling processing on the pooling features to obtain the downsampling features of the first level; perform a second convolution processing on the downsampling features to obtain the signal features of the first level.

[0027] In the above scheme, the first downsampling process is implemented through M cascaded coding layers, and the first feature extraction module is also used to perform a first downsampling process on the pooling feature through the first coding layer of the M cascaded coding layers to obtain the downsampling result of the first coding layer; through the jth coding layer of the M cascaded coding layers, the downsampling result of the (j-1)th coding layer is subjected to a first downsampling process to obtain the downsampling result of the jth coding layer; wherein, the M and the j are integers greater than 1, and the j is less than or equal to the M; the j is traversed to obtain the downsampling result of the Mth coding layer, and the downsampling result of the Mth coding layer is used as the downsampling feature of the first level.

[0028] In the above scheme, the second feature extraction module is also used to perform a third convolution processing on the splicing features to obtain the convolution features of the i-th level; perform a second pooling processing on the convolution features to obtain the pooling features of the i-th level; perform a second downsampling processing on the pooling features to obtain the downsampling features of the i-th level; perform a fourth convolution processing on the downsampling features to obtain the signal features of the i-th level.

[0029] In the above scheme, the encoding module is also used to perform quantization processing on the signal characteristics of the first layer and the signal characteristics of each layer in the N layers, respectively, to obtain the quantization results of the signal characteristics of each layer; and perform entropy coding processing on the quantization results of the signal characteristics of each layer to obtain the code stream of the audio signal at each layer.

[0030] In the above scheme, the signal characteristics include low-frequency signal characteristics and high-frequency signal characteristics. The encoding module is also used to encode the low-frequency signal characteristics of the first layer and the low-frequency signal characteristics of each layer in the N layers, respectively, to obtain the low-frequency code stream of the audio signal at each layer; encode the high-frequency signal characteristics of the first layer and the high-frequency signal characteristics of each layer in the N layers, respectively, to obtain the high-frequency code stream of the audio signal at each layer; and use the low-frequency code stream and high-frequency code stream of the audio signal at each layer as the code stream of the audio signal at the corresponding layer.

[0031] In the above scheme, the signal characteristics include low-frequency signal characteristics and high-frequency signal characteristics. The encoding module is also used to encode the low-frequency signal characteristics of the first level according to the first encoding rate to obtain the first code stream of the first level, and encode the high-frequency signal characteristics of the first level according to the second encoding rate to obtain the second code stream of the first level; for the signal characteristics of each level in the N levels, the following processing is performed respectively: according to the third encoding rate of the level, the signal characteristics of the level are respectively encoded to obtain the second code stream of each level; the second code stream of the first level and the second code stream of each level in the N levels are used as the code stream of the audio signal at each level; wherein the first encoding rate is greater than the second encoding rate, the second encoding rate is greater than the third encoding rate of any level in the N levels, and the encoding rate of the level is positively correlated with the decoding quality index of the code stream of the corresponding level.

[0032] In the above scheme, the encoding module is also used to perform the following processing for each of the layers: configuring a corresponding layer transmission priority for the code stream of the audio signal at the layer; wherein the layer transmission priority is negatively correlated with the layer number of the layer, and the layer transmission priority is positively correlated with the decoding quality index of the code stream of the corresponding layer.

[0033] In the above scheme, the signal characteristics include low-frequency signal characteristics and high-frequency signal characteristics, and the code stream of the audio signal at each level includes: a low-frequency code stream obtained by encoding based on the low-frequency signal characteristics, and a high-frequency code stream obtained by encoding based on the high-frequency signal characteristics; the encoding module is also used to perform the following processing for each of the levels: configure a first transmission priority for the low-frequency code stream of the level, and configure a second transmission priority for the high-frequency code stream of the level; wherein the first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th level is lower than the first transmission priority of the i-th level, and the transmission priority of the code stream is positively correlated with the decoding quality index of the corresponding code stream.

[0034] The present application also provides an audio decoding device, including:

[0035] A receiving module, used for receiving code streams corresponding to multiple layers obtained by encoding the audio signal;

[0036] A decoding module, used for respectively decoding the bit streams of each layer to obtain signal features of each layer, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal;

[0037] A feature reconstruction module, used to reconstruct the signal features of each level respectively to obtain the hierarchical audio signals of each level;

[0038] The audio synthesis module is used to perform audio synthesis on the hierarchical audio signals of the multiple levels to obtain the audio signal.

[0039] In the above scheme, the code stream includes a low-frequency code stream and a high-frequency code stream, and the decoding module is also used to decode the low-frequency code streams of each level respectively to obtain the low-frequency signal features of each level, and decode the high-frequency code streams of each level respectively to obtain the high-frequency signal features of each level; correspondingly, the feature reconstruction module is also used to reconstruct the low-frequency signal features of each level respectively to obtain the level low-frequency sub-band signals of each level, and reconstruct the high-frequency signal features of each level respectively to obtain the level high-frequency sub-band signals of each level; the level low-frequency sub-band signals and the level high-frequency sub-band signals are used as the level audio signals of the level; correspondingly, the audio synthesis module is also used to add the level low-frequency sub-band signals of multiple levels to obtain the low-frequency sub-band signals, and add the level high-frequency sub-band signals of multiple levels to obtain the high-frequency sub-band signals; the low-frequency sub-band signals and the high-frequency sub-band signals are synthesized to obtain the audio signal.

[0040] In the above scheme, the audio synthesis module is also used to up-sample the low-frequency sub-band signal to obtain a low-pass filtered signal; up-sample the high-frequency sub-band signal to obtain a high-frequency filtered signal; and perform filter synthesis processing on the low-pass filtered signal and the high-frequency filtered signal to obtain the audio signal.

[0041] In the above scheme, the feature reconstruction module is also used to perform the following processing on the signal features of each level: perform a first convolution processing on the signal features to obtain the convolution features of the level; perform upsampling processing on the convolution features to obtain the upsampled features of the level; perform pooling processing on the upsampled features to obtain the pooled features of the level; perform a second convolution processing on the pooled features to obtain the level audio signal of the level.

[0042] In the above scheme, the upsampling process is implemented through L cascaded decoding layers, and the feature reconstruction module is also used to upsample the pooled features through the first decoding layer of the L cascaded decoding layers to obtain the upsampling result of the first decoding layer; through the kth decoding layer of the L cascaded decoding layers, the first upsampling result of the (k-1)th decoding layer is upsampled to obtain the upsampling result of the kth decoding layer; wherein, L and k are integers greater than 1, and k is less than or equal to L; traverse the k to obtain the upsampling result of the Lth decoding layer, and use the upsampling result of the Lth decoding layer as the upsampling feature of the level.

[0043] In the above scheme, the decoding module is also used to perform the following processing for each of the layers: entropy decoding the code stream of the layer to obtain the quantization value of the code stream; inverse quantization processing is performed on the quantization value of the code stream to obtain the signal characteristics of the layer.

[0044] The present application also provides an electronic device, including:

[0045] A memory for storing executable instructions;

[0046] The processor is used to implement the method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0047] The embodiment of the present application also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implements the method provided in the embodiment of the present application.

[0048] An embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method provided in the embodiment of the present application.

[0049] The embodiments of the present application have the following beneficial effects:

[0050] Hierarchical coding of audio signals is implemented: first, the audio signal is subjected to a first-level feature extraction process to obtain the signal features of the first level; then, for the i-th (i is an integer greater than 1, i is less than or equal to N) level among N (N is an integer greater than 1) levels, the audio signal and the signal features of the (i-1)-th level are concatenated to obtain concatenated features, and the i-th level feature extraction process is performed on the concatenated features to obtain the signal features of the i-th level; then, the signal features of each level among the N levels are obtained by traversing i; finally, the signal features of the first level and the signal features of each level among the N levels are encoded respectively to obtain the code streams of the audio signal at each level.

[0051] First, the data dimension of the extracted signal features is smaller than that of the audio signal. This reduces the data dimension of the data processed during the audio encoding process and improves the encoding efficiency of the audio signal.

[0052] Second, when extracting the signal features of the audio signal in layers, the output of each layer is used as the input of the next layer, so that each layer combines the signal features extracted by the previous layer to extract more accurate features of the audio signal. As the number of layers increases, the information loss of the audio signal in the feature extraction process can be minimized. In this way, the multiple code streams obtained by encoding the signal features extracted in this way contain audio signal information that is closer to the original audio signal, reducing the information loss of the audio signal in the encoding process and ensuring the encoding quality of the audio encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a schematic diagram of the architecture of an audio coding system 100 provided in an embodiment of the present application;

[0054] Figure 2 is a structural diagram of an electronic device 500 for implementing an audio encoding method provided in an embodiment of the present application;

[0055] Figure 3 It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0056] Figure 4 It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0057] Figure 5 It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0058] Figure 6 It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0059] Figure 7 It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0060] Figure 8 It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0061] Fig. 9 It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0062] Fig.10 It is a flowchart of the audio decoding method provided in an embodiment of the present application;

[0063] Fig.11 It is a flowchart of the audio decoding method provided in an embodiment of the present application;

[0064] Fig.12 is a schematic diagram of spectrum comparison at different bit rates provided in an embodiment of the present application;

[0065] Fig.13 It is a flowchart of audio encoding and audio decoding provided by an embodiment of the present application;

[0066] Fig.14 is a schematic diagram of a voice communication link provided in an embodiment of the present application;

[0067] Fig.15 is a schematic diagram of a filter bank provided in an embodiment of the present application;

[0068] Fig.16A is a schematic diagram of a common convolutional network provided in an embodiment of the present application;

[0069] Fig. 16B is a schematic diagram of a dilated convolutional network provided in an embodiment of the present application;

[0070] Fig.17 It is a structural diagram of the first layer low-frequency analysis neural network model provided in the embodiment of the present application;

[0071] Fig.18 It is a structural diagram of the second-layer low-frequency analysis neural network model provided in an embodiment of the present application;

[0072] Fig.19 is a model schematic diagram of the first layer low-frequency synthesis neural network model provided in an embodiment of the present application;

[0073] Fig. 20 It is a structural diagram of the second-layer low-frequency synthesis neural network model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0075] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0076] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0078] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0079] 1) Client: an application running in a terminal to provide various services, such as an instant messaging client or an audio playback client.

[0080] 2) Audio Coding: an application of data compression on digital audio signals containing speech.

[0081] 3) Quadrature Mirror Filters (QMF): The QMF filter bank is used to decompose the sub-band signal into multiple signals, thereby reducing the signal bandwidth. The decomposed signals are filtered through their respective channels.

[0082] 4) Quantization refers to the process of approximating the continuous value of a signal (or a large number of possible discrete values) to a finite number of (or fewer) discrete values, including vector quantization, scalar quantization, etc.

[0083] 5) Vector quantization: combine several scalar data into a vector, divide the vector space into several small areas, and find a representative vector for each small area. When quantizing, the vector that falls into the small area is replaced by the corresponding representative vector, that is, it is quantized to the representative vector.

[0084] 6) Scalar quantization: divide the entire dynamic range into several small intervals, each of which has a representative value. During quantization, the signal value that falls into the small interval is replaced by the corresponding representative value, that is, the signal value is quantized to the representative value.

[0085] 7) Entropy coding, that is, coding in which no information is lost according to the entropy principle during the coding process. Information entropy is the average amount of information in the source. Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0086] 8) Neural Network (NN): It is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.

[0087] 9) Deep Learning (DL): It is a new research direction in the field of Machine Learning (ML). Deep learning is to learn the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0088] The embodiments of the present application provide an audio encoding method, an audio decoding method, an apparatus, a device, a storage medium and a computer program product, which can improve the audio encoding efficiency and ensure the audio encoding quality.

[0089] The following describes the implementation scenario of the audio encoding method provided by the embodiment of the present application. Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the audio coding system 100 provided in an embodiment of the present application. To support an exemplary application, a terminal (exemplarily showing terminal 400-1 and terminal 400-2) is connected to server 200 via network 300. Network 300 can be a wide area network or a local area network, or a combination of the two, using a wireless or wired link to achieve data transmission. Among them, terminal 400-1 is the sender of the audio signal, and terminal 400-2 is the receiver of the audio signal.

[0090] In the process of terminal 400-1 sending an audio signal to terminal 400-2 (such as the process of terminal 400-1 and terminal 400-2 making a remote call based on the set client), terminal 400-1 is used to perform a first-level feature extraction process on the audio signal to obtain a signal feature of the first level; for the i-th level among the N levels, the audio signal and the signal feature of the (i-1)-th level are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is subjected to an i-th level feature extraction process to obtain a signal feature of the i-th level, where N and i are integers greater than 1, and i is less than or equal to N; i is traversed to obtain a signal feature of each level in the N levels, and the data dimension of the signal feature is smaller than the data dimension of the audio signal; the signal feature of the first level and the signal feature of each level in the N levels are respectively encoded to obtain a code stream of the audio signal at each level; and the code stream of the audio signal at each level is sent to the server 200;

[0091] The server 200 is used to receive the code streams corresponding to the multiple layers obtained by encoding the audio signal by the terminal 400-1; and send the code streams corresponding to the multiple layers to the terminal 400-2;

[0092] Terminal 400-2 is used to receive the code streams corresponding to multiple layers obtained by encoding the audio signal sent by the server 200; decode the code streams of each layer respectively to obtain signal characteristics of each layer, and the data dimension of the signal characteristics is smaller than the data dimension of the audio signal; reconstruct the signal characteristics of each layer respectively to obtain hierarchical audio signals of each layer; perform audio synthesis on the hierarchical audio signals of multiple layers to obtain an audio signal.

[0093] In some embodiments, the audio encoding method provided in the embodiments of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal alone, it can be implemented by a server alone, or it can be implemented by a terminal and a server in collaboration. For example, the terminal alone executes the audio encoding method provided in the embodiments of the present application, or the terminal sends a coding request for an audio signal to the server, and the server executes the audio encoding method provided in the embodiments of the present application according to the received coding request. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0094] In some embodiments, the electronic device for implementing audio coding provided in the embodiments of the present application may be various types of terminal devices or servers. Among them, the server (e.g., server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. The terminal (e.g., terminal 400) may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device (e.g., an intelligent speaker), an intelligent home appliance (e.g., a smart TV), a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application do not limit this.

[0095] In some embodiments, the audio encoding method provided in the embodiments of the present application can be implemented with the aid of cloud technology (Cloud Technology), which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks in a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the application of cloud computing business model, which can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources. As an example, the above-mentioned server (such as server 200) can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0096] In some embodiments, the terminal or server can implement the audio encoding method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0097] In some embodiments, multiple servers may form a blockchain, and the servers are nodes on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the audio encoding method provided in the embodiment of the present application (such as the code stream of the audio signal at each level, the neural network model for feature extraction, etc.) may be stored on the blockchain.

[0098] The following describes an electronic device for implementing an audio encoding method provided by an embodiment of the present application. Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 for implementing an audio coding method provided in an embodiment of the present application. Figure 1 Taking the terminal shown (such as terminal 400-1) as an example, the electronic device 500 for implementing the audio encoding method provided in the embodiment of the present application includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 540 .

[0099] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0100] The memory 550 may be removable, non-removable, or a combination thereof. The memory 550 may optionally include one or more storage devices physically located away from the processor 510. The memory 550 may include a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0101] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0102] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0103] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0104] In some embodiments, the audio encoding device provided in the embodiments of the present application can be implemented in software. Figure 2 An audio encoding device 553 stored in a memory 550 is shown, which may be software in the form of a program or plug-in, including the following software modules: a first feature extraction module 5531, a second feature extraction module 5532, a traversal module 5533, and an encoding module 5534. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0105] The following describes the audio encoding method provided by the embodiment of the present application. In some embodiments, the audio encoding method provided by the embodiment of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal alone, it can also be implemented by a server alone, or it can be implemented by a terminal and a server in collaboration. Figure 3 , Figure 3 : is a flowchart of an audio encoding method provided in an embodiment of the present application. The audio encoding method provided in an embodiment of the present application includes:

[0106] Step 101: The terminal performs first-level feature extraction processing on the audio signal to obtain first-level signal features.

[0107] In practical applications, the audio signal can be a voice signal during a call (such as an Internet call or a telephone call), a voice message (such as a voice message sent in an instant messaging client), music played, audio, and the like. The audio signal needs to be encoded during transmission, so that the transmitting end of the audio signal can transmit the encoded code stream, and the receiving end of the code stream can decode the received code stream to obtain the audio signal. Next, the encoding process of the audio signal is described. In an embodiment of the present application, a hierarchical coding method is used to encode the audio signal. The hierarchical coding method is implemented by encoding the audio signal at multiple levels. The encoding process of each level is described below. First, for the first level, the terminal can perform a first-level feature extraction process on the audio signal to obtain the signal features of the audio signal extracted by the first level, that is, the signal features of the first level.

[0108] In some embodiments, the audio signal includes a low-frequency sub-band signal and a high-frequency sub-band signal. When the audio signal is processed (such as feature extraction processing and encoding processing), the low-frequency sub-band signal and the high-frequency sub-band signal included in the audio signal can be processed separately. Based on this, see Figure 4 , Figure 4 is a flowchart of an audio encoding method provided in an embodiment of the present application, Figure 4 Show Figure 3 Step 101 can be implemented through steps 201-203: Step 201, performing sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal; Step 202, performing a first-level feature extraction processing on the low-frequency sub-band signal to obtain a first-level low-frequency signal feature, and performing a first-level feature extraction processing on the high-frequency sub-band signal to obtain a first-level high-frequency signal feature; Step 203, using the low-frequency signal feature and the high-frequency signal feature as the first-level signal feature.

[0109] It should be noted that in step 201, during the process of extracting features from the audio signal at the first level, the terminal may first perform sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal, thereby performing feature extraction on the low-frequency sub-band signal and the high-frequency sub-band signal respectively. Figure 5 , Figure 5 is a flowchart of an audio encoding method provided in an embodiment of the present application, Figure 5 Show Figure 4Step 201 can be implemented through steps 2011-2013: step 2011, sampling the audio signal according to the first sampling frequency to obtain a sampling signal; step 2012, low-pass filtering the sampling signal to obtain a low-pass filtered signal, and down-sampling the low-pass filtered signal to obtain a low-frequency sub-band signal of a second sampling frequency; step 2013, high-pass filtering the sampling signal to obtain a high-pass filtered signal, and down-sampling the high-pass filtered signal to obtain a high-frequency sub-band signal of the second sampling frequency. The second sampling frequency is less than the first sampling frequency.

[0110] In step 2011, the audio signal may be sampled at a first sampling frequency to obtain a sampled signal, and the first sampling frequency may be preset. In practical applications, the audio signal is a continuous analog signal, and the audio signal is sampled at the first sampling frequency to obtain a discrete digital signal, i.e., a sampled signal, which includes a plurality of sample points (i.e., sampling values) sampled from the audio signal.

[0111] In step 2012, the adopted signal is subjected to low-pass filtering to obtain a low-pass filtered signal, and the low-pass filtered signal is subjected to down-sampling to obtain a low-frequency sub-band signal of a second sampling frequency. In step 2013, the sampled signal is subjected to high-pass filtering to obtain a high-pass filtered signal, and the high-pass filtered signal is subjected to down-sampling to obtain a high-frequency sub-band signal of a second sampling frequency. In steps 202 and 203, the low-pass filtering and high-pass filtering can be implemented by a QMF analysis filter. In actual implementation, the second sampling frequency can be half of the first sampling frequency, so that a low-frequency sub-band signal and a high-frequency sub-band signal with the same frequency can be obtained.

[0112] In step 202, after obtaining the low-frequency subband signal and the high-frequency subband signal of the audio signal, the low-frequency subband signal of the audio signal is subjected to a first-level feature extraction process to obtain a first-level low-frequency signal feature, and the high-frequency subband signal is subjected to a first-level feature extraction process to obtain a first-level high-frequency signal feature. In step 203, the low-frequency signal feature and the high-frequency signal feature are used as the signal feature of the first level.

[0113] In some embodiments, see Figure 6 , Figure 6 is a flowchart of an audio encoding method provided in an embodiment of the present application, Figure 6 Show Figure 3Step 101 can also be implemented through steps 301-304: step 301, performing a first convolution process on the audio signal to obtain a first-level convolution feature; step 302, performing a first pooling process on the convolution feature to obtain a first-level pooling feature; step 303, performing a first downsampling process on the pooling feature to obtain a first-level downsampling feature; step 304, performing a second convolution process on the downsampling feature to obtain a first-level signal feature.

[0114] It should be noted that, in step 301, the audio signal may be subjected to a first convolution process. In practical applications, the first convolution process may be performed by calling a causal convolution of a preset number of channels (eg, 24 channels) to obtain a first-level convolution feature.

[0115] In step 302, a first pooling process is performed on the convolutional features obtained in step 301. In practical applications, the first pooling process may pre-set a pooling factor (such as 2), and then the convolutional features are first pooled based on the pooling factor to obtain a first-level pooling feature.

[0116] In step 303, the pooled features obtained in step 302 are subjected to a first downsampling process. In practical applications, a downsampling factor may be pre-set, and then the downsampling process is performed based on the downsampling factor. The first downsampling process may be implemented by one coding layer or by multiple coding layers. In some embodiments, the first downsampling process is implemented by M cascaded coding layers. Figure 7 , Figure 7 is a flowchart of an audio encoding method provided in an embodiment of the present application, Figure 7 Show Figure 6 Step 303 can also be implemented through steps 3031-3033: Step 3031, through the first coding layer of M cascaded coding layers, the pooling feature is first down-sampled to obtain the down-sampling result of the first coding layer; Step 3032, through the j-th coding layer of M cascaded coding layers, the down-sampling result of the (j-1)-th coding layer is first down-sampled to obtain the down-sampling result of the j-th coding layer; wherein M and j are integers greater than 1, and j is less than or equal to M; Step 3033, j is traversed to obtain the down-sampling result of the M-th coding layer, and the down-sampling result of the M-th coding layer is used as the down-sampling feature of the first level.

[0117] It should be noted that, in step 3031 to step 3033, the downsampling factor of each coding layer may be the same or different. In practical applications, the downsampling factor is equivalent to the pooling factor, which plays the role of downsampling.

[0118] In step 304, a second convolution process may be performed on the downsampled features. In practical applications, the second convolution process may be performed by calling a causal convolution of a preset number of channels to obtain the signal features of the first level.

[0119] In practical applications, Figure 6 Steps 301 to 304 shown can be implemented by calling a first neural network model, which includes a first convolution layer, a pooling layer, a downsampling processing layer, and a second convolution layer. In this way, the first convolution layer can be called to perform a first convolution process on the audio signal to obtain a first-level convolution feature; the pooling layer can be called to perform a first pooling process on the convolution feature to obtain a first-level pooling feature; the downsampling processing layer can be called to perform a first downsampling process on the pooling feature to obtain a first-level downsampling feature; the second convolution layer can be called to perform a second convolution process on the downsampling feature to obtain a first-level signal feature.

[0120] It should be noted that when performing the first-level feature extraction on the audio signal, Figure 6 Steps 301 to 304 are shown, and the low-frequency sub-band signal and the high-frequency sub-band signal of the audio signal are respectively subjected to the first-level feature extraction processing (i.e. Figure 4 Step 202 shown). That is, a first convolution process is performed on the low-frequency subband signal of the audio signal to obtain a first convolution feature of a first level; a first pooling process is performed on the first convolution feature to obtain a first pooling feature of a first level; a first downsampling process is performed on the first pooling feature to obtain a first downsampling feature of a first level; a second convolution process is performed on the first downsampling feature to obtain a low-frequency signal feature of a first level. A first convolution process is performed on the high-frequency subband signal of the audio signal to obtain a second convolution feature of a first level; a first pooling process is performed on the second convolution feature to obtain a second pooling feature of a first level; a first downsampling process is performed on the second pooling feature to obtain a second downsampling feature of a first level; a second convolution process is performed on the second downsampling feature to obtain a high-frequency signal feature of a first level.

[0121] Step 102: For the i-th level among the N levels, perform concatenation processing on the audio signal and the signal features of the (i-1)-th level to obtain concatenated features, and perform feature extraction processing of the i-th level on the concatenated features to obtain signal features of the i-th level.

[0122] Wherein, N and i are integers greater than 1, and i is less than or equal to N.

[0123] After performing the first-level feature extraction process on the audio signal, the audio signal can also be subjected to feature extraction processes of the remaining levels. In an embodiment of the present application, the remaining levels include N levels, and for the i-th level among the N levels, the audio signal and the signal features of the (i-1)-th level are spliced ​​to obtain spliced ​​features, and the spliced ​​features are subjected to feature extraction processes of the i-th level to obtain signal features of the i-th level. For example, for the second level, the audio signal and the signal features of the first level are spliced ​​to obtain spliced ​​features, and the spliced ​​features are subjected to feature extraction processes of the second level to obtain signal features of the second level; for the third level, the audio signal and the signal features of the second level are spliced ​​to obtain spliced ​​features, and the spliced ​​features are subjected to feature extraction processes of the third level to obtain signal features of the third level; for the fourth level, the audio signal and the signal features of the third level are spliced ​​to obtain spliced ​​features, and the spliced ​​features are subjected to feature extraction processes of the fourth level to obtain signal features of the fourth level, and so on.

[0124] In some embodiments, the audio signal includes a low-frequency sub-band signal and a high-frequency sub-band signal. When the audio signal is processed (such as feature extraction processing and encoding processing), the low-frequency sub-band signal and the high-frequency sub-band signal included in the audio signal can be processed separately. Based on this, for the i-th level among the N levels, the audio signal can also be subjected to sub-band decomposition processing to obtain the low-frequency sub-band signal and the high-frequency sub-band signal of the audio signal. The process of sub-band decomposition processing can refer to the above steps 2011-2013. In this way, for the i-th level among the N levels, the data output by the feature extraction processing includes: the low-frequency signal features of the i-th level, and the high-frequency signal features of the i-th level.

[0125] Accordingly, see Figure 8 , Figure 8 is a flowchart of an audio encoding method provided in an embodiment of the present application, Figure 8 Show Figure 3 Step 102 can be implemented through steps 401-403: Step 401, the low-frequency subband signal of the audio signal and the low-frequency signal feature of the (i-1)th level are spliced ​​to obtain a first spliced ​​feature, and the first spliced ​​feature is subjected to the feature extraction process of the i-th level to obtain the low-frequency signal feature of the i-th level; Step 402, the high-frequency subband signal of the audio signal and the high-frequency signal feature of the (i-1)th level are spliced ​​to obtain a second spliced ​​feature, and the second spliced ​​feature is subjected to the feature extraction process of the i-th level to obtain the high-frequency signal feature of the i-th level; Step 403, the low-frequency signal feature of the i-th level and the high-frequency signal feature of the i-th level are used as the signal feature of the i-th level.

[0126] It should be noted that, in step 401, after obtaining the low-frequency subband signal and the high-frequency subband signal of the audio signal, the low-frequency subband signal of the audio signal and the low-frequency signal features extracted at the (i-1)th level are spliced ​​to obtain a first spliced ​​feature, and then the first spliced ​​feature is subjected to the feature extraction process of the i-th level to obtain the low-frequency signal features of the i-th level. Similarly, in step 402, the high-frequency subband signal of the audio signal and the high-frequency signal features extracted at the (i-1)th level are spliced ​​to obtain a second spliced ​​feature, and then the second spliced ​​feature is subjected to the feature extraction process of the i-th level to obtain the high-frequency signal features of the i-th level. Thus, in step 403, the low-frequency signal features of the i-th level and the high-frequency signal features of the i-th level are used as the signal features of the i-th level.

[0127] In some embodiments, see Fig. 9 , Fig. 9 is a flowchart of an audio encoding method provided in an embodiment of the present application, Fig. 9 Show Figure 3 Step 102 can also be implemented through steps 501-504: step 501, performing a third convolution process on the splicing features to obtain the convolution features of the i-th level; step 502, performing a second pooling process on the convolution features to obtain the pooling features of the i-th level; step 503, performing a second downsampling process on the pooling features to obtain the downsampled features of the i-th level; step 504, performing a fourth convolution process on the downsampled features to obtain the signal features of the i-th level.

[0128] It should be noted that, in step 501, the concatenated feature (obtained by concatenating the audio signal and the signal feature of the (i-1)th level) may be subjected to a third convolution process. In practical applications, the third convolution process may be performed by calling a causal convolution of a preset number of channels to obtain a convolution feature of the i-th level.

[0129] In step 502, a second pooling process is performed on the convolutional features obtained in step 501. In practical applications, the second pooling process may pre-set a pooling factor, and then the second pooling process is performed on the convolutional features based on the pooling factor to obtain the pooling features of the i-th level.

[0130] In step 503, the pooled features obtained in step 502 are subjected to a second downsampling process. In practical applications, a downsampling factor may be pre-set, and the downsampling process is performed based on the downsampling factor. The second downsampling process may be implemented through one coding layer or through multiple coding layers. In some embodiments, the second downsampling process may be implemented through X cascaded coding layers. Accordingly, Fig. 9Step 503 may also be implemented through steps 5031-5033: Step 5031, performing a second downsampling process on the pooling feature through the first coding layer of the X cascaded coding layers to obtain a downsampling result of the first coding layer; Step 5032, performing a second downsampling process on the downsampling result of the (g-1)th coding layer through the g-th coding layer of the X cascaded coding layers to obtain a downsampling result of the g-th coding layer; wherein X and g are integers greater than 1, and g is less than or equal to X; Step 5033, traversing g to obtain a downsampling result of the X-th coding layer, and using the downsampling result of the X-th coding layer as the downsampling feature of the i-th level.

[0131] It should be noted that, in step 5031 to step 5033, the downsampling factor of each coding layer may be the same or different. In practical applications, the downsampling factor is equivalent to the pooling factor, which plays the role of downsampling.

[0132] In step 504, a fourth convolution process may be performed on the downsampled features. In practical applications, the fourth convolution process may be performed by calling a causal convolution of a preset number of channels to obtain the signal features of the i-th level.

[0133] In practical applications, Fig. 9 Steps 501 to 504 shown can be implemented by calling a second neural network model, which includes a third convolution layer, a pooling layer, a downsampling processing layer, and a fourth convolution layer. In this way, the third convolution layer can be called to perform a third convolution process on the splicing to obtain the convolution features of the i-th level; the pooling layer can be called to perform a second pooling process on the convolution features to obtain the pooling features of the i-th level; the downsampling processing layer can be called to perform a second downsampling process on the pooling features to obtain the downsampling features of the i-th level; the fourth convolution layer can be called to perform a fourth convolution process on the downsampling features to obtain the signal features of the i-th level. In actual implementation, the feature dimension of the signal features output by the second neural network can be less than the feature dimension of the signal features input by the first neural network.

[0134] It should be noted that when extracting features at the i-th level, Fig. 9In the steps 501 to 504 shown, the low-frequency subband signal and the high-frequency subband signal of the audio signal are subjected to feature extraction processing of the i-th level respectively. That is, for the i-th level, the low-frequency splicing feature (obtained by splicing the low-frequency subband signal and the low-frequency signal feature of the (i-1)th level) is subjected to the third convolution processing to obtain the convolution feature of the i-th level, the convolution feature is subjected to the second pooling processing to obtain the pooling feature of the i-th level; the pooling feature is subjected to the second downsampling processing to obtain the downsampled feature of the i-th level; the downsampled feature is subjected to the fourth convolution processing to obtain the low-frequency signal feature of the i-th level. For the i-th level, the high-frequency splicing features (obtained by splicing the high-frequency sub-band signal and the high-frequency signal features of the (i-1)th level) are subjected to a third convolution process to obtain the convolution features of the i-th level; the convolution features are subjected to a second pooling process to obtain the pooling features of the i-th level; the pooling features are subjected to a second downsampling process to obtain the downsampling features of the i-th level; the downsampling features are subjected to a fourth convolution process to obtain the high-frequency signal features of the i-th level.

[0135] Step 103: traverse i to obtain the signal features of each level in the N levels.

[0136] Among them, the data dimension of the signal feature is smaller than the data dimension of the audio signal.

[0137] In step 102, the feature extraction process for the i-th level is described. In practical applications, i needs to be traversed to obtain the signal features of each level in the N levels. In the embodiment of the present application, the data dimension of the signal features output by each level is smaller than the data dimension of the audio signal. In this way, the data dimension of the data involved in the audio encoding process can be reduced, and the encoding efficiency of the audio encoding can be improved.

[0138] Step 104: Encode the signal features of the first layer and the signal features of each layer in the N layers respectively to obtain the code stream of the audio signal at each layer.

[0139] In practical applications, after obtaining the signal features of the first level and the signal features of each level in the N levels, the signal features of the first level and the signal features of each level in the N levels can be encoded respectively to obtain the code stream of the audio signal at each level. The code stream can be transmitted to the receiving end of the audio signal, so that the receiving end can decode the audio signal as a decoding end.

[0140] It should be noted that the signal features output by the i-th level among the N levels can be understood as the residual signal features between the signal features output by the (i-1)-th level and the original audio signal. In this way, the signal features of the extracted audio signal include both the signal features of the audio signal extracted by the first level and the residual signal features extracted by each level among the N levels, so that the signal features of the extracted audio signal are more comprehensive and accurate, reducing the information loss of the audio signal during the feature extraction process. Therefore, when the signal features of the first level and the signal features of each level among the N levels are encoded separately, the quality of the encoded code stream is higher, the information of the audio signal contained therein is closer to the original audio signal, and the encoding quality of the audio encoding is improved.

[0141] In some embodiments, Figure 3 The shown step 104 can be implemented through steps 104a1-104a2: step 104a1, quantizing the signal characteristics of the first level and the signal characteristics of each level in the N levels, respectively, to obtain the quantization results of the signal characteristics of each level; step 104a2, entropy coding the quantization results of the signal characteristics of each level to obtain the code stream of the audio signal at each level.

[0142] It should be noted that, in step 104a1, a quantization table may be pre-set, and the quantization table includes a correspondence between signal features and quantization values. When performing quantization processing, the preset quantization table may be queried for the signal features of the first level and the signal features of each level in the N levels, and the corresponding quantization values ​​may be queried respectively, so that the quantization values ​​obtained by the query are used as quantization results. In step 104a2, the quantization results of the signal features of each level are entropy coded respectively to obtain the code stream of the audio signal at each level.

[0143] In practical applications, the audio signal includes a low-frequency sub-band signal and a high-frequency sub-band signal, and accordingly, the signal features output by each level include low-frequency signal features and high-frequency signal features. Based on this, when the signal features include low-frequency signal features and high-frequency signal features, in some embodiments, Figure 3 The shown step 104 can also be implemented through steps 104b1-104b3: Step 104b1, encode the low-frequency signal features of the first level and the low-frequency signal features of each level in the N levels, respectively, to obtain the low-frequency code stream of the audio signal at each level; Step 104b2, encode the high-frequency signal features of the first level and the high-frequency signal features of each level in the N levels, respectively, to obtain the high-frequency code stream of the audio signal at each level; Step 104b3, use the low-frequency code stream and the high-frequency code stream of the audio signal at each level as the code stream of the audio signal at the corresponding level.

[0144] It should be noted that the encoding process of the low-frequency signal features in step 104b1 can also be implemented by steps similar to steps 104a1-104a2, that is, the low-frequency signal features of the first level and the low-frequency signal features of each level in the N levels are quantized respectively to obtain the quantization results of the low-frequency signal features of each level; the quantization results of the low-frequency signal features of each level are entropy encoded to obtain the low-frequency code stream of the audio signal at each level. The encoding process of the high-frequency signal features in step 104b2 can also be implemented by steps similar to steps 104a1-104a2, that is, the high-frequency signal features of the first level and the high-frequency signal features of each level in the N levels are quantized respectively to obtain the quantization results of the high-frequency signal features of each level; the quantization results of the high-frequency signal features of each level are entropy encoded to obtain the high-frequency code stream of the audio signal at each level.

[0145] In practical applications, the audio signal includes a low-frequency sub-band signal and a high-frequency sub-band signal, and accordingly, the signal features output by each level include low-frequency signal features and high-frequency signal features. Based on this, when the signal features include low-frequency signal features and high-frequency signal features, in some embodiments, Figure 3 The shown step 104 can also be implemented through steps 104c1-104c3: Step 104c1, encode the low-frequency signal characteristics of the first level according to the first encoding rate to obtain the first code stream of the first level, and encode the high-frequency signal characteristics of the first level according to the second encoding rate to obtain the second code stream of the first level; Step 104c2, for the signal characteristics of each level in the N levels, perform the following processing respectively: encode the signal characteristics of the level according to the third encoding rate of the level to obtain the second code stream of each level; Step 104c3, use the second code stream of the first level and the second code stream of each level in the N levels as the code stream of the audio signal at each level.

[0146] It should be noted that the first encoding rate is greater than the second encoding rate, the second encoding rate is greater than the third encoding rate of any level in the N levels, and the encoding rate of the level is positively correlated with the decoding quality index of the code stream of the corresponding level. In step 104c2, the corresponding third encoding rate can be set for each level in the N levels. The third encoding rate of each level in the N levels can be the same, or partly the same and partly different, or completely different. Here, the encoding bit rate of the layer is positively correlated with the decoding quality index of the code stream of the corresponding layer, that is, the higher the encoding bit rate, the higher the decoding quality index (value) of the obtained code stream, and the audio signal features contained in the low-frequency signal features of the first layer are the most. Therefore, the first encoding bit rate used for the low-frequency signal features of the first layer is the largest to ensure the encoding effect of the audio signal; at the same time, for the high-frequency signal features of the first layer, a second encoding bit rate lower than the first encoding bit rate is used for encoding, and for the signal features of each layer in the N layers, a third encoding bit rate lower than the second encoding bit rate is used for encoding. While adding more features of the audio signal (including high-frequency signal features and residual signal features), the encoding efficiency of the audio signal is improved by reasonably allocating the encoding bit rate of each layer.

[0147] In some embodiments, after obtaining the code stream of the audio signal at each layer, the terminal can also perform the following processing for each layer: configure the corresponding layer transmission priority for the code stream of the audio signal at the layer; wherein the layer transmission priority is negatively correlated with the number of layers of the layer, and the layer transmission priority is positively correlated with the decoding quality index of the code stream of the corresponding layer.

[0148] It should be noted that the layer transmission priority of the layer is used to characterize the transmission priority of the code stream of the layer. The layer transmission priority is negatively correlated with the layer number of the layer, that is, the larger the layer number, the lower the corresponding layer transmission priority. For example, the layer transmission priority of the first layer (layer number is 1) is higher than the layer transmission priority of the second layer (layer number is 2). Based on this, when the code streams of each layer are transmitted to the decoding end, the code streams of the corresponding layers can be transmitted according to the configured layer transmission priority. In actual applications, when the code streams of audio signals at multiple layers are transmitted to the decoding end, the code streams of some layers can be transmitted, or the code streams of all layers can be transmitted. When the code streams of some layers are transmitted, the code streams of the corresponding layers can be transmitted according to the configured layer transmission priority.

[0149] In some embodiments, the signal characteristics include low-frequency signal characteristics and high-frequency signal characteristics, and the code stream of the audio signal at each layer includes: a low-frequency code stream obtained based on the low-frequency signal characteristic encoding, and a high-frequency code stream obtained based on the high-frequency signal characteristic encoding; after obtaining the code stream of the audio signal at each layer, the terminal can also perform the following processing for each layer: configure a first transmission priority for the low-frequency code stream of the layer, and configure a second transmission priority for the high-frequency code stream of the layer; wherein the first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th layer is lower than the first transmission priority of the i-th layer, and the transmission priority of the code stream is positively correlated with the decoding quality indicator of the corresponding code stream.

[0150] It should be noted that, since the transmission priority of the code stream is positively correlated with the decoding quality index of the corresponding code stream, and since the data dimension of the high-frequency code stream is smaller than the data dimension of the low-frequency code stream, the original information of the audio signal contained in the low-frequency code stream of each layer exceeds the original information of the audio signal contained in the high-frequency code stream. That is to say, in order to ensure that the decoding quality index of the low-frequency code stream is better than the decoding quality of the high-frequency code stream, the first transmission priority can be configured for the low-frequency code stream of each layer, and the second transmission priority can be configured for the high-frequency code stream of the layer, and the first transmission priority is higher than the second transmission priority. At the same time, the second transmission priority of the (i-1)th layer can also be configured to be lower than the first transmission priority of the i-th layer, that is, for each layer, the transmission priority of the low-frequency code stream is higher than the transmission priority of the high-frequency code stream, so that the low-frequency code stream of each layer can be transmitted first; for multiple layers, the transmission priority of the low-frequency code stream of the i-th layer is higher than the transmission priority of the high-frequency code stream of the (i-1)th layer, so that all low-frequency code streams of multiple layers can be transmitted first.

[0151] By applying the above-mentioned embodiments of the present application, hierarchical encoding of audio signals is realized: first, the audio signal is subjected to a first-level feature extraction process to obtain the signal features of the first level; then, for the i-th level (i is an integer greater than 1, i is less than or equal to N) of N levels (N is an integer greater than 1), the audio signal and the signal features of the (i-1)-th level are spliced ​​to obtain spliced ​​features, and the i-th level feature extraction process is performed on the spliced ​​features to obtain the signal features of the i-th level; then, by traversing i, the signal features of each level in the N levels are obtained; finally, the signal features of the first level and the signal features of each level in the N levels are encoded respectively to obtain the code streams of the audio signal at each level.

[0152] First, the data dimension of the extracted signal features is smaller than that of the audio signal. This reduces the data dimension of the data processed during the audio encoding process and improves the encoding efficiency of the audio signal.

[0153] Second, when extracting the signal features of the audio signal in layers, the output of each layer is used as the input of the next layer, so that each layer combines the signal features extracted by the previous layer to extract more accurate features of the audio signal. As the number of layers increases, the information loss of the audio signal in the feature extraction process can be minimized. In this way, the multiple code streams obtained by encoding the signal features extracted in this way contain audio signal information that is closer to the original audio signal, reducing the information loss of the audio signal in the encoding process and ensuring the encoding quality of the audio encoding.

[0154] The following describes the audio decoding method provided by the embodiment of the present application. In some embodiments, the audio decoding method provided by the embodiment of the present application can be implemented by various electronic devices, for example, it can be implemented by a terminal alone, it can also be implemented by a server alone, or it can be implemented by a terminal and a server in collaboration. Fig.10 , Fig.10 : is a flowchart of an audio decoding method provided in an embodiment of the present application. The audio decoding method provided in an embodiment of the present application includes:

[0155] Step 601: The terminal receives code streams corresponding to multiple layers obtained by encoding an audio signal.

[0156] Here, the terminal acts as a decoding end and receives code streams corresponding to multiple layers obtained by encoding the audio signal.

[0157] Step 602: Decode the code streams at each level respectively to obtain signal features at each level.

[0158] Among them, the data dimension of the signal feature is smaller than the data dimension of the audio signal.

[0159] In some embodiments, the terminal may decode the code stream of each layer respectively in the following manner to obtain the signal characteristics of each layer: for each layer, the following processing is performed respectively: entropy decoding is performed on the code stream of the layer to obtain the quantization value of the code stream; inverse quantization is performed on the quantization value of the code stream to obtain the signal characteristics of the layer.

[0160] In practical applications, the following processing can be performed on the bit streams of each level: the bit stream of that level is entropy decoded to obtain the quantization value of the bit stream; then, based on the quantization table used in the process of encoding the audio signal to obtain the bit stream, the quantization value of the bit stream is inversely quantized, that is, the signal characteristics corresponding to the quantization value of the bit stream are queried through the quantization table to obtain the signal characteristics of the level.

[0161] In practical applications, the received code streams of each level may include low-frequency code streams and high-frequency code streams, wherein the low-frequency code stream is obtained by encoding based on the low-frequency signal characteristics of the audio signal, and the high-frequency code stream is obtained by encoding based on the high-frequency signal characteristics of the audio signal. In this way, when decoding the code streams of each level, the low-frequency code streams and high-frequency code streams of each level may be decoded separately. Among them, the decoding process of the high-frequency code stream and the low-frequency code stream is similar to the decoding process of the code stream, that is, for the low-frequency code streams of each level, the following processing is performed respectively: the low-frequency code stream of the level is entropy decoded to obtain the quantization value of the low-frequency code stream; the quantization value of the low-frequency code stream is inversely quantized to obtain the low-frequency signal characteristics of the level. For the high-frequency code streams of each level, the following processing is performed respectively: the high-frequency code stream of the level is entropy decoded to obtain the quantization value of the high-frequency code stream; the quantization value of the high-frequency code stream is inversely quantized to obtain the high-frequency signal characteristics of the level.

[0162] Step 603: reconstruct the signal features of each level respectively to obtain the hierarchical audio signals of each level.

[0163] In practical applications, after decoding to obtain the signal features of each level, the signal features of each level are reconstructed to obtain the hierarchical audio signals of each level. In some embodiments, the terminal can reconstruct the signal features of each level in the following manner to obtain the hierarchical audio signals of each level: for the signal features of each level, the following processing is performed respectively: a first convolution processing is performed on the signal features to obtain the convolution features of the level; an upsampling processing is performed on the convolution features to obtain the upsampled features of the level; a pooling processing is performed on the upsampled features to obtain the pooled features of the level; a second convolution processing is performed on the pooled features to obtain the hierarchical audio signals of the level.

[0164] In practical applications, the following processing is performed for the signal features of each level: first, the signal features are subjected to a first convolution process, which can be processed by calling a causal convolution with a preset number of channels, thereby obtaining the convolution features of this level. Then, the convolution features are upsampled, and an upsampling factor can be set in advance, so that upsampling is performed based on the upsampling factor to obtain the upsampled features of this level. Then, the upsampled features are pooled, and the pooling process can pre-set a pooling factor, and then the upsampled features are pooled based on the pooling factor to obtain the pooled features of this level. Finally, the pooled features are subjected to a second convolution process, and the second convolution process can be processed by calling a causal convolution with a preset number of channels, thereby obtaining the hierarchical audio signal of this level.

[0165] The upsampling process can be implemented through one decoding layer or through multiple decoding layers. When the upsampling process can be implemented through L (L>1) cascaded decoding layers, the terminal can perform upsampling on the convolutional features in the following manner to obtain the upsampling features of the level: perform upsampling on the pooling features through the first decoding layer in the L cascaded decoding layers to obtain the upsampling result of the first decoding layer; perform upsampling on the first upsampling result of the (k-1)th decoding layer through the kth decoding layer in the L cascaded decoding layers to obtain the upsampling result of the kth decoding layer; wherein L and k are integers greater than 1, and k is less than or equal to L; perform traversal on k to obtain the upsampling result of the Lth decoding layer, and use the upsampling result of the Lth decoding layer as the upsampling feature of the level.

[0166] It should be noted that the upsampling factor of each decoding layer may be the same or different.

[0167] Step 604: Perform audio synthesis on the hierarchical audio signals of multiple levels to obtain an audio signal.

[0168] In practical applications, after obtaining the hierarchical audio signals of each level, audio synthesis is performed on the hierarchical audio signals of multiple levels to obtain an audio signal.

[0169] In some embodiments, the code stream includes a low-frequency code stream and a high-frequency code stream. Fig.10 The step 602 shown can be implemented by the following steps: respectively decoding the low-frequency code streams of each level to obtain the low-frequency signal features of each level, and respectively decoding the high-frequency code streams of each level to obtain the high-frequency signal features of each level; accordingly, Fig.10 The step 603 shown can be implemented by the following steps: step 6031, respectively reconstructing the low-frequency signal features of each level to obtain the level low-frequency sub-band signals of each level, and respectively reconstructing the high-frequency signal features of each level to obtain the level high-frequency sub-band signals of each level; step 6032, using the level low-frequency sub-band signals and the level high-frequency sub-band signals as the level audio signals of the level; accordingly, Fig.10 The shown step 604 can be implemented by the following steps: step 6041, adding the hierarchical low-frequency sub-band signals of multiple levels to obtain a low-frequency sub-band signal, and adding the hierarchical high-frequency sub-band signals of multiple levels to obtain a high-frequency sub-band signal; step 6042, synthesizing the low-frequency sub-band signal and the high-frequency sub-band signal to obtain an audio signal.

[0170] In some embodiments, step 6042 can be implemented by the following steps: step 60421, upsampling the low-frequency subband signal to obtain a low-pass filter signal; step 60422, upsampling the high-frequency subband signal to obtain a high-frequency filter signal; step 60423, filtering and synthesizing the low-pass filter signal and the high-frequency filter signal to obtain an audio signal. It should be noted that in step 60423, the audio signal can be synthesized by a QMF synthesis filter.

[0171] Based on this, when the code stream includes low-frequency code stream and high-frequency code stream, refer to Fig.11 , Fig.11 701 is a flowchart of an audio decoding method provided by an embodiment of the present application. The audio decoding method provided by an embodiment of the present application includes: step 701, receiving low-frequency code streams and high-frequency code streams corresponding to multiple levels obtained by encoding an audio signal; step 702a, respectively decoding the low-frequency code streams of each level to obtain low-frequency signal features of each level; step 702b, respectively decoding the high-frequency code streams of each level to obtain high-frequency signal features of each level; step 703a, respectively reconstructing the low-frequency signal features of each level to obtain hierarchical low-frequency sub-band signals of each level; step 703b , reconstruct the high-frequency signal features of each level respectively to obtain the hierarchical high-frequency sub-band signals of each level; step 704a, add the hierarchical low-frequency sub-band signals of multiple levels to obtain the low-frequency sub-band signals; step 704b, add the hierarchical high-frequency sub-band signals of multiple levels to obtain the high-frequency sub-band signals; step 705a, up-sample the low-frequency sub-band signals to obtain the low-pass filtered signals; step 705b, up-sample the high-frequency sub-band signals to obtain the high-frequency filtered signals; step 706, filter and synthesize the low-pass filtered signals and the high-frequency filtered signals to obtain the audio signals.

[0172] It should be noted that the feature reconstruction process of the high-frequency signal features and the low-frequency signal features can refer to the feature reconstruction process of the signal features in step 603. That is, for the high-frequency signal features of each level, the following processing is performed respectively: the high-frequency signal features are subjected to a first convolution processing to obtain the high-frequency convolution features of the level; the high-frequency convolution features are subjected to an upsampling processing to obtain the high-frequency up-sampled features of the level; the high-frequency up-sampled features are subjected to a pooling processing to obtain the high-frequency pooled features of the level; the high-frequency pooled features are subjected to a second convolution processing to obtain the high-frequency level audio signal of the level. For the low-frequency signal features of each level, the following processing is performed respectively: the low-frequency signal features are subjected to a first convolution processing to obtain the low-frequency convolution features of the level; the low-frequency convolution features are subjected to an upsampling processing to obtain the low-frequency up-sampled features of the level; the low-frequency up-sampled features are subjected to a pooling processing to obtain the low-frequency pooled features of the level; the low-frequency pooled features are subjected to a second convolution processing to obtain the low-frequency level audio signal of the level.

[0173] By using the above-mentioned embodiments of the present application, the code streams of multiple levels are decoded and processed respectively to obtain the signal features of each level, and the signal features of each level are reconstructed to obtain the hierarchical audio signals of each level, and the hierarchical audio signals of multiple levels are synthesized to obtain the audio signals. Since the data dimension of the signal features in the code stream is smaller than the data dimension of the audio signal, the data dimension of the code stream obtained by directly encoding the original audio signal in the related art is smaller, which reduces the data dimension of the data processed in the audio decoding process and improves the decoding efficiency of the audio signal.

[0174] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0175] Audio codec technology is to use less network bandwidth resources to transmit as much voice information as possible. The compression rate of audio codec can reach more than 10 times, that is, the original 10MB of voice data only needs 1MB to be transmitted after compression by the encoder, which greatly reduces the bandwidth resources consumed to transmit information. In the communication system, in order to ensure smooth communication, the industry deploys standard voice codec protocols, such as standards from international and domestic standard organizations such as ITU-T, 3GPP, IETF, AVS, CCSA, G.711, G.722, AMR series, EVS, OPUS and other standards. Fig.12 A spectrum comparison diagram at different bit rates is given to demonstrate the relationship between compression bit rate and quality. Curve 1201 is the spectrum curve of the original speech, that is, the signal without compression; Curve 1202 is the spectrum curve of the OPUS encoder at a bit rate of 20kbps; Curve 1203 is the spectrum curve of the OPUS encoder at a bit rate of 6kbps. Fig.12 It can be seen that as the encoding bit rate increases, the compressed signal is closer to the original signal.

[0176] Traditional audio coding can be divided into two categories: time domain coding and frequency domain coding, both of which are compression methods based on signal processing. Among them, 1) time domain coding, such as waveform coding (waveform speech coding), directly encodes the waveform of the speech signal. The advantage of this coding method is that the encoded speech quality is high, but the coding efficiency is not high. In particular, if it is a speech signal, parameter coding can be used, and what the encoding end has to do is to extract the corresponding parameters of the speech signal to be transmitted. However, the advantage of parameter coding is that the coding efficiency is extremely high, but the quality of the restored speech is very low. 2) Frequency domain coding is to transform the audio signal into the frequency domain, extract the frequency domain coefficients, and then encode the frequency domain coefficients, but the coding efficiency is also not ideal. In this way, the compression method based on signal processing cannot improve the coding efficiency while ensuring the coding quality.

[0177] Based on this, the embodiments of the present application provide an audio encoding method and an audio decoding method to improve the encoding efficiency while ensuring the encoding quality. In the embodiments of the present application, the freedom to select different encoding methods can be achieved according to the encoding content and network bandwidth, even in a low bit rate range; and the encoding efficiency can be improved under the condition that the complexity and encoding quality are acceptable. Fig.13 , Fig.13 It is a flowchart of audio encoding and audio decoding provided by an embodiment of the present application. Here, taking the number of layers as two as an example (the present application does not limit the iterative operation of the third layer or higher layers), the audio encoding method provided by the embodiment of the present application includes:

[0178] (1) Perform sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal. In actual implementation, the audio signal can be sampled at a first sampling frequency to obtain a sampled signal, and then the sampled signal can be sub-band decomposed to obtain a sub-band signal with a frequency lower than the first sampling frequency, including a low-frequency sub-band signal and a high-frequency sub-band signal. For example, for the audio signal x(n) of the nth frame, an analysis filter (such as a QMF filter) is used to decompose it into a low-frequency sub-band signal x LB (n) and the high frequency subband signal x HB (n).

[0179] (2) Analyze the low-frequency subband signal based on the first-layer low-frequency analysis neural network to obtain the first-layer low-frequency signal features. For example, for the low-frequency subband signal x LB (n), call the first layer of low-frequency analysis neural network to obtain the low-dimensional first layer of low-frequency signal features F LB (n). It should be noted that the dimension of the signal feature is smaller than the dimension of the low-frequency subband signal (to reduce the amount of data), and the neural network includes but is not limited to Dilated CNN, Autoencoder, Full-connection, LSTM, CNN+LSTM, etc.

[0180] (3) Analyze the high-frequency sub-band signal based on the first-layer high-frequency analysis neural network to obtain the first-layer high-frequency signal features. For example, for the high-frequency sub-band signal x HB (n), call the first layer of high-frequency analysis neural network to obtain the low-dimensional first layer high-frequency signal feature F HB (n).

[0181] (4) Based on the second-layer low-frequency analysis neural network, the low-frequency subband signal and the first-layer low-frequency signal features are analyzed to obtain the second-layer low-frequency signal features (i.e., the second-layer low-frequency residual signal features). LB (n) and F LB(n), call the second layer low-frequency analysis neural network to obtain the low-dimensional second layer low-frequency signal feature F LB,e (n).

[0182] (5) Based on the second-layer high-frequency analysis neural network, the high-frequency subband signal and the first-layer high-frequency signal features are analyzed to obtain the second-layer high-frequency signal features (i.e., the second-layer high-frequency residual signal features). HB (n) and F HB (n), call the second-layer high-frequency analysis neural network to obtain the low-dimensional second-layer high-frequency signal feature F HB,e (n).

[0183] (6) Through the quantization and coding part, the two-layer signal characteristics (including the first-layer low-frequency signal characteristics, the first-layer high-frequency signal characteristics, the second-layer low-frequency signal characteristics, and the second-layer high-frequency signal characteristics) are quantized and encoded to obtain the code stream of the audio signal at each layer; and the corresponding transmission priority is configured for the code stream of each layer, for example, the first layer is transmitted with a higher priority, the second layer is the second, and so on.

[0184] In actual applications, the decoder may only receive one layer of code stream, such as Fig.13 As shown, the decoding process can be performed in a "one-layer decoding" manner. Based on this, the audio decoding method provided in the embodiment of the present application includes: (1) decoding a received layer of code stream to obtain the low-frequency signal characteristics and high-frequency signal characteristics of the layer; (2) based on the first layer of low-frequency synthesis neural network, analyzing the low-frequency signal characteristics to obtain the low-frequency subband signal estimation value. For example, based on the quantization value F of the low-frequency signal characteristics L ' B (n), call the first layer of low-frequency synthesis neural network to generate the low-frequency subband signal estimate x L ' B (n); (3) Based on the first layer of high-frequency synthetic neural network, the high-frequency signal characteristics are analyzed to obtain the high-frequency subband signal estimation value. For example, the quantized value F based on the high-frequency signal characteristics H ' B (n), call the first layer of high-frequency synthesis neural network to generate the high-frequency subband signal estimation value x′ HB (n). (4) Based on the low-frequency subband signal estimate x L ' B (n) and the high frequency subband signal estimate x′ HB (n), and performs synthesis filtering through a synthesis filter to obtain the final reconstructed audio signal x′(n) at the original sampling frequency to complete the decoding process.

[0185] In actual applications, the decoding end may receive both layers of code streams, such as Fig.13As shown, the decoding process can be performed in a "two-layer decoding" manner. Based on this, the audio decoding method provided in the embodiment of the present application includes:

[0186] (1) Decode the received code streams of each layer to obtain the low-frequency signal characteristics and high-frequency signal characteristics of each layer.

[0187] (2) Based on the first layer low-frequency synthesis neural network, the first layer low-frequency signal characteristics are analyzed to obtain the first layer low-frequency subband signal estimation value. For example, based on the quantized value F of the first layer low-frequency signal characteristics L ' B (n), call the first layer of low-frequency synthesis neural network to generate the first layer of low-frequency subband signal estimation value x L ' B (n).

[0188] (3) Based on the first-layer high-frequency synthetic neural network, the first-layer high-frequency signal characteristics are analyzed to obtain the first-layer high-frequency subband signal estimation value. For example, based on the quantized value F of the first-layer high-frequency signal characteristics H ' B (n), call the first layer of high-frequency synthesis neural network to generate the first layer of high-frequency subband signal estimation value x′ HB (n).

[0189] (4) Based on the second-layer low-frequency synthesis neural network, the second-layer low-frequency signal characteristics are analyzed to obtain the second-layer low-frequency sub-band residual signal estimation value. For example, the quantized value F based on the second-layer low-frequency signal characteristics L ' B,e (n), call the second layer of low-frequency synthesis neural network to generate the low-frequency sub-band residual signal estimate x L ' B,e (n).

[0190] (5) Based on the second-layer high-frequency synthetic neural network, the second-layer high-frequency signal characteristics are analyzed to obtain the second-layer high-frequency sub-band residual signal estimation value. For example, the quantized value F based on the second-layer high-frequency signal characteristics H ' B,e (n), call the second layer of high-frequency synthesis neural network to generate the high-frequency sub-band residual signal estimate x′ HB,e (n).

[0191] (6) Through the low-frequency part, the first-layer low-frequency subband signal estimation value and the low-frequency subband residual signal estimation value are summed to obtain the low-frequency subband signal estimation value. L ' B (n) and x L ' B,e (n) Sum and obtain the low-frequency subband signal estimation value.

[0192] (7) Through the high frequency part, the high frequency subband signal estimation value of the first layer is summed with the high frequency subband residual signal estimation value to obtain the high frequency subband signal estimation value. For example, x′ HB (n) and x′ HB,e (n) Sum to obtain a high-quality high-frequency sub-band signal estimate.

[0193] (8) Based on the low-frequency subband signal estimation value and the high-frequency subband signal estimation value, synthesis filtering is performed through a synthesis filter to obtain the final reconstructed audio signal x′(n) at the original sampling frequency to complete the decoding process.

[0194] The embodiments of the present application can be applied to various audio scenarios, such as remote voice communication. Fig.14 , Fig.14 It is a schematic diagram of the voice communication link provided by the embodiment of the present application. Here, taking the voice transmission based on the Internet interconnection protocol (VoIP, Voice over Internet Protocol) conference system as an example, the voice encoding and decoding technology involved in the embodiment of the present application is deployed in the encoding and decoding parts to solve the basic function of voice compression. The encoder is deployed on the uplink client 1401, and the decoder is deployed on the downlink client 1402. The voice is collected through the uplink client, and pre-processing enhancement, encoding and other processing are performed. The encoded code stream is transmitted to the downlink client 1402 through the network, and decoding, enhancement and other processing are performed by the downlink client 1402 to play the decoded voice on the downlink client 1402.

[0195] Considering forward compatibility (that is, the new encoder is compatible with the existing encoder), it is necessary to deploy a transcoder in the background of the system (that is, the server) to solve the problem of interoperability between the new encoder and the existing encoder. For example, if the sending end (uplink client) is a new NN encoder, and the receiving end (downlink client) is a decoder of the public switched telephone network (PSTN, Public Switched Telephone Network) (such as a G.722 decoder). Therefore, after receiving the code stream sent by the sender, the server first needs to execute the NN decoder to generate a voice signal, and then call the G.722 encoder to generate a specific code stream so that the receiving end can decode it correctly. Similar transcoding scenarios are no longer expanded.

[0196] Before specifically introducing the audio encoding method and audio decoding method provided in the embodiments of the present application, the QMF filter group and the dilated convolutional network are first introduced.

[0197] The QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, which represent the QMF low-pass signal and the QMF high-pass signal respectively. Fig.15 The spectral response of the low-pass part (H_Low(z)) and the high-pass part (H_High(z)) of the QMF filter shown. Based on the relevant theoretical knowledge of the QMF analysis filter bank, the correlation between the coefficients of the above low-pass filter and high-pass filter can be easily described, as shown in formula (1):

[0198] h High (k) = -1 k h Low (k) (1)

[0199] Among them, h Low (k) represents the coefficient of low-pass filtering, h High (k) represents the coefficient of high-pass filtering.

[0200] Similarly, according to QMF related theory, the QMF synthesis filter group can be described based on the QMF analysis filter group H_Low(z) and H_High(z), as shown in formula (2).

[0201] G Low (z) = H Low (z)

[0202] G High (z)=(-1)*H High (z) (2)

[0203] Among them, G Low (z) represents the restored low-pass signal, G High (z) represents the recovered high-pass signal.

[0204] The low-pass and high-pass signals recovered by the decoding end are synthesized by the QMF synthesis filter bank, so that the reconstructed signal with the sampling rate Fs corresponding to the input signal can be restored.

[0205] See also Fig.16A and Fig. 16B , Fig.16A is a schematic diagram of a common convolutional network provided in an embodiment of the present application, Fig. 16B Schematic diagram of a dilated convolutional network provided in an embodiment of the present application. Compared with ordinary convolutional networks, dilated convolution can increase the receptive field while keeping the size of the feature map unchanged, and can also avoid errors caused by upsampling and downsampling. Fig.16A and Fig. 16B The convolution kernel size shown in is 3×3; however, Fig.16A The receptive field of the ordinary convolution shown is only 3, while Fig. 16B The receptive field 902 of the dilated convolution shown reaches 5. That is, for a convolution kernel of size 3×3, Fig.16A The receptive field of the ordinary convolution shown is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1; Fig. 16B The dilated convolution shown has a receptive field of 5 and a dilation rate of 2.

[0206] The convolution kernel can also be used in similar Fig.16A or Fig. 16B Move on the plane, here involves the concept of shift rate (Stride Rate). For example, each time the convolution kernel shifts 1 grid, the corresponding shift rate is 1. In addition, there is the concept of the number of convolution channels, which is the number of parameters corresponding to the convolution kernel used to perform convolution analysis. In theory, the more channels there are, the more comprehensive the analysis of the signal and the higher the accuracy; however, the higher the channel, the higher the complexity. For example, a 1×320 tensor can use a 24-channel convolution operation, and the output is a 24×320 tensor. It should be noted that the size of the hole convolution kernel (for example: for speech signals, the size of the convolution kernel can be set to 1×3), the expansion rate, the shift rate and the number of channels can be defined by yourself according to the actual application needs, and the embodiments of the present application do not make specific limitations on this.

[0207] The following takes an audio signal with Fs=32000 Hz as an example (the embodiments of the present application are also applicable to scenarios with other sampling frequencies, including but not limited to 8000 Hz, 16000 Hz, 48000 Hz, etc.), where the frame length is set to 20 ms. For Fs=32000 Hz, it is equivalent to containing 640 sample points per frame.

[0208] Next, see Fig.13 , the audio encoding method and the audio decoding method provided in the embodiment of the present application are described in detail respectively. Among them, the audio encoding method provided in the embodiment of the present application includes:

[0209] Step 1: Generation of input signal.

[0210] Here, the 640 sample points of the nth frame are recorded as x(n).

[0211] Step 2: QMF subband signal decomposition.

[0212] Here, a QMF analysis filter (such as a 2-channel QMF filter) is called for filtering, and the filtered signal obtained by filtering is downsampled to obtain two sub-band signals, namely, the low-frequency sub-band signal x LB (n) and the high frequency subband signal x HB(n). Among them, the low-frequency subband signal x LB The effective bandwidth of (n) is 0-8kHz, and the high-frequency sub-band signal x HB The effective bandwidth of (n) is 8-16kHz, and the number of samples per frame is 320.

[0213] Step 3: First layer low frequency analysis.

[0214] Here, the purpose of calling the first layer of low-frequency analysis neural network is to LB (n), generate the first layer low-frequency signal features F of lower dimension LB (n). In this example, x LB The data dimension of (n) is 320, and F LB The data dimension of (n) is 64. From the data volume, it is obvious that after the first layer of low-frequency analysis neural network, it has played a role of "dimensionality reduction", which can be understood as data compression. As an example, see Fig.17 , Fig.17 is a schematic diagram of the structure of the first layer of low-frequency analysis neural network provided in the embodiment of the present application, for the low-frequency sub-band signal x LB The processing flow of (n) includes:

[0215] (1) Call a 24-channel causal convolution to convert the input tensor (i.e. x LB (n)), expanded to a 24*320 tensor.

[0216] (2) Preprocess the 24*320 tensor. In practical applications, a pooling operation with a pooling factor of 2 and an activation function of ReLU can be performed to generate a 24*160 tensor.

[0217] (3) Cascade three coding blocks with different downsampling factors (Down_factor). Taking the coding block (Down_factor=4) as an example, one or more dilated convolutions can be performed first, and the size of each convolution kernel is fixed to 1*3, and the shift rate (Striderate) is 1. In addition, the dilation rate (Dilation rate) of the one or more dilated convolutions can be set as needed, such as 3; of course, the embodiment of the present application does not limit the setting of different dilation rates for different dilated convolutions. Then, the Down_factor of the three coding blocks is set to 4, 5, and 8, respectively, which is equivalent to setting pooling factors of different sizes, which plays a downsampling role. Finally, the number of channels of the three coding blocks is set to 48, 96, and 192, respectively. Therefore, after three cascaded coding blocks, the 24*160 tensor is converted into 48*40, 96*8 and 192*1 tensors respectively.

[0218] (4) After similar preprocessing of the 192*1 tensor, a 64-dimensional feature vector is output, i.e., the first-layer low-frequency signal feature F LB (n).

[0219] Step 4: First layer high frequency analysis.

[0220] Here, the purpose of calling the first layer of high-frequency analysis neural network is to HB (n), generate the first layer high-frequency signal features F of lower dimension HB (n). In this example, the structure of the first layer of high-frequency analysis neural network can be consistent with the first layer of low-frequency analysis neural network, that is, the input (i.e., x HB (n)) has a data dimension of 320, and the output (i.e., F HB The data dimension of (n) is 64. Considering that the high-frequency subband signal is less important than the low-frequency subband signal, the output dimension can be appropriately reduced, which can reduce the complexity of the first layer of high-frequency analysis neural network, which is not limited in this example.

[0221] Step 5: Second layer of low frequency analysis.

[0222] Here, the purpose of calling the second layer of low-frequency analysis neural network is to LB (n) and the first layer low-frequency signal feature F LB (n), and obtain the second-layer low-frequency signal feature F with lower dimension LB,e (n). The second-layer low-frequency signal characteristics reflect the residual of the reconstructed audio signal at the decoding end relative to the original audio signal of the output of the first-layer low-frequency analysis neural network; therefore, at the decoding end, the F LB,e (n) Predicting the residual signal of the low-frequency sub-band signal and summing it with the estimated value of the low-frequency sub-band signal predicted by the output of the first layer of low-frequency analysis neural network to obtain a more accurate estimated value of the low-frequency sub-band signal.

[0223] The second layer of low frequency analysis neural network adopts a similar structure to the first layer of low frequency analysis neural network. Fig.18 , Fig.18 : is a schematic diagram of the structure of the second layer of low frequency analysis neural network provided by the embodiment of the present application. Here, the main differences from the first layer of low frequency analysis neural network type include: (1) The input of the second layer of low frequency analysis neural network includes the low frequency subband signal x LB (n), and also includes the output F of the first layer of low-frequency analysis neural network LB (n), x LB (n) and F Lb(n) The two variables can be concatenated into a 384-dimensional concatenated feature. (2) Considering that the second layer of low-frequency analysis processes the residual signal, the output F of the second layer of low-frequency analysis neural network LB,e The dimension of (n) is set to 28.

[0224] Step 6: Second level high frequency analysis.

[0225] Here, the purpose of calling the second layer of high-frequency analysis neural network is to HB (n) and the first layer high frequency signal feature F HB (n), and obtain the second-layer high-frequency signal feature F with lower dimensions HB,e (n). The second layer of high frequency analysis neural network and structure can be the same as the second layer of low frequency analysis neural network structure, that is, input (x HB (n) and F HB The data dimension of the concatenated features of (n) is 384, and the output (F HB,e The data dimension of (n)) is 28.

[0226] Step 7: Quantization encoding.

[0227] By querying a pre-set quantization table, the signal features of the 2-layer output are quantized, and the quantization result obtained by quantization is encoded, wherein the quantization can be performed in a scalar quantization manner (each component is quantized separately), and the encoding can be performed in an entropy coding manner. In addition, the embodiment of the present application does not limit the technical combination of vector quantization (adjacent multiple components are combined into a vector for joint quantization) and entropy coding.

[0228] In actual implementation, the first layer of low-frequency signal features F LB (n) is a 64-dimensional feature, which can be encoded using 8kbps. The average bit rate of quantizing one parameter per frame is 2.5 bits. The first layer of high-frequency signal features F HB (n) is a 64-dimensional feature, which can be encoded using 6 kbps. The average bit rate of quantizing one parameter per frame is 1.875 bits. Therefore, the first layer of encoding is 14 kbps in total.

[0229] In actual implementation, the second layer low-frequency signal feature F LB,e (n) is a 28-dimensional feature, which can be encoded using 3.5 kbps. The average bit rate of quantizing one parameter per frame is 2.5 bits. The second-layer high-frequency signal feature F HB,e (n) is a 28-dimensional feature, which can be encoded using 3.5 kbps. The average bit rate for quantizing one parameter per frame is 2.5 bits. Therefore, the second layer of encoding is 7 kbps in total.

[0230] Based on this, different feature vectors can be progressively encoded by layered coding; according to different application scenarios, the embodiments of the present application do not limit the code rate distribution of other methods, for example, the third layer or higher layer coding can be introduced iteratively. After quantization coding, a code stream can be generated. Different transmission strategies can be adopted for code streams of different layers to ensure transmission with different priorities. For example, a forward error correction mechanism (Forward Error Correction, FEC) can be adopted to improve the quality of transmission through redundant transmission. The redundancy multiples of different layers are different. For example, the redundancy multiple of the first layer can be set higher.

[0231] Taking the case where the bitstreams of all layers are received and accurately decoded by the decoding end as an example, the audio encoding method provided in the embodiment of the present application includes:

[0232] Step 1, decoding.

[0233] Here, decoding is the inverse process of encoding. The received bitstream is parsed, and the low-frequency signal feature estimation value and the high-frequency signal feature estimation value are obtained by looking up the quantization table. Specifically, at the first level, the quantization value F of the 64-dimensional signal feature of the low-frequency subband signal is obtained. L ' B (n), and the quantized value F of the 64-dimensional signal feature of the high-frequency subband signal H ' B (n); The second layer obtains the quantized value F of the 28-dimensional signal feature of the low-frequency subband signal L ' B,e (n), and the quantized value F of the 28-dimensional signal feature of the high-frequency subband signal H ' B,e (n).

[0234] Step 2: The first layer of low frequency synthesis.

[0235] Here, the purpose of calling the first layer of low-frequency synthesis neural network is to obtain the quantized value F′ of the low-frequency feature vector LB (n), generate the first layer low frequency subband signal estimate x′ LB (n). For example, see Fig.19 , Fig.19 This is a schematic diagram of the model of the first layer of low-frequency synthesis neural network provided by the embodiment of the present application. Here, the processing flow of the first layer of low-frequency synthesis neural network is similar to the processing flow of the first layer of low-frequency analysis neural network, such as causal convolution; the post-processing structure of the first layer of low-frequency synthesis neural network is similar to the pre-processing structure of the first layer of low-frequency analysis neural network; the decoding block structure is symmetrical to the encoding block structure: the encoding block on the encoding side first performs dilated convolution and then pooling to complete downsampling, and the decoding block on the decoding side first performs pooling to complete upsampling and then performs dilated convolution.

[0236] Step 3: First layer of high frequency synthesis.

[0237] Here, the structure of the first layer of high-frequency synthesis neural network is the same as that of the first layer of low-frequency synthesis neural network. H ' B (n), obtain the first high frequency sub-band signal estimation value x′ HB (n).

[0238] Step 4: Second layer of low frequency synthesis.

[0239] Here, the purpose of calling the second layer of low-frequency synthesis neural network is to quantize the value F′ based on the characteristics of the second layer of low-frequency signal. LB,e (n), generate the low-frequency sub-band residual signal estimate x′ LB,e (n). See Fig. 20 , Fig. 20 It is a schematic diagram of the structure of the second layer of low-frequency synthesis neural network provided in an embodiment of the present application. The structure of the second layer of low-frequency synthesis neural network is similar to the result of the first layer of low-frequency synthesis neural network, and the difference is that the input data dimension is 28 dimensions.

[0240] Step 5: Second layer of high frequency synthesis.

[0241] Here, the structure of the second layer low-frequency synthesis neural network is the same as that of the second layer low-frequency synthesis neural network, which can be based on the quantization value F of the second layer low-frequency signal feature. H ' B,e (n), generate the high-frequency sub-band residual signal estimate x′ HB,e (n).

[0242] Step 6: Synthesis filtering.

[0243] Based on the previous step, the decoder obtains the low-frequency subband signal estimate x′ LB (n) and high frequency subband signal x′ HB (n), and the low-frequency subband residual signal estimate x′ LB,e (n) and the high-frequency subband signal residual estimate x′ HB,e (n). LB (n) and x′ LB,e (n) are added to generate a high-precision low-frequency subband signal estimate; x′ HB (n) and x′ HB,e (n) are added to generate a high-precision high-frequency subband signal estimation value. Finally, the low-frequency subband signal estimation value and the high-frequency subband signal estimation value are upsampled, and the QMF synthesis filter is called to perform synthesis filtering on the upsampled result to generate a 640-point reconstructed audio signal x′(n).

[0244] In the embodiment of the present application, the relevant neural networks of the encoding end and the decoding end can be jointly trained by collecting data to obtain the optimal parameters, so that the trained network model can be put into use. In the embodiment of the present application, only a specific embodiment of network input, network structure and network output is disclosed; engineers in related fields can further modify the above configuration as needed.

[0245] By applying the above embodiments of the present application, a low-bitrate audio encoding and decoding solution based on signal processing and deep learning network can be completed. Through the organic combination of signal decomposition and related signal processing technology with deep neural network, the coding efficiency has been significantly improved compared with the related technology, and the coding quality has also been improved under the condition of acceptable complexity. According to different coding contents and bandwidth conditions, the encoding end selects different layered transmission strategies to transmit the code stream. The decoding end receives the low-level code stream and outputs an audio signal of acceptable quality. If other high-level code streams are also received, high-quality audio can be output.

[0246] It is understandable that in the embodiments of the present application, when user information (such as audio signals sent by users) and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0247] The following further describes an exemplary structure of the audio encoding device 553 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the audio encoding device 553 of the memory 550 may include:

[0248] The first feature extraction module 5531 is used to perform a first-level feature extraction process on the audio signal to obtain the signal features of the first level; the second feature extraction module 5532 is used to perform a splicing process on the audio signal and the signal features of the (i-1)th level for the i-th level among the N levels to obtain a splicing feature, and perform an i-th level feature extraction process on the splicing feature to obtain the signal features of the i-th level, wherein N and i are integers greater than 1, and i is less than or equal to N; the traversal module 5533 is used to traverse i to obtain the signal features of each level among the N levels, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal; the encoding module 5534 is used to perform encoding processing on the signal features of the first level and the signal features of each level among the N levels, respectively, to obtain the code streams of the audio signal at each level.

[0249] In some embodiments, the first feature extraction module 5531 is further used to perform sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal; perform a first-level feature extraction processing on the low-frequency sub-band signal to obtain the low-frequency signal features of the first level, and perform a first-level feature extraction processing on the high-frequency sub-band signal to obtain the high-frequency signal features of the first level; and use the low-frequency signal features and the high-frequency signal features as the signal features of the first level.

[0250] In some embodiments, the first feature extraction module 5531 is further used to sample the audio signal at a first sampling frequency to obtain a sampled signal; perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, and down-sample the low-pass filtered signal to obtain the low-frequency sub-band signal of a second sampling frequency; perform high-pass filtering on the sampled signal to obtain a high-pass filtered signal, and down-sample the high-pass filtered signal to obtain the high-frequency sub-band signal of the second sampling frequency; wherein the second sampling frequency is less than the first sampling frequency.

[0251] In some embodiments, the second feature extraction module 5532 is further used to perform splicing processing on the low-frequency subband signal of the audio signal and the low-frequency signal feature of the (i-1)th level to obtain a first splicing feature, and perform the feature extraction processing of the i-th level on the first splicing feature to obtain the low-frequency signal feature of the i-th level; perform splicing processing on the high-frequency subband signal of the audio signal and the high-frequency signal feature of the (i-1)th level to obtain a second splicing feature, and perform the feature extraction processing of the i-th level on the second splicing feature to obtain the high-frequency signal feature of the i-th level; and use the low-frequency signal feature of the i-th level and the high-frequency signal feature of the i-th level as the signal feature of the i-th level.

[0252] In some embodiments, the first feature extraction module 5531 is also used to perform a first convolution process on the audio signal to obtain convolution features of the first level; perform a first pooling process on the convolution features to obtain pooling features of the first level; perform a first downsampling process on the pooling features to obtain downsampling features of the first level; perform a second convolution process on the downsampling features to obtain signal features of the first level.

[0253] In some embodiments, the first downsampling process is implemented through M cascaded coding layers, and the first feature extraction module 5531 is also used to perform a first downsampling process on the pooling feature through the first coding layer of the M cascaded coding layers to obtain the downsampling result of the first coding layer; perform a first downsampling process on the downsampling result of the (j-1)th coding layer through the jth coding layer of the M cascaded coding layers to obtain the downsampling result of the jth coding layer; wherein, M and j are integers greater than 1, and j is less than or equal to M; traverse j to obtain the downsampling result of the Mth coding layer, and use the downsampling result of the Mth coding layer as the downsampling feature of the first level.

[0254] In some embodiments, the second feature extraction module 5532 is also used to perform a third convolution process on the splicing features to obtain the convolution features of the i-th level; perform a second pooling process on the convolution features to obtain the pooling features of the i-th level; perform a second downsampling process on the pooling features to obtain the downsampling features of the i-th level; perform a fourth convolution process on the downsampling features to obtain the signal features of the i-th level.

[0255] In some embodiments, the encoding module 5534 is also used to perform quantization processing on the signal characteristics of the first layer and the signal characteristics of each layer in the N layers, respectively, to obtain the quantization results of the signal characteristics of each layer; perform entropy coding processing on the quantization results of the signal characteristics of each layer, to obtain the code stream of the audio signal at each layer.

[0256] In some embodiments, the signal characteristics include low-frequency signal characteristics and high-frequency signal characteristics. The encoding module 5534 is also used to encode the low-frequency signal characteristics of the first layer and the low-frequency signal characteristics of each layer in the N layers, respectively, to obtain the low-frequency code stream of the audio signal at each layer; encode the high-frequency signal characteristics of the first layer and the high-frequency signal characteristics of each layer in the N layers, respectively, to obtain the high-frequency code stream of the audio signal at each layer; and use the low-frequency code stream and high-frequency code stream of the audio signal at each layer as the code stream of the audio signal at the corresponding layer.

[0257] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features. The encoding module 5534 is further used to encode the low-frequency signal features of the first level according to a first encoding rate to obtain a first code stream of the first level, and to encode the high-frequency signal features of the first level according to a second encoding rate to obtain a second code stream of the first level; for the signal features of each of the N levels, the following processing is performed respectively: according to the third encoding rate of the level, the signal features of the level are respectively encoded to obtain the second code stream of each of the levels; the second code stream of the first level and the second code stream of each of the N levels are used as the code stream of the audio signal at each level; wherein the first encoding rate is greater than the second encoding rate, the second encoding rate is greater than the third encoding rate of any level of the N levels, and the encoding rate of the level is positively correlated with the decoding quality index of the code stream of the corresponding level.

[0258] In some embodiments, the encoding module 5534 is also used to perform the following processing for each of the layers: configuring a corresponding layer transmission priority for the code stream of the audio signal at the layer; wherein the layer transmission priority is negatively correlated with the layer number of the layer, and the layer transmission priority is positively correlated with the decoding quality index of the code stream of the corresponding layer.

[0259] In some embodiments, the signal characteristics include low-frequency signal characteristics and high-frequency signal characteristics, and the code stream of the audio signal at each level includes: a low-frequency code stream obtained by encoding based on the low-frequency signal characteristics, and a high-frequency code stream obtained by encoding based on the high-frequency signal characteristics; the encoding module 5534 is also used to perform the following processing for each of the levels: configure a first transmission priority for the low-frequency code stream of the level, and configure a second transmission priority for the high-frequency code stream of the level; wherein the first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th level is lower than the first transmission priority of the i-th level, and the transmission priority of the code stream is positively correlated with the decoding quality index of the corresponding code stream.

[0260] By applying the above-mentioned embodiments of the present application, hierarchical encoding of audio signals is realized: first, the audio signal is subjected to a first-level feature extraction process to obtain the signal features of the first level; then, for the i-th level (i is an integer greater than 1, i is less than or equal to N) of N levels (N is an integer greater than 1), the audio signal and the signal features of the (i-1)-th level are spliced ​​to obtain spliced ​​features, and the i-th level feature extraction process is performed on the spliced ​​features to obtain the signal features of the i-th level; then, by traversing i, the signal features of each level in the N levels are obtained; finally, the signal features of the first level and the signal features of each level in the N levels are encoded respectively to obtain the code streams of the audio signal at each level.

[0261] First, the data dimension of the extracted signal features is smaller than that of the audio signal. This reduces the data dimension of the data processed during the audio encoding process and improves the encoding efficiency of the audio signal.

[0262] Second, when extracting the signal features of the audio signal in layers, the output of each layer is used as the input of the next layer, so that each layer combines the signal features extracted by the previous layer to extract more accurate features of the audio signal. As the number of layers increases, the information loss of the audio signal in the feature extraction process can be minimized. In this way, the multiple code streams obtained by encoding the signal features extracted in this way contain audio signal information that is closer to the original audio signal, reducing the information loss of the audio signal in the encoding process and ensuring the encoding quality of the audio encoding.

[0263] The audio decoding device provided by the embodiment of the present application is described below. The audio decoding device provided by the embodiment of the present application includes: a receiving module, which is used to receive the code streams corresponding to the multiple layers obtained by encoding the audio signal; a decoding module, which is used to decode the code streams of each layer respectively to obtain the signal characteristics of each layer, and the data dimension of the signal characteristics is smaller than the data dimension of the audio signal; a feature reconstruction module, which is used to reconstruct the signal characteristics of each layer respectively to obtain the hierarchical audio signals of each layer; an audio synthesis module, which is used to perform audio synthesis on the hierarchical audio signals of multiple layers to obtain the audio signal.

[0264] In some embodiments, the code stream includes a low-frequency code stream and a high-frequency code stream, and the decoding module is further used to decode the low-frequency code streams of each of the layers respectively to obtain the low-frequency signal features of each of the layers, and to decode the high-frequency code streams of each of the layers respectively to obtain the high-frequency signal features of each of the layers; accordingly, the feature reconstruction module is further used to reconstruct the low-frequency signal features of each of the layers respectively to obtain the hierarchical low-frequency sub-band signals of each of the layers, and to reconstruct the high-frequency signal features of each of the layers respectively to obtain the hierarchical high-frequency sub-band signals of each of the layers; the hierarchical low-frequency sub-band signals and the hierarchical high-frequency sub-band signals are used as the hierarchical audio signals of the layers; accordingly, the audio synthesis module is further used to add the hierarchical low-frequency sub-band signals of multiple layers to obtain the low-frequency sub-band signals, and to add the hierarchical high-frequency sub-band signals of multiple layers to obtain the high-frequency sub-band signals; the low-frequency sub-band signals and the high-frequency sub-band signals are synthesized to obtain the audio signal.

[0265] In some embodiments, the audio synthesis module is further used to upsample the low-frequency subband signal to obtain a low-pass filtered signal; upsample the high-frequency subband signal to obtain a high-frequency filtered signal; and perform filter synthesis processing on the low-pass filtered signal and the high-frequency filtered signal to obtain the audio signal.

[0266] In some embodiments, the feature reconstruction module is also used to perform the following processing on the signal features of each of the levels: performing a first convolution processing on the signal features to obtain the convolution features of the level; performing upsampling processing on the convolution features to obtain the upsampled features of the level; performing pooling processing on the upsampled features to obtain the pooled features of the level; performing a second convolution processing on the pooled features to obtain the level audio signal of the level.

[0267] In some embodiments, the upsampling process is implemented through L cascaded decoding layers, and the feature reconstruction module is also used to upsample the pooled features through the first decoding layer of the L cascaded decoding layers to obtain the upsampling result of the first decoding layer; upsample the first upsampling result of the (k-1)th decoding layer through the kth decoding layer of the L cascaded decoding layers to obtain the upsampling result of the kth decoding layer; wherein, L and k are integers greater than 1, and k is less than or equal to L; traverse k to obtain the upsampling result of the Lth decoding layer, and use the upsampling result of the Lth decoding layer as the upsampling feature of the level.

[0268] In some embodiments, the decoding module is also used to perform the following processing for each of the layers: performing entropy decoding processing on the code stream of the layer to obtain the quantization value of the code stream; performing inverse quantization processing on the quantization value of the code stream to obtain the signal characteristics of the layer.

[0269] By using the above-mentioned embodiments of the present application, the code streams of multiple levels are decoded and processed respectively to obtain the signal features of each level, and the signal features of each level are reconstructed to obtain the hierarchical audio signals of each level, and the hierarchical audio signals of multiple levels are synthesized to obtain the audio signals. Since the data dimension of the signal feature is smaller than the data dimension of the audio signal, the data dimension of the data processed in the audio decoding process is reduced, and the decoding efficiency of the audio signal is improved.

[0270] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the embodiment of the present application.

[0271] An embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application.

[0272] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0273] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0274] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0275] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0276] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An audio encoding method, characterized in that: The method comprises: Performing a first-level feature extraction process on the audio signal to obtain the signal features of the first level; The audio signal and the signal features of the (i-1)th level are concatenated to obtain concatenated features, and Performing feature extraction processing of the i-th level on the splicing features to obtain the signal features of the i-th level, where i is a positive integer that increases in sequence, N is a positive integer greater than 1, and 1<i≤N; The signal features of each level in the N levels are quantized and encoded respectively to obtain the code stream of the audio signal at each level.

2. The method according to claim 1, characterized in that The performing a first-level feature extraction process on the audio signal to obtain the first-level signal features includes: Performing sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal; Performing a first-level feature extraction process on the low-frequency sub-band signal to obtain the first-level low-frequency signal features, and performing a first-level feature extraction process on the high-frequency sub-band signal to obtain the first-level high-frequency signal features; The low-frequency signal features and the high-frequency signal features are used as signal features of the first level.

3. The method according to claim 1 or 2, characterized in that The performing a first-level feature extraction process on the audio signal to obtain the first-level signal features includes: Performing a first convolution process on the audio signal to obtain convolution features of the first level; Performing a first pooling process on the convolutional features to obtain pooling features of the first level; Performing a first downsampling process on the pooled features to obtain downsampling features of the first level; A second convolution process is performed on the downsampled features to obtain signal features of the first level.

4. The method according to claim 3, characterized in that The first downsampling process is implemented by M cascaded coding layers, and the first downsampling process is performed on the pooled features to obtain the first-level downsampling features, including: Performing a first downsampling process on the pooled features through a first coding layer of the M cascaded coding layers to obtain a downsampling result of the first coding layer; Performing a first downsampling process on the downsampling result of the (j-1)th coding layer through the jth coding layer of the M cascaded coding layers to obtain the downsampling result of the jth coding layer, wherein j is a positive integer that increases successively, M is an integer greater than 1, and 1<j≤M; The downsampling result of the M-th coding layer is used as the downsampling feature of the first level.

5. The method according to claim 1, characterized in that The step of performing splicing processing on the audio signal and the signal feature of the (i-1)th level to obtain the splicing feature includes: Performing concatenation processing on the low-frequency subband signal of the audio signal and the low-frequency signal feature of the (i-1)th level to obtain a first feature; Performing concatenation processing on the high frequency subband signal of the audio signal and the high frequency signal feature of the (i-1)th level to obtain a second feature; Using the first feature and the second feature as the splicing feature; Performing feature extraction processing of the i-th level on the splicing features to obtain signal features of the i-th level includes: Performing feature extraction processing of the i-th level on the first feature to obtain the i-th level low-frequency signal feature; Performing feature extraction processing of the i-th level on the second feature to obtain the i-th level high-frequency signal feature; The low-frequency signal features of the i-th level and the high-frequency signal features of the i-th level are used as the signal features of the i-th level.

6. The method according to claim 1 or 5, characterized in that The step of performing feature extraction processing at the i-th level on the splicing features to obtain signal features at the i-th level includes: Performing a third convolution process on the concatenated features to obtain the convolution features of the i-th level; Performing a second pooling process on the convolutional features to obtain the pooling features of the i-th level; Performing a second downsampling process on the pooled features to obtain the downsampling features of the i-th level; Perform a fourth convolution process on the downsampled features to obtain the signal features of the i-th level.

7. The method according to claim 1, characterized in that The signal features include low-frequency signal features and high-frequency signal features. The signal features of each level in the N levels are quantized and encoded respectively to obtain the code stream of the audio signal at each level, including: The low-frequency signal features of each level in the N levels are respectively encoded to obtain the low-frequency code stream of the audio signal at each level; The high-frequency signal features of each level in the N levels are respectively encoded to obtain a high-frequency code stream of the audio signal at each level; The low-frequency code stream and the high-frequency code stream of the audio signal at each level are used as the code stream of the audio signal at the corresponding level.

8. The method according to claim 1, characterized in that The signal features include low-frequency signal features and high-frequency signal features. The signal features of each level in the N levels are encoded respectively to obtain the code stream of the audio signal at each level, including: According to a first coding rate, the low-frequency signal features of the first layer are encoded to obtain a low-frequency code stream of the first layer, and according to a second coding rate, the high-frequency signal features of the first layer are encoded to obtain a high-frequency code stream of the first layer, and the low-frequency code stream and the high-frequency code stream are used as the code stream of the first layer; Performing the following processing on the signal features of other layers: encoding the signal features of the other layers according to the third encoding code rate of the other layers to obtain the code stream of the other layers, wherein the other layers are layers other than the first layer in the N layers; Among them, the first encoding bit rate is greater than the second encoding bit rate, the second encoding bit rate is greater than the third encoding bit rate of any level in the N levels, and the encoding bit rate of each level is positively correlated with the decoding quality index of the code stream of each level.

9. The method according to claim 1, characterized in that After encoding the signal features of each level in the N levels to obtain the code stream of the audio signal at each level, the method further includes: For each of the above mentioned levels, the following processing is performed respectively: Configuring a corresponding layer transmission priority for the code stream of the audio signal at the layer; The layer transmission priority is negatively correlated with the layer number of the layer, and the layer transmission priority is positively correlated with the decoding quality index of the code stream of the corresponding layer.

10. The method according to claim 1, characterized in that The signal features include low-frequency signal features and high-frequency signal features, and the code streams of the audio signal at each level include: a low-frequency code stream obtained by encoding the low-frequency signal features, and a high-frequency code stream obtained by encoding the high-frequency signal features; The method further comprises: For each of the layers, the following processing is performed respectively: a first transmission priority is configured for a low-frequency code stream of the layer, and a second transmission priority is configured for a high-frequency code stream of the layer; Among them, the first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th level is lower than the first transmission priority of the i-th level, and the transmission priority of the code stream of each level is positively correlated with the decoding quality index of the corresponding code stream.

11. An audio decoding method, characterized in that: The method comprises: Receiving code streams corresponding to multiple layers obtained by encoding the audio signal; Decoding the code streams of each layer respectively to obtain signal characteristics of each layer; Reconstructing the signal features of each of the layers respectively to obtain the hierarchical audio signals of each of the layers; Audio synthesis is performed on the hierarchical audio signals of the multiple levels to obtain a reconstructed audio signal.

12. The method according to claim 11, characterized in that The code stream includes a low-frequency code stream and a high-frequency code stream, and the code streams of each layer are decoded to obtain signal characteristics of each layer, including: Decoding the low-frequency code streams of each layer to obtain the low-frequency signal characteristics of each layer, and decoding the high-frequency code streams of each layer to obtain the high-frequency signal characteristics of each layer; The step of respectively reconstructing the signal features of each of the layers to obtain the hierarchical audio signals of each of the layers comprises: Respectively reconstructing the low-frequency signal features of each level to obtain the level low-frequency sub-band signals of each level, and reconstructing the high-frequency signal features of each level to obtain the level high-frequency sub-band signals of each level; Using the hierarchical low-frequency subband signal and the hierarchical high-frequency subband signal as a hierarchical audio signal of the hierarchical level; The step of performing audio synthesis on the hierarchical audio signals of the multiple hierarchical levels to obtain a reconstructed audio signal comprises: Adding the hierarchical low-frequency subband signals of the multiple levels to obtain a low-frequency subband signal, and adding the hierarchical high-frequency subband signals of the multiple levels to obtain a high-frequency subband signal; The low-frequency sub-band signal and the high-frequency sub-band signal are synthesized to obtain a reconstructed audio signal.

13. The method according to claim 12, characterized in that The synthesizing the low-frequency sub-band signal and the high-frequency sub-band signal to obtain a reconstructed audio signal includes: Performing up-sampling processing on the low-frequency sub-band signal to obtain a low-pass filtered signal; Performing up-sampling processing on the high frequency sub-band signal to obtain a high frequency filtered signal; The low-pass filtered signal and the high-frequency filtered signal are subjected to filtering and synthesis processing to obtain a reconstructed audio signal.

14. The method according to claim 11, characterized in that The step of respectively reconstructing the signal features of each of the layers to obtain the hierarchical audio signals of each of the layers comprises: For the signal characteristics of each level, the following processing is performed respectively: Performing a first convolution process on the signal feature to obtain a convolution feature of the level; Performing upsampling processing on the convolutional features to obtain upsampled features of the level; Performing a de-pooling process on the up-sampled features to obtain de-pooled features of the level; A second convolution process is performed on the pooled features to obtain the hierarchical audio signal of the hierarchical level.

15. The method according to claim 14, characterized in that The upsampling process is implemented by L cascaded decoding layers, and the upsampling process is performed on the convolutional features to obtain the upsampling features of the level, including: Performing upsampling processing on the pooled features through a first decoding layer in the L cascaded decoding layers to obtain an upsampling result of the first decoding layer; Performing upsampling processing on the first upsampling result of the (k-1)th decoding layer through the kth decoding layer of the L cascaded decoding layers to obtain the upsampling result of the kth decoding layer, wherein k is a positive integer that increases successively, L is an integer greater than 1, and 1<k≤L; The upsampling result of the L-th decoding layer is used as the upsampling feature of the level.

16. A method for processing a code stream, characterized in that: A code stream is stored on a non-volatile computer-readable storage medium, wherein the code stream is generated based on the audio encoding method according to any one of claims 1 to 10, or is decoded according to the audio decoding method according to any one of claims 11 to 15.

17. An audio encoding device, characterized in that: The device comprises: A first feature extraction module, used to perform a first-level feature extraction process on the audio signal to obtain the signal features of the first level; A second feature extraction module is used to perform splicing processing on the audio signal and the signal features of the (i-1)th level to obtain splicing features, and perform feature extraction processing of the i-th level on the splicing features to obtain the signal features of the i-th level, where i is a positive integer that increases in sequence, N is a positive integer greater than 1, and 1<i≤N; The encoding module is used to encode the signal features of each level in the N levels respectively to obtain the code stream of the audio signal at each level.

18. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 15 when executing the executable instructions stored in the memory.

19. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented.

20. A computer program product comprising executable instructions, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images