Audio encoding methods, audio decoding methods, devices, equipment, storage media and software products

By using a hierarchical coding method to extract and concatenate features from audio signals at multiple levels, the problem of low audio coding efficiency in existing technologies is solved, achieving efficient audio coding while ensuring audio quality.

CN119517050BActive Publication Date: 2026-11-10TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411430198.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-15
Publication Date
2026-11-10
Estimated Expiration
2042-06-15

AI Technical Summary

Technical Problem

Current technologies have not yet effectively improved audio coding efficiency while ensuring audio coding quality.

Method used

A hierarchical coding method is adopted to perform multi-level feature extraction and splicing processing on audio signals. The feature extraction module and the coding module encode the signal features of each level respectively, thereby reducing data dimensionality and improving coding efficiency.

Benefits of technology

By using a layered coding method, information loss during the audio coding process is reduced, coding efficiency is improved, and audio coding quality is guaranteed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119517050B_ABST
    Figure CN119517050B_ABST
Patent Text Reader

Abstract

This application provides an audio encoding method, apparatus, device, storage medium, and computer program product. The method includes: performing feature extraction processing on an audio signal at a first level to obtain signal features at the first level; for the i-th level among N levels, concatenating the audio signal and the signal features of the (i-1)-th level to obtain concatenated features, and performing feature extraction processing on the concatenated features at the i-th level to obtain signal features at the i-th level, where N and i are integers greater than 1, and i is less than or equal to N; traversing i to obtain signal features at each of the N levels, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal; and encoding the signal features at the first level and the signal features at each of the N levels to obtain the bitstream of the audio signal at each level. This application can improve audio encoding efficiency and ensure audio encoding quality.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of patent application No. 202210677636.4, filed on June 15, 2022, entitled "Audio Encoding Method, Apparatus, Device, Storage Medium and Program Product". Technical Field

[0002] This application relates to the field of audio processing technology, and in particular to an audio encoding method, an audio decoding method, an apparatus, a device, a storage medium, and a computer program product. Background Technology

[0003] Audio encoding and decoding technology is a core technology used in communication services, including remote audio and video calls. Audio encoding technology can be understood as transmitting as much voice information as possible with less network bandwidth resources. Audio encoding is a source encoding method. The purpose of source encoding is to compress the amount of data that the user wants to transmit as much as possible at the encoding end, remove redundancy in the information, and at the decoding end, it can be recovered without loss (or with near lossless quality).

[0004] However, there is still no effective solution in terms of how to improve the efficiency of audio encoding while ensuring audio encoding quality. Summary of the Invention

[0005] This application provides an audio encoding method, apparatus, device, storage medium, and computer program product that can improve audio encoding efficiency and ensure audio encoding quality.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides an audio encoding method, including:

[0008] The audio signal is subjected to first-level feature extraction processing to obtain the first-level signal features;

[0009] For the i-th level out of N levels, the audio signal and the signal features of the (i-1)-th level are spliced ​​together to obtain the spliced ​​features.

[0010] The spliced ​​features are subjected to feature extraction processing at the i-th level to obtain the signal features at the i-th level, where N and i are integers greater than 1, and i is less than or equal to N;

[0011] By iterating through i, the signal features of each of the N levels are obtained, where the data dimension of the signal features is smaller than the data dimension of the audio signal.

[0012] The signal features of the first level and the signal features of each of the N levels are encoded to obtain the bitstream of the audio signal at each level.

[0013] This application also provides an audio decoding method, including:

[0014] Receive the bitstreams corresponding to multiple levels obtained by encoding the audio signal;

[0015] The bitstreams of each of the aforementioned levels are decoded to obtain the signal features of each of the aforementioned levels, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal;

[0016] Feature reconstruction is performed on the signal features of each level to obtain the level audio signal of each level;

[0017] The audio signal is obtained by synthesizing the audio signals from multiple levels.

[0018] This application also provides an audio encoding device, including:

[0019] The first feature extraction module is used to perform first-level feature extraction processing on the audio signal to obtain the first-level signal features;

[0020] The second feature extraction module is used to perform splicing processing on the audio signal and the signal features of the (i-1)th level for the i-th level of N levels to obtain spliced ​​features, and to perform feature extraction processing on the spliced ​​features at the i-th level to obtain the signal features of the i-th level, where N and i are integers greater than 1, and i is less than or equal to N;

[0021] The traversal module is used to traverse i to obtain the signal features of each of the N levels, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal;

[0022] The encoding module is used to encode the signal features of the first level and the signal features of each of the N levels respectively to obtain the bitstream of the audio signal at each level.

[0023] In the above scheme, the first feature extraction module is further configured to perform subband decomposition processing on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal; perform first-level feature extraction processing on the low-frequency subband signals to obtain first-level low-frequency signal features, and perform first-level feature extraction processing on the high-frequency subband signals to obtain first-level high-frequency signal features; and use the low-frequency signal features and the high-frequency signal features as the first-level signal features.

[0024] In the above scheme, the first feature extraction module is further configured to sample the audio signal according to the first sampling frequency to obtain a sampled signal; perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, and perform downsampling on the low-pass filtered signal to obtain the low-frequency sub-band signal at the second sampling frequency; perform high-pass filtering on the sampled signal to obtain a high-pass filtered signal, and perform downsampling on the high-pass filtered signal to obtain the high-frequency sub-band signal at the second sampling frequency; wherein, the second sampling frequency is less than the first sampling frequency.

[0025] In the above scheme, the second feature extraction module is further configured to perform splicing processing on the low-frequency sub-band signal of the audio signal and the low-frequency signal features of the (i-1)th level to obtain a first spliced ​​feature, and perform feature extraction processing on the first spliced ​​feature at the i-th level to obtain the low-frequency signal features of the i-th level; perform splicing processing on the high-frequency sub-band signal of the audio signal and the high-frequency signal features of the (i-1)th level to obtain a second spliced ​​feature, and perform feature extraction processing on the second spliced ​​feature at the i-th level to obtain the high-frequency signal features of the i-th level; and use the low-frequency signal features of the i-th level and the high-frequency signal features of the i-th level as the signal features of the i-th level.

[0026] In the above scheme, the first feature extraction module is further configured to perform a first convolution processing on the audio signal to obtain the first level of convolutional features; perform a first pooling processing on the convolutional features to obtain the first level of pooling features; perform a first downsampling processing on the pooling features to obtain the first level of downsampled features; and perform a second convolution processing on the downsampled features to obtain the first level of signal features.

[0027] In the above scheme, the first downsampling process is implemented through M cascaded coding layers. The first feature extraction module is further configured to perform a first downsampling process on the pooled features through the first coding layer in the M cascaded coding layers to obtain the downsampling result of the first coding layer; and to perform a first downsampling process on the downsampling result of the (j-1)th coding layer through the j-th coding layer in the M cascaded coding layers to obtain the downsampling result of the j-th coding layer; wherein M and j are integers greater than 1, and j is less than or equal to M; and the downsampling result of the M-th coding layer is obtained by traversing j, and the downsampling result of the M-th coding layer is used as the downsampling feature of the first level.

[0028] In the above scheme, the second feature extraction module is further configured to perform a third convolution process on the spliced ​​features to obtain the convolutional features of the i-th level; perform a second pooling process on the convolutional features to obtain the pooling features of the i-th level; perform a second downsampling process on the pooling features to obtain the downsampled features of the i-th level; and perform a fourth convolution process on the downsampled features to obtain the signal features of the i-th level.

[0029] In the above scheme, the encoding module is further configured to quantize the signal features of the first level and the signal features of each of the N levels respectively to obtain the quantization results of the signal features of each level; and to perform entropy encoding processing on the quantization results of the signal features of each level to obtain the bitstream of the audio signal at each level.

[0030] In the above scheme, the signal features include low-frequency signal features and high-frequency signal features. The encoding module is further configured to encode the low-frequency signal features of the first level and the low-frequency signal features of each of the N levels respectively to obtain the low-frequency code stream of the audio signal at each level; to encode the high-frequency signal features of the first level and the high-frequency signal features of each of the N levels respectively to obtain the high-frequency code stream of the audio signal at each level; and to use the low-frequency code stream and the high-frequency code stream of the audio signal at each level as the code stream of the audio signal at the corresponding level.

[0031] In the above scheme, the signal features include low-frequency signal features and high-frequency signal features. The encoding module is further configured to encode the low-frequency signal features of the first level according to a first encoding code rate to obtain a first bitstream of the first level, and to encode the high-frequency signal features of the first level according to a second encoding code rate to obtain a second bitstream of the first level. For the signal features of each of the N levels, the following processing is performed: the signal features of the level are encoded according to a third encoding code rate of the level to obtain a second bitstream of each level; the second bitstream of the first level and the second bitstream of each of the N levels are used as the bitstream of the audio signal at each level; wherein, the first encoding code rate is greater than the second encoding code rate, the second encoding code rate is greater than the third encoding code rate of any of the N levels, and the encoding code rate of the level is positively correlated with the decoding quality index of the corresponding level bitstream.

[0032] In the above scheme, the encoding module is further configured to perform the following processing for each of the layers: configure the corresponding layer transmission priority for the bitstream of the audio signal at the layer; wherein the layer transmission priority is negatively correlated with the number of layers of the layer, and the layer transmission priority is positively correlated with the decoding quality index of the bitstream of the corresponding layer.

[0033] In the above scheme, the signal features include low-frequency signal features and high-frequency signal features. The audio signal bitstream at each level includes: a low-frequency bitstream encoded based on the low-frequency signal features and a high-frequency bitstream encoded based on the high-frequency signal features. The encoding module is further configured to perform the following processing for each level: configure a first transmission priority for the low-frequency bitstream of the level and configure a second transmission priority for the high-frequency bitstream of the level. The first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th level is lower than the first transmission priority of the i-th level, and the transmission priority of the bitstream is positively correlated with the decoding quality index of the corresponding bitstream.

[0034] This application also provides an audio decoding device, including:

[0035] The receiving module is used to receive the bitstreams corresponding to multiple levels obtained by encoding the audio signal;

[0036] The decoding module is used to decode the bitstream of each layer to obtain the signal features of each layer, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal;

[0037] The feature reconstruction module is used to reconstruct the signal features of each level to obtain the level audio signal of each level.

[0038] An audio synthesis module is used to synthesize audio signals from multiple levels of the audio signal to obtain the audio signal.

[0039] In the above scheme, the bitstream includes a low-frequency bitstream and a high-frequency bitstream. The decoding module is further configured to decode the low-frequency bitstream of each level to obtain the low-frequency signal features of each level, and to decode the high-frequency bitstream of each level to obtain the high-frequency signal features of each level. Correspondingly, the feature reconstruction module is further configured to reconstruct the low-frequency signal features of each level to obtain the level-specific low-frequency sub-band signal, and to reconstruct the high-frequency signal features of each level to obtain the level-specific high-frequency sub-band signal. The level-specific low-frequency sub-band signal and the level-specific high-frequency sub-band signal are used as the level-specific audio signal of the level. Correspondingly, the audio synthesis module is further configured to add the level-specific low-frequency sub-band signals of multiple levels to obtain the low-frequency sub-band signal, and add the level-specific high-frequency sub-band signals of multiple levels to obtain the high-frequency sub-band signal. The low-frequency sub-band signal and the high-frequency sub-band signal are synthesized to obtain the audio signal.

[0040] In the above scheme, the audio synthesis module is further configured to upsample the low-frequency subband signal to obtain a low-pass filtered signal; upsample the high-frequency subband signal to obtain a high-frequency filtered signal; and filter and synthesize the low-pass filtered signal and the high-frequency filtered signal to obtain the audio signal.

[0041] In the above scheme, the feature reconstruction module is further configured to perform the following processing on the signal features of each level: perform a first convolution processing on the signal features to obtain the convolution features of the level; perform upsampling processing on the convolution features to obtain the upsampled features of the level; perform pooling processing on the upsampled features to obtain the pooled features of the level; and perform a second convolution processing on the pooled features to obtain the level audio signal of the level.

[0042] In the above scheme, the upsampling process is implemented through L cascaded decoding layers. The feature reconstruction module is further used to perform upsampling processing on the pooled features through the first decoding layer in the L cascaded decoding layers to obtain the upsampling result of the first decoding layer; and to perform upsampling processing on the first upsampling result of the (k-1)th decoding layer through the kth decoding layer in the L cascaded decoding layers to obtain the upsampling result of the kth decoding layer; wherein L and k are integers greater than 1, and k is less than or equal to L; and the upsampling result of the Lth decoding layer is obtained by traversing k, and the upsampling result of the Lth decoding layer is used as the upsampling feature of the layer.

[0043] In the above scheme, the decoding module is further configured to perform the following processing for each of the layers: perform entropy decoding processing on the bitstream of the layer to obtain the quantization value of the bitstream; and perform inverse quantization processing on the quantization value of the bitstream to obtain the signal characteristics of the layer.

[0044] This application also provides an electronic device, including:

[0045] Memory, used to store executable instructions;

[0046] A processor, when executing executable instructions stored in the memory, implements the method provided in the embodiments of this application.

[0047] This application also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the method provided in this application.

[0048] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method provided in this application.

[0049] The embodiments of this application have the following beneficial effects:

[0050] A layered encoding method for audio signals is implemented: First, the audio signal undergoes first-level feature extraction processing to obtain first-level signal features; then, for the i-th level (i is an integer greater than 1, i is less than or equal to N) of N levels (N is an integer greater than 1), the audio signal and the signal features of the (i-1)-th level are concatenated to obtain concatenated features, and the concatenated features undergo i-level feature extraction processing to obtain the signal features of the i-th level; then, by traversing i, the signal features of each of the N levels are obtained; finally, the signal features of the first level and the signal features of each of the N levels are encoded separately to obtain the bitstream of the audio signal at each level.

[0051] First, the data dimension of the extracted signal features is smaller than that of the audio signal. This reduces the data dimension of the data processed during audio encoding, thus improving the encoding efficiency of the audio signal.

[0052] Secondly, when extracting signal features from audio signals in a hierarchical manner, the output of each level serves as the input to the next level. This allows each level to combine the signal features extracted from the previous level, resulting in more accurate feature extraction of the audio signal. As the number of levels increases, information loss during feature extraction is minimized. Thus, multiple bitstreams obtained by encoding the signal features extracted in this way contain audio signal information closer to the original audio signal, reducing information loss during encoding and ensuring the quality of audio encoding. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the architecture of the audio encoding system 100 provided in an embodiment of this application;

[0054] Figure 2 This is a schematic diagram of the structure of an electronic device 500 implementing the audio encoding method provided in this application embodiment;

[0055] Figure 3 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application;

[0056] Figure 4 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application;

[0057] Figure 5 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application;

[0058] Figure 6 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application;

[0059] Figure 7 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application;

[0060] Figure 8 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application;

[0061] Figure 9 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application;

[0062] Figure 10 This is a flowchart illustrating the audio decoding method provided in an embodiment of this application;

[0063] Figure 11 This is a flowchart illustrating the audio decoding method provided in an embodiment of this application;

[0064] Figure 12 This is a schematic diagram comparing the spectrum at different code rates provided in the embodiments of this application;

[0065] Figure 13 This is a schematic diagram of the audio encoding and audio decoding process provided in the embodiments of this application;

[0066] Figure 14 This is a schematic diagram of a voice communication link provided in an embodiment of this application;

[0067] Figure 15 This is a schematic diagram of the filter bank provided in an embodiment of this application;

[0068] Figure 16A This is a schematic diagram of a common convolutional network provided in an embodiment of this application;

[0069] Figure 16B This is a schematic diagram of the dilated convolutional network provided in an embodiment of this application;

[0070] Figure 17 This is a schematic diagram of the structure of the first-layer low-frequency analysis neural network model provided in the embodiments of this application;

[0071] Figure 18 This is a schematic diagram of the structure of the second-layer low-frequency analysis neural network model provided in the embodiments of this application;

[0072] Figure 19 This is a schematic diagram of the first-layer low-frequency synthetic neural network model provided in the embodiments of this application;

[0073] Figure 20 This is a schematic diagram of the structure of the second-layer low-frequency synthetic neural network model provided in the embodiments of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0076] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0078] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0079] 1) Client: An application that runs on a terminal and provides various services, such as an instant messaging client or an audio playback client.

[0080] 2) Audio coding is an application of data compression for digital audio signals containing speech.

[0081] 3) Quadrature Mirror Filters (QMF): QMF filter banks are used to decompose subband signals into multiple signals, thereby reducing signal bandwidth. The decomposed signals are filtered through their respective channels.

[0082] 4) Quantization refers to the process of approximating the continuous values ​​(or a large number of possible discrete values) of a signal to a finite number (or a small number) of discrete values, including vector quantization, scalar quantization, etc.

[0083] 5) Vector quantization combines several scalar data into a vector, divides the vector space into several small regions, finds a representative vector for each small region, and replaces the vector that falls into the small region with the corresponding representative vector during quantization, that is, it is quantized into the representative vector.

[0084] 6) Scalar quantization divides the entire dynamic range into several small intervals, each with a representative value. During quantization, the signal value falling into the small interval is replaced by the corresponding representative value, that is, the signal value is quantized into the representative value.

[0085] 7) Entropy coding is a coding method that does not lose any information during the coding process according to the entropy principle. Information entropy is the average amount of information in the source. Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0086] 8) Neural Network (NN): A mathematical model that mimics the behavioral characteristics of animal neural networks to perform distributed parallel information processing. This type of network relies on the complexity of the system and adjusts the relationships between a large number of interconnected nodes to achieve the purpose of information processing.

[0087] 9) Deep Learning (DL): This is a new research direction in the field of Machine Learning (ML). Deep learning learns the inherent patterns and representational layers of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to be able to recognize data such as text, images, and sound.

[0088] This application provides an audio encoding method, an audio decoding method, an apparatus, a device, a storage medium, and a computer program product, which can improve audio encoding efficiency and ensure audio encoding quality.

[0089] The following describes the implementation scenarios of the audio encoding method provided in the embodiments of this application. See also... Figure 1 , Figure 1 This is a schematic diagram of the architecture of the audio encoding system 100 provided in this application embodiment. To support an exemplary application, terminals (terminals 400-1 and 400-2 are shown exemplarily) connect to server 200 through network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both, using wireless or wired links to achieve data transmission. Terminal 400-1 is the audio signal transmitter, and terminal 400-2 is the audio signal receiver.

[0090] During the process of sending audio signals from terminal 400-1 to terminal 400-2 (such as during a remote call between terminal 400-1 and terminal 400-2 based on a configured client), terminal 400-1 performs first-level feature extraction processing on the audio signal to obtain first-level signal features; for the i-th level among N levels, it concatenates the audio signal and the signal features of the (i-1)-th level to obtain concatenated features, and performs i-level feature extraction processing on the concatenated features to obtain the signal features of the i-th level, where N and i are integers greater than 1, and i is less than or equal to N; it iterates through i to obtain the signal features of each of the N levels, where the data dimension of the signal features is smaller than the data dimension of the audio signal; it encodes the first-level signal features and the signal features of each of the N levels to obtain the bitstream of the audio signal at each level; and it sends the bitstream of the audio signal at each level to server 200.

[0091] Server 200 is used to receive the bitstreams corresponding to multiple levels obtained by encoding the audio signal by terminal 400-1; and send the bitstreams corresponding to multiple levels to terminal 400-2.

[0092] Terminal 400-2 is used to receive multiple levels of bitstreams corresponding to the encoded audio signal sent by server 200; decode each level of bitstream to obtain the signal features of each level, where the data dimension of the signal features is smaller than the data dimension of the audio signal; reconstruct the signal features of each level to obtain the level audio signal; and synthesize the multiple levels of level audio signals to obtain the audio signal.

[0093] In some embodiments, the audio encoding method provided in this application can be implemented by various electronic devices. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. For instance, the terminal executes the audio encoding method provided in this application independently, or the terminal sends an encoding request for an audio signal to the server, and the server executes the audio encoding method provided in this application according to the received encoding request. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0094] In some embodiments, the electronic device implementing audio encoding provided in this application can be various types of terminal devices or servers. The server (e.g., server 200) can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. The terminal (e.g., terminal 400) can be a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device (e.g., smart speaker), smart home appliance (e.g., smart TV), smartwatch, vehicle terminal, etc., but is not limited thereto. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this.

[0095] In some embodiments, the audio encoding method provided in this application can be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form a resource pool, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technology network system require a large amount of computing and storage resources. As an example, the aforementioned server (e.g., server 200) can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0096] In some embodiments, the terminal or server can implement the audio encoding method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run; it can also be a mini-program, that is, a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0097] In some embodiments, multiple servers can form a blockchain, with each server being a node on the blockchain. Information connections can exist between each node in the blockchain, allowing for information transmission between nodes. Furthermore, data related to the audio encoding method provided in this application (e.g., audio signal bitstreams at various levels, neural network models for feature extraction) can be stored on the blockchain.

[0098] The following describes an electronic device implementing the audio encoding method provided in an embodiment of this application. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 implementing the audio encoding method provided in an embodiment of this application. The electronic device 500 is used as an example. Figure 1 Taking the terminal shown (such as terminal 400-1) as an example, the electronic device 500 implementing the audio encoding method provided in this application embodiment includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0099] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0100] Memory 550 may be removable, non-removable, or a combination thereof. Memory 550 may optionally include one or more storage devices physically located away from processor 510. Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0101] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0102] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0103] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0104] In some embodiments, the audio encoding apparatus provided in this application can be implemented in software. Figure 2 An audio encoding device 553 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: a first feature extraction module 5531, a second feature extraction module 5532, a traversal module 5533, and an encoding module 5534. These modules are logical and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0105] The following describes the audio encoding method provided in the embodiments of this application. In some embodiments, the audio encoding method provided in the embodiments of this application can be implemented by various electronic devices. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. Taking terminal implementation as an example, see [link to documentation]. Figure 3 , Figure 3 This is a flowchart illustrating the audio encoding method provided in this application embodiment. The audio encoding method provided in this application embodiment includes:

[0106] Step 101: The terminal performs first-level feature extraction processing on the audio signal to obtain the first-level signal features.

[0107] In practical applications, the audio signal can be voice signals during calls (such as internet calls or telephone calls), voice messages (such as voice messages sent in instant messaging clients), played music, audio, etc. Audio signals need to be encoded during transmission so that the transmitting end can transmit the encoded bitstream, and the receiving end can decode the received bitstream to obtain the audio signal. The audio signal encoding process will be described below. In this embodiment, a layered encoding method is used to encode the audio signal. This layered encoding method is implemented by encoding the audio signal at multiple levels. The encoding process for each level is described below. First, for the first level, the terminal can perform first-level feature extraction processing on the audio signal to obtain the signal features extracted from the audio signal at the first level, i.e., the first-level signal features.

[0108] In some embodiments, the audio signal includes a low-frequency sub-band signal and a high-frequency sub-band signal. When processing the audio signal (such as feature extraction processing or encoding processing), the low-frequency sub-band signal and the high-frequency sub-band signal included in the audio signal can be processed separately. Based on this, see [link to relevant documentation]. Figure 4 , Figure 4 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application. Figure 4 Show Figure 3 Step 101 can be implemented through steps 201-203: Step 201, perform subband decomposition processing on the audio signal to obtain the low-frequency subband signal and the high-frequency subband signal of the audio signal; Step 202, perform first-level feature extraction processing on the low-frequency subband signal to obtain the first-level low-frequency signal features, and perform first-level feature extraction processing on the high-frequency subband signal to obtain the first-level high-frequency signal features; Step 203, use the low-frequency signal features and the high-frequency signal features as the first-level signal features.

[0109] It should be noted that in step 201, during the feature extraction of the audio signal through the first level, the terminal can first perform sub-band decomposition processing on the audio signal to obtain the low-frequency sub-band signal and the high-frequency sub-band signal, and then perform feature extraction on the low-frequency sub-band signal and the high-frequency sub-band signal respectively. In some embodiments, see... Figure 5 , Figure 5 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application. Figure 5 Show Figure 4Step 201 can be implemented through steps 2011-2013: Step 2011, sample the audio signal according to the first sampling frequency to obtain a sampled signal; Step 2012, perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, and then perform downsampling on the low-pass filtered signal to obtain a low-frequency sub-band signal at the second sampling frequency; Step 2013, perform high-pass filtering on the sampled signal to obtain a high-pass filtered signal, and then perform downsampling on the high-pass filtered signal to obtain a high-frequency sub-band signal at the second sampling frequency. The second sampling frequency is less than the first sampling frequency.

[0110] In step 2011, the audio signal can be sampled at a first sampling frequency to obtain a sampled signal. This first sampling frequency can be preset. In practical applications, the audio signal is a continuous analog signal. By sampling the audio signal using the first sampling frequency, a discrete digital signal, i.e., a sampled signal, is obtained. This sampled signal includes multiple sample points (i.e., sampled values) obtained from the audio signal.

[0111] In step 2012, the sampled signal is low-pass filtered to obtain a low-pass filtered signal, and then downsampled to obtain a low-frequency subband signal at the second sampling frequency. In step 2013, the sampled signal is high-pass filtered to obtain a high-pass filtered signal, and then downsampled to obtain a high-frequency subband signal at the second sampling frequency. In steps 202 and 203, the low-pass and high-pass filtering processes can be implemented using a QMF analysis filter. In practical implementation, the second sampling frequency can be half of the first sampling frequency, thus obtaining low-frequency and high-frequency subband signals with the same frequency.

[0112] In step 202, after obtaining the low-frequency sub-band signal and the high-frequency sub-band signal of the audio signal, the low-frequency sub-band signal is subjected to first-level feature extraction processing to obtain first-level low-frequency signal features, and the high-frequency sub-band signal is subjected to first-level feature extraction processing to obtain first-level high-frequency signal features. In step 203, the low-frequency signal features and the high-frequency signal features are used as the first-level signal features.

[0113] In some embodiments, see Figure 6 , Figure 6 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application. Figure 6 Show Figure 3Step 101 can also be implemented through steps 301-304: Step 301, perform a first convolution process on the audio signal to obtain the first level of convolution features; Step 302, perform a first pooling process on the convolution features to obtain the first level of pooling features; Step 303, perform a first downsampling process on the pooling features to obtain the first level of downsampling features; Step 304, perform a second convolution process on the downsampling features to obtain the first level of signal features.

[0114] It should be noted that in step 301, the audio signal can undergo a first convolution process. In practical applications, this first convolution process can be performed by calling a causal convolution with a preset number of channels (such as 24 channels) to obtain the first-level convolutional features.

[0115] In step 302, the convolutional features obtained in step 301 are subjected to a first pooling process. In practical applications, this first pooling process can be performed by pre-setting a pooling factor (e.g., 2), and then the convolutional features are subjected to the first pooling process based on this pooling factor to obtain the first-level pooled features.

[0116] In step 303, the pooling features obtained in step 302 undergo a first downsampling process. In practical applications, a downsampling factor can be preset, and downsampling processing can be performed based on this downsampling factor. This first downsampling process can be implemented using one coding layer or multiple coding layers. In some embodiments, the first downsampling process is implemented using M cascaded coding layers. Accordingly, see [link to relevant documentation]. Figure 7 , Figure 7 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application. Figure 7 Show Figure 6 Step 303 can also be implemented through steps 3031-3033: Step 3031, the pooling feature is downsampled through the first coding layer in the M cascaded coding layers to obtain the downsampling result of the first coding layer; Step 3032, the downsampling result of the (j-1)th coding layer is downsampled through the jth coding layer in the M cascaded coding layers to obtain the downsampling result of the jth coding layer; where M and j are integers greater than 1, and j is less than or equal to M; Step 3033, j is iterated to obtain the downsampling result of the Mth coding layer, and the downsampling result of the Mth coding layer is used as the downsampling feature of the first level.

[0117] It should be noted that in steps 3031-3033, the downsampling factor for each coding layer can be the same or different. In practical applications, the downsampling factor is equivalent to the pooling factor, serving to reduce sampling.

[0118] In step 304, a second convolution process can be performed on the downsampled features. In practical applications, this second convolution process can be performed by calling a causal convolution with a preset number of channels to obtain the signal features of the first level.

[0119] In practical applications, Figure 6 Steps 301-304 shown can be implemented by calling a first neural network model, which includes a first convolutional layer, a pooling layer, a downsampling layer, and a second convolutional layer. Thus, by calling the first convolutional layer to perform a first convolutional process on the audio signal, first-level convolutional features are obtained; by calling the pooling layer to perform a first pooling process on the convolutional features, first-level pooled features are obtained; by calling the downsampling layer to perform a first downsampling process on the pooled features, first-level downsampling features are obtained; and by calling the second convolutional layer to perform a second convolutional process on the downsampling features, first-level signal features are obtained.

[0120] It should be noted that, during the first-level feature extraction of audio signals, it is also possible to... Figure 6 Steps 301-304 shown involve performing first-level feature extraction processing on the low-frequency subband signal and the high-frequency subband signal of the audio signal respectively (i.e. Figure 4 Step 202 is shown. Specifically, the low-frequency sub-band signal of the audio signal undergoes a first convolution process to obtain a first-level first convolution feature; the first convolution feature undergoes a first pooling process to obtain a first-level first pooling feature; the first pooling feature undergoes a first downsampling process to obtain a first-level first downsampling feature; and the first downsampling feature undergoes a second convolution process to obtain a first-level low-frequency signal feature. Similarly, the high-frequency sub-band signal of the audio signal undergoes a first convolution process to obtain a first-level second convolution feature; the second convolution feature undergoes a first pooling process to obtain a first-level second pooling feature; the second pooling feature undergoes a first downsampling process to obtain a first-level second downsampling feature; and the second downsampling feature undergoes a second convolution process to obtain a first-level high-frequency signal feature.

[0121] Step 102: For the i-th level among the N levels, the audio signal and the signal features of the (i-1)-th level are spliced ​​together to obtain spliced ​​features, and the spliced ​​features are then subjected to feature extraction processing of the i-th level to obtain the signal features of the i-th level.

[0122] Where N and i are integers greater than 1, and i is less than or equal to N.

[0123] After performing the first-level feature extraction processing on the audio signal, the remaining levels of feature extraction processing can be performed on the audio signal. In this embodiment, the remaining levels include N levels. For the i-th level among the N levels, the audio signal and the signal features of the (i-1)-th level are spliced ​​together to obtain spliced ​​features, and the spliced ​​features are then subjected to the i-th level feature extraction processing to obtain the i-th level signal features. For example, for the second level, the audio signal and the signal features of the first level are spliced ​​together to obtain spliced ​​features, and the spliced ​​features are then subjected to the second-level feature extraction processing to obtain the second-level signal features; for the third level, the audio signal and the signal features of the second level are spliced ​​together to obtain spliced ​​features, and the spliced ​​features are then subjected to the third-level feature extraction processing to obtain the third-level signal features; for the fourth level, the audio signal and the signal features of the third level are spliced ​​together to obtain spliced ​​features, and the spliced ​​features are then subjected to the fourth-level feature extraction processing to obtain the fourth-level signal features, and so on.

[0124] In some embodiments, the audio signal includes low-frequency subband signals and high-frequency subband signals. When processing the audio signal (such as feature extraction processing or encoding processing), the low-frequency subband signals and high-frequency subband signals included in the audio signal can be processed separately. Based on this, for the i-th level in N levels, subband decomposition processing can also be performed on the audio signal to obtain the low-frequency subband signals and high-frequency subband signals of the audio signal. The subband decomposition process can be found in steps 2011-2013 above. Thus, for the i-th level in N levels, the data output by performing feature extraction processing includes: the low-frequency signal features of the i-th level and the high-frequency signal features of the i-th level.

[0125] Accordingly, see Figure 8 , Figure 8 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application. Figure 8 Show Figure 3 Step 102 can be implemented through steps 401-403: Step 401, splice the low-frequency subband signal of the audio signal and the low-frequency signal features of the (i-1)th level to obtain the first spliced ​​feature, and perform feature extraction processing of the first spliced ​​feature at the i-th level to obtain the low-frequency signal features of the i-th level; Step 402, splice the high-frequency subband signal of the audio signal and the high-frequency signal features of the (i-1)th level to obtain the second spliced ​​feature, and perform feature extraction processing of the second spliced ​​feature at the i-th level to obtain the high-frequency signal features of the i-th level; Step 403, use the low-frequency signal features of the i-th level and the high-frequency signal features of the i-th level as the signal features of the i-th level.

[0126] It should be noted that in step 401, after obtaining the low-frequency sub-band signal and the high-frequency sub-band signal of the audio signal, the low-frequency sub-band signal of the audio signal and the low-frequency signal features extracted at level (i-1) are spliced ​​together to obtain the first spliced ​​feature. Then, the first spliced ​​feature is subjected to feature extraction processing at level i to obtain the low-frequency signal features at level i. Similarly, in step 402, the high-frequency sub-band signal of the audio signal and the high-frequency signal features extracted at level (i-1) are spliced ​​together to obtain the second spliced ​​feature. Then, the second spliced ​​feature is subjected to feature extraction processing at level i to obtain the high-frequency signal features at level i. Thus, in step 403, the low-frequency signal features at level i and the high-frequency signal features at level i are used as the signal features at level i.

[0127] In some embodiments, see Figure 9 , Figure 9 This is a flowchart illustrating the audio encoding method provided in an embodiment of this application. Figure 9 Show Figure 3 Step 102 can also be implemented through steps 501-504: Step 501, perform a third convolution on the spliced ​​features to obtain the convolutional features of the i-th level; Step 502, perform a second pooling on the convolutional features to obtain the pooling features of the i-th level; Step 503, perform a second downsampling on the pooling features to obtain the downsampled features of the i-th level; Step 504, perform a fourth convolution on the downsampled features to obtain the signal features of the i-th level.

[0128] It should be noted that in step 501, a third convolution process can be performed on the spliced ​​features (obtained by splicing the audio signal and the signal features of the (i-1)th level). In practical applications, this third convolution process can be performed by calling a causal convolution with a preset number of channels to obtain the convolutional features of the i-th level.

[0129] In step 502, the convolutional features obtained in step 501 are subjected to a second pooling process. In practical applications, this second pooling process can be performed by pre-setting a pooling factor, and then performing the second pooling process on the convolutional features based on the pooling factor to obtain the pooled features of the i-th layer.

[0130] In step 503, a second downsampling process is performed on the pooling features obtained in step 502. In practical applications, a downsampling factor can be preset, and downsampling processing can be performed based on this downsampling factor. This second downsampling process can be implemented through one coding layer or multiple coding layers. In some embodiments, the second downsampling process can be implemented through X cascaded coding layers. Accordingly, Figure 9Step 503 can also be implemented through steps 5031-5033: Step 5031, the pooling features are subjected to a second downsampling process through the first coding layer in the X cascaded coding layers to obtain the downsampling result of the first coding layer; Step 5032, the downsampling result of the (g-1)th coding layer is subjected to a second downsampling process through the g-th coding layer in the X cascaded coding layers to obtain the downsampling result of the g-th coding layer; where X and g are integers greater than 1, and g is less than or equal to X; Step 5033, g is traversed to obtain the downsampling result of the X-th coding layer, and the downsampling result of the X-th coding layer is used as the downsampling feature of the i-th level.

[0131] It should be noted that in steps 5031-5033, the downsampling factor for each coding layer can be the same or different. In practical applications, the downsampling factor is equivalent to the pooling factor, serving to reduce sampling.

[0132] In step 504, a fourth convolution process can be performed on the downsampled features. In practical applications, this fourth convolution process can be performed by calling a causal convolution with a preset number of channels to obtain the signal features of the i-th level.

[0133] In practical applications, Figure 9 Steps 501-504 shown can be implemented by calling a second neural network model, which includes a third convolutional layer, a pooling layer, a downsampling layer, and a fourth convolutional layer. Thus, the convolutional features of the i-th level can be obtained by calling the third convolutional layer to perform a third convolutional process on the concatenated features; the pooling layer can perform a second pooling process on the convolutional features to obtain the pooled features of the i-th level; the downsampling layer can perform a second downsampling process on the pooled features to obtain the downsampled features of the i-th level; and the fourth convolutional layer can perform a fourth convolutional process on the downsampled features to obtain the signal features of the i-th level. In actual implementation, the feature dimension of the signal features output by the second neural network can be less than the feature dimension of the signal features input by the first neural network.

[0134] It should be noted that when performing feature extraction at the i-th level, it is also possible to... Figure 9Steps 501-504, as shown, involve performing feature extraction processing at the i-th level on the low-frequency sub-band signal and the high-frequency sub-band signal of the audio signal, respectively. Specifically, for the i-th level, the low-frequency spliced ​​features (obtained by splicing the low-frequency sub-band signal and the low-frequency signal features of the (i-1)-th level) undergo a third convolution process to obtain the convolutional features of the i-th level; the convolutional features undergo a second pooling process to obtain the pooled features of the i-th level; the pooled features undergo a second downsampling process to obtain the downsampled features of the i-th level; and the downsampled features undergo a fourth convolution process to obtain the low-frequency signal features of the i-th level. For the i-th level, the high-frequency splicing features (obtained by splicing the high-frequency sub-band signal and the high-frequency signal features of the (i-1)-th level) are subjected to a third convolution process to obtain the convolution features of the i-th level; the convolution features are subjected to a second pooling process to obtain the pooling features of the i-th level; the pooling features are subjected to a second downsampling process to obtain the downsampling features of the i-th level; and the downsampling features are subjected to a fourth convolution process to obtain the high-frequency signal features of the i-th level.

[0135] Step 103: Iterate through i to obtain the signal characteristics of each of the N levels.

[0136] The data dimension of signal features is smaller than that of audio signals.

[0137] Step 102 describes the feature extraction process for the i-th level. In practical applications, it is necessary to traverse i to obtain the signal features of each of the N levels. In this embodiment, the data dimension of the signal features output by each level is smaller than the data dimension of the audio signal. This reduces the data dimension of the data involved in the audio encoding process and improves the encoding efficiency of audio encoding.

[0138] Step 104: Encode the signal features of the first level and the signal features of each of the N levels to obtain the bitstream of the audio signal at each level.

[0139] In practical applications, after obtaining the signal features of the first level and the signal features of each of the N levels, the signal features of the first level and the signal features of each of the N levels can be encoded separately to obtain the audio signal bitstream at each level. This bitstream can be transmitted to the audio signal receiving end, allowing the receiving end to act as a decoding end to decode the audio signal.

[0140] It should be noted that the signal features output by the i-th level in these N levels can be understood as the residual signal features between the signal features output by the (i-1)-th level and the original audio signal. Thus, the extracted audio signal features include not only the signal features extracted by the first level, but also the residual signal features extracted by each of the N levels. This makes the extracted audio signal features more comprehensive and accurate, reducing information loss during feature extraction. Consequently, when encoding the signal features of the first level and the signal features of each of the N levels, the quality of the encoded bitstream is higher, and the information contained in the audio signal is closer to the original audio signal, thus improving the encoding quality of the audio signal.

[0141] In some embodiments, Figure 3 Step 104 shown can be implemented through steps 104a1-104a2: Step 104a1, the signal features of the first level and the signal features of each of the N levels are quantized to obtain the quantization results of the signal features of each level; Step 104a2, the quantization results of the signal features of each level are entropy encoded to obtain the bitstream of the audio signal at each level.

[0142] It should be noted that in step 104a1, a quantization table can be pre-set, which includes the correspondence between signal features and quantization values. During quantization processing, the pre-set quantization table can be consulted to find the corresponding quantization values ​​for the signal features of the first level and for the signal features of each of the N levels, and the quantization values ​​obtained are used as the quantization results. In step 104a2, the quantization results of the signal features of each level are subjected to entropy encoding processing to obtain the audio signal bitstream at each level.

[0143] In practical applications, audio signals include low-frequency sub-band signals and high-frequency sub-band signals. Therefore, the signal characteristics output at each level correspondingly include low-frequency signal characteristics and high-frequency signal characteristics. Based on this, when the signal characteristics include both low-frequency and high-frequency signal characteristics, in some embodiments... Figure 3 Step 104 shown can also be implemented through steps 104b1-104b3: Step 104b1, encode the low-frequency signal features of the first level and the low-frequency signal features of each of the N levels to obtain the low-frequency bitstream of the audio signal at each level; Step 104b2, encode the high-frequency signal features of the first level and the high-frequency signal features of each of the N levels to obtain the high-frequency bitstream of the audio signal at each level; Step 104b3, use the low-frequency bitstream and high-frequency bitstream of the audio signal at each level as the bitstream of the audio signal at the corresponding level.

[0144] It should be noted that the encoding process of low-frequency signal features in step 104b1 can also be implemented using steps similar to steps 104a1-104a2. That is, the low-frequency signal features of the first level and the low-frequency signal features of each of the N levels are quantized separately to obtain the quantization results of the low-frequency signal features of each level; the quantization results of the low-frequency signal features of each level are then subjected to entropy coding to obtain the low-frequency bitstream of the audio signal at each level. Similarly, the encoding process of high-frequency signal features in step 104b2 can also be implemented using steps similar to steps 104a1-104a2. That is, the high-frequency signal features of the first level and the high-frequency signal features of each of the N levels are quantized separately to obtain the quantization results of the high-frequency signal features of each level; the quantization results of the high-frequency signal features of each level are then subjected to entropy coding to obtain the high-frequency bitstream of the audio signal at each level.

[0145] In practical applications, audio signals include low-frequency sub-band signals and high-frequency sub-band signals. Therefore, the signal characteristics output at each level correspondingly include low-frequency signal characteristics and high-frequency signal characteristics. Based on this, when the signal characteristics include both low-frequency and high-frequency signal characteristics, in some embodiments... Figure 3 Step 104 shown can also be implemented through steps 104c1-104c3: Step 104c1, according to the first coding rate, the low-frequency signal features of the first level are encoded to obtain the first bitstream of the first level, and according to the second coding rate, the high-frequency signal features of the first level are encoded to obtain the second bitstream of the first level; Step 104c2, for the signal features of each of the N levels, the following processing is performed respectively: according to the third coding rate of the level, the signal features of the level are encoded to obtain the second bitstream of each level; Step 104c3, the second bitstream of the first level and the second bitstream of each of the N levels are used as the bitstream of the audio signal at each level.

[0146] It should be noted that the first encoding bitrate is greater than the second encoding bitrate, the second encoding bitrate is greater than the third encoding bitrate of any of the N levels, and the encoding bitrate of a level is positively correlated with the decoding quality index of the corresponding level's bitstream. In step 104c2, a corresponding third encoding bitrate can be set for each of the N levels. The third encoding bitrates of each of the N levels can be the same, partially the same and partially different, or completely different. Here, the coding bitrate of a level is positively correlated with the decoding quality index of the corresponding level's bitstream. That is, the higher the coding bitrate, the higher the decoding quality index (value) of the resulting bitstream. Since the low-frequency signal features of the first level contain the most audio signal features, the first coding bitrate used for the low-frequency signal features of the first level is the highest to ensure the encoding effect of the audio signal. At the same time, for the high-frequency signal features of the first level, a second coding bitrate lower than the first coding bitrate is used for encoding, and for the signal features of each of the N levels, a third coding bitrate lower than the second coding bitrate is used for encoding. While increasing more features of the audio signal (including high-frequency signal features and residual signal features), the encoding efficiency of the audio signal is improved by reasonably allocating the coding bitrate of each level.

[0147] In some embodiments, after obtaining the audio signal bitstream at each level, the terminal may perform the following processing for each level: configure the corresponding level transmission priority for the audio signal bitstream at each level; wherein, the level transmission priority is negatively correlated with the number of levels and positively correlated with the decoding quality index of the corresponding level bitstream.

[0148] It should be noted that the layer transmission priority is used to characterize the transmission priority of the bitstream at that layer. The layer transmission priority is negatively correlated with the number of layers; that is, the higher the layer number, the lower the corresponding layer transmission priority. For example, the layer transmission priority of the first layer (layer number 1) is higher than that of the second layer (layer number 2). Based on this, when transmitting the bitstreams of each layer to the decoding end, the corresponding layer's bitstream can be transmitted according to the configured layer transmission priority. In practical applications, when transmitting audio signals at multiple layers to the decoding end, some layers or all layers of bitstreams can be transmitted. When transmitting only some layers of bitstreams, the configured layer transmission priority can be used to transmit the corresponding layer's bitstream.

[0149] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features. The audio signal bitstream at each level includes: a low-frequency bitstream obtained by encoding based on low-frequency signal features and a high-frequency bitstream obtained by encoding based on high-frequency signal features. After obtaining the audio signal bitstream at each level, the terminal can also perform the following processing for each level: configure a first transmission priority for the low-frequency bitstream of the level and configure a second transmission priority for the high-frequency bitstream of the level; wherein, the first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th level is lower than the first transmission priority of the i-th level, and the transmission priority of the bitstream is positively correlated with the decoding quality index of the corresponding bitstream.

[0150] It should be noted that since the transmission priority of a bitstream is positively correlated with its corresponding decoding quality index, and since the data dimension of a high-frequency bitstream is smaller than that of a low-frequency bitstream, the original audio signal information contained in the low-frequency bitstream at each level is greater than that contained in the high-frequency bitstream. In other words, to ensure that the decoding quality index of the low-frequency bitstream is higher than that of the high-frequency bitstream, a first transmission priority can be configured for the low-frequency bitstream at each level, and a second transmission priority can be configured for the high-frequency bitstream at each level. The first transmission priority is higher than the second transmission priority. Simultaneously, the second transmission priority of the (i-1)th level can be configured to be lower than the first transmission priority of the ith level. That is, for each level, the transmission priority of the low-frequency bitstream is higher than that of the high-frequency bitstream, thus ensuring that the low-frequency bitstream at each level can be transmitted preferentially. For multiple levels, the transmission priority of the low-frequency bitstream at the ith level is higher than that of the high-frequency bitstream at the (i-1)th level, thus ensuring that all low-frequency bitstreams across multiple levels can be transmitted preferentially.

[0151] By applying the above embodiments of this application, layered encoding of audio signals is realized: First, the audio signal is subjected to feature extraction processing of the first level to obtain the signal features of the first level; then, for the i-th level (i is an integer greater than 1, i is less than or equal to N) of the N (N is an integer greater than 1) levels, the audio signal and the signal features of the (i-1)-th level are concatenated to obtain concatenated features, and the concatenated features are subjected to feature extraction processing of the i-th level to obtain the signal features of the i-th level; then, by traversing i, the signal features of each of the N levels are obtained; finally, the signal features of the first level and the signal features of each of the N levels are encoded respectively to obtain the bitstream of the audio signal at each level.

[0152] First, the data dimension of the extracted signal features is smaller than that of the audio signal. This reduces the data dimension of the data processed during audio encoding, thus improving the encoding efficiency of the audio signal.

[0153] Secondly, when extracting signal features from audio signals in a hierarchical manner, the output of each level serves as the input to the next level. This allows each level to combine the signal features extracted from the previous level, resulting in more accurate feature extraction of the audio signal. As the number of levels increases, information loss during feature extraction is minimized. Thus, multiple bitstreams obtained by encoding the signal features extracted in this way contain audio signal information closer to the original audio signal, reducing information loss during encoding and ensuring the quality of audio encoding.

[0154] The following describes the audio decoding method provided in the embodiments of this application. In some embodiments, the audio decoding method provided in the embodiments of this application can be implemented by various electronic devices. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. Taking a terminal implementation as an example, see [link to documentation]. Figure 10 , Figure 10 This is a flowchart illustrating the audio decoding method provided in this application embodiment. The audio decoding method provided in this application embodiment includes:

[0155] Step 601: The terminal receives the bitstreams corresponding to the multiple levels obtained by encoding the audio signal.

[0156] Here, the terminal acts as the decoding end, receiving the bitstreams corresponding to multiple levels obtained by encoding the audio signal.

[0157] Step 602: Decode the bitstream of each level to obtain the signal characteristics of each level.

[0158] The data dimension of signal features is smaller than that of audio signals.

[0159] In some embodiments, the terminal may decode the bitstream of each level in the following manner to obtain the signal characteristics of each level: For each level, the following processing is performed: entropy decoding is performed on the bitstream of the level to obtain the quantized value of the bitstream; inverse quantization is performed on the quantized value of the bitstream to obtain the signal characteristics of the level.

[0160] In practical applications, the following processing can be performed on the bitstream at each level: entropy decoding is performed on the bitstream at that level to obtain the quantized value of the bitstream; then, based on the quantization table used in the process of encoding the audio signal to obtain the bitstream, the quantized value of the bitstream is dequantized, that is, the signal characteristics corresponding to the quantized value of the bitstream are queried through the quantization table to obtain the signal characteristics of that level.

[0161] In practical applications, the received bitstreams at each level can include low-frequency and high-frequency bitstreams. The low-frequency bitstream is obtained by encoding the low-frequency signal characteristics of the audio signal, and the high-frequency bitstream is obtained by encoding the high-frequency signal characteristics of the audio signal. Therefore, when decoding the bitstreams at each level, the low-frequency and high-frequency bitstreams at each level can be decoded separately. The decoding process for the high-frequency and low-frequency bitstreams is similar to the decoding process for the entire bitstream; that is, for each level of low-frequency bitstream, the following processing is performed: entropy decoding is performed on the low-frequency bitstream at that level to obtain the quantized value of the low-frequency bitstream; the quantized value of the low-frequency bitstream is then dequantized to obtain the low-frequency signal characteristics of that level. Similarly, for each level of high-frequency bitstream, the following processing is performed: entropy decoding is performed on the high-frequency bitstream at that level to obtain the quantized value of the high-frequency bitstream; the quantized value of the high-frequency bitstream is then dequantized to obtain the high-frequency signal characteristics of that level.

[0162] Step 603: Perform feature reconstruction on the signal features of each level to obtain the level audio signal of each level.

[0163] In practical applications, after decoding the signal features of each level, feature reconstruction is performed on the signal features of each level to obtain the layered audio signals of each level. In some embodiments, the terminal can perform feature reconstruction on the signal features of each level to obtain the layered audio signals of each level in the following manner: For the signal features of each level, the following processing is performed respectively: Perform a first convolution processing on the signal features to obtain the convolutional features of the level; perform upsampling processing on the convolutional features to obtain the upsampled features of the level; perform pooling processing on the upsampled features to obtain the pooled features of the level; perform a second convolution processing on the pooled features to obtain the layered audio signals of the level.

[0164] In practical applications, the following processing is performed on the signal features at each level: First, a first convolution process is applied to the signal features. This first convolution process can be performed by calling a causal convolution with a preset number of channels to obtain the convolutional features for that level. Then, the convolutional features are upsampled. An upsampling factor can be preset, and upsampling is performed based on this factor to obtain the upsampled features for that level. Next, the upsampled features are pooled. This pooling process can be preset, and pooling is performed based on this factor to obtain the pooled features for that level. Finally, the pooled features are subjected to a second convolution process. This second convolution process can be performed by calling a causal convolution with a preset number of channels to obtain the layered audio signal for that level.

[0165] This upsampling process can be implemented through a single decoding layer or multiple decoding layers. When the upsampling process can be implemented through L (L>1) cascaded decoding layers, the terminal can perform upsampling on the convolutional features to obtain the layer-level upsampling features as follows: Upsample the pooling features through the first decoding layer in the L cascaded decoding layers to obtain the upsampling result of the first decoding layer; Upsample the first upsampling result of the (k-1)th decoding layer through the kth decoding layer in the L cascaded decoding layers to obtain the upsampling result of the kth decoding layer; where L and k are integers greater than 1, and k is less than or equal to L; Iterate through k to obtain the upsampling result of the Lth decoding layer, and use the upsampling result of the Lth decoding layer as the layer-level upsampling feature.

[0166] It should be noted that the upsampling factor for each decoding layer can be the same or different.

[0167] Step 604: Perform audio synthesis on the hierarchical audio signals of multiple levels to obtain an audio signal.

[0168] In practical applications, after obtaining the audio signals of each level, the audio signals of multiple levels are synthesized to obtain the audio signal.

[0169] In some embodiments, the bitstream includes a low-frequency bitstream and a high-frequency bitstream. Figure 10 Step 602 shown can be implemented through the following steps: decoding the low-frequency code streams of each level to obtain the low-frequency signal characteristics of each level, and decoding the high-frequency code streams of each level to obtain the high-frequency signal characteristics of each level; correspondingly, Figure 10 Step 603 shown can be implemented through the following steps: Step 6031, perform feature reconstruction on the low-frequency signal features of each level to obtain the level low-frequency sub-band signal of each level, and perform feature reconstruction on the high-frequency signal features of each level to obtain the level high-frequency sub-band signal of each level; Step 6032, use the level low-frequency sub-band signal and the level high-frequency sub-band signal as the level audio signal of the level; accordingly, Figure 10 Step 604 shown can be implemented by the following steps: Step 6041, add the low-frequency subband signals of multiple levels to obtain a low-frequency subband signal, and add the high-frequency subband signals of multiple levels to obtain a high-frequency subband signal; Step 6042, synthesize the low-frequency subband signal and the high-frequency subband signal to obtain an audio signal.

[0170] In some embodiments, step 6042 can be implemented through the following steps: Step 60421, upsampling the low-frequency subband signal to obtain a low-pass filtered signal; Step 60422, upsampling the high-frequency subband signal to obtain a high-frequency filtered signal; Step 60423, filtering and combining the low-pass filtered signal and the high-frequency filtered signal to obtain an audio signal. It should be noted that in step 60423, a QMF synthesis filter can be used to perform the synthesis process to obtain the audio signal.

[0171] Therefore, when the bitstream includes both low-frequency and high-frequency bitstreams, see [reference needed]. Figure 11 , Figure 11 This is a flowchart illustrating an audio decoding method provided in an embodiment of this application. The audio decoding method provided in this application includes: Step 701, receiving low-frequency and high-frequency bitstreams corresponding to multiple levels obtained by encoding an audio signal; Step 702a, decoding the low-frequency bitstreams of each level to obtain low-frequency signal features of each level; Step 702b, decoding the high-frequency bitstreams of each level to obtain high-frequency signal features of each level; Step 703a, reconstructing features from the low-frequency signal features of each level to obtain the low-frequency sub-band signal of each level; Step 703b... Step 704a: Feature reconstruction is performed on the high-frequency signal features of each level to obtain the high-frequency sub-band signal of each level; Step 704b: The low-frequency sub-band signals of multiple levels are added together to obtain the low-frequency sub-band signal; Step 705a: The low-frequency sub-band signal is upsampled to obtain the low-pass filtered signal; Step 705b: The high-frequency sub-band signal is upsampled to obtain the high-frequency filtered signal; Step 706: The low-pass filtered signal and the high-frequency filtered signal are filtered and synthesized to obtain the audio signal.

[0172] It should be noted that the feature reconstruction process for high-frequency and low-frequency signal features can refer to the feature reconstruction process for signal features in step 603. Specifically, for each level of high-frequency signal features, the following processing is performed: First convolution processing is applied to the high-frequency signal features to obtain the level's high-frequency convolutional features; upsampling processing is applied to the high-frequency convolutional features to obtain the level's high-frequency upsampled features; pooling processing is applied to the high-frequency upsampled features to obtain the level's high-frequency pooled features; and second convolution processing is applied to the high-frequency pooled features to obtain the level's high-frequency level audio signal. For each level of low-frequency signal features, the following processing is performed: First convolution processing is applied to the low-frequency signal features to obtain the level's low-frequency convolutional features; upsampling processing is applied to the low-frequency convolutional features to obtain the level's low-frequency upsampled features; pooling processing is applied to the low-frequency upsampled features to obtain the level's low-frequency pooled features; and second convolution processing is applied to the low-frequency pooled features to obtain the level's low-frequency level audio signal.

[0173] By applying the embodiments described above, multiple levels of bitstreams are decoded separately to obtain signal features at each level. Feature reconstruction is then performed on these signal features to obtain the level-specific audio signals. Finally, the level-specific audio signals from multiple levels are synthesized to obtain the final audio signal. Since the data dimension of the signal features in the bitstream is smaller than that of the audio signal, compared to the bitstream obtained by directly encoding the original audio signal in related technologies, the data dimension of the data processed during audio decoding is reduced, thus improving the decoding efficiency of the audio signal.

[0174] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0175] Audio codec technology aims to transmit as much voice information as possible with less network bandwidth. Audio codecs can achieve compression rates of over 10 times, meaning that 10MB of voice data can be compressed to only 1MB for transmission, significantly reducing the bandwidth required for information transmission. In communication systems, to ensure smooth communication, the industry deploys standard voice codec protocols, such as those from international and domestic standards organizations like ITU-T, 3GPP, IETF, AVS, and CCSA, including standards like G.711, G.722, AMR series, EVS, and OPUS. Figure 12 A comparative spectral analysis diagram at different bitrates is provided to illustrate the relationship between compression bitrate and quality. Curve 1201 represents the spectral curve of the original speech, i.e., the uncompressed signal; curve 1202 represents the spectral curve of the OPUS encoder at a bitrate of 20kbps; and curve 1203 represents the spectral curve of the OPUS encoding at a bitrate of 6kbps. Figure 12 It can be seen that as the encoding bit rate increases, the compressed signal is closer to the original signal.

[0176] Traditional audio coding can be divided into two categories: time-domain coding and frequency-domain coding, both of which are compression methods based on signal processing. 1) Time-domain coding, such as waveform speech coding, directly encodes the waveform of the speech signal. This method offers high-quality speech encoding but is inefficient. Specifically, for speech signals, parametric coding can be used, where the encoder extracts the corresponding parameters of the desired speech signal. Parametric coding boasts extremely high efficiency but results in very low-quality speech reconstruction. 2) Frequency-domain coding transforms the audio signal to the frequency domain, extracts frequency coefficients, and then encodes these coefficients. However, its efficiency is also not ideal. Therefore, signal processing-based compression methods cannot improve coding efficiency while maintaining coding quality.

[0177] Based on this, embodiments of this application provide an audio encoding method and an audio decoding method to improve encoding efficiency while ensuring encoding quality. In these embodiments, the degree of freedom in selecting different encoding methods is achieved based on the encoding content and network bandwidth, even in low bitrate ranges; and encoding efficiency is improved while maintaining acceptable complexity and encoding quality. See also Figure 13 , Figure 13 This is a schematic diagram of the audio encoding and decoding process provided in the embodiments of this application. Here, taking a two-layer system as an example (this application does not limit iterative operations to a third or higher layer), the audio encoding method provided in the embodiments of this application includes:

[0178] (1) Subband decomposition is performed on the audio signal to obtain low-frequency subband signals and high-frequency subband signals. In actual implementation, the audio signal can be sampled according to the first sampling frequency to obtain the sampled signal, and then the sampled signal can be decomposed into subbands to obtain subband signals with frequencies lower than the first sampling frequency, including low-frequency subband signals and high-frequency subband signals. For example, for the audio signal of the nth frame The signal is decomposed into low-frequency subband signals using analytical filters (such as QMF filters). and high-frequency subband signals .

[0179] (2) The low-frequency subband signal is analyzed based on the first-layer low-frequency analysis neural network to obtain the first-layer low-frequency signal characteristics. For example, for the low-frequency subband signal... The first-layer low-frequency analysis neural network is invoked to obtain low-dimensional first-layer low-frequency signal features. It should be noted that the dimensionality of the signal features is smaller than that of the low-frequency subband signal (in order to reduce the amount of data). Neural networks include, but are not limited to, Dilated CNN, Autoencoder, Full-connection, LSTM, CNN+LSTM, etc.

[0180] (3) Analyze the high-frequency subband signal based on the first-layer high-frequency analysis neural network to obtain the first-layer high-frequency signal features. For example, for the high-frequency subband signal... The first-layer high-frequency analysis neural network is invoked to obtain low-dimensional first-layer high-frequency signal features. .

[0181] (4) Based on the second-layer low-frequency analysis neural network, the features of the low-frequency subband signal and the first-layer low-frequency signal are analyzed to obtain the features of the second-layer low-frequency signal (i.e., the features of the second-layer low-frequency residual signal). For example, combined with... and The second-layer low-frequency analysis neural network is invoked to obtain low-dimensional second-layer low-frequency signal features. .

[0182] (5) Based on the second-layer high-frequency analysis neural network, the high-frequency subband signal and the first-layer high-frequency signal features are analyzed to obtain the second-layer high-frequency signal features (i.e., the second-layer high-frequency residual signal features). For example, combined with... and The second-layer high-frequency analysis neural network is invoked to obtain low-dimensional second-layer high-frequency signal features. .

[0183] (6) The two layers of signal features (including the first layer of low-frequency signal features, the first layer of high-frequency signal features, the second layer of low-frequency signal features and the second layer of high-frequency signal features) are quantized and encoded through the quantization coding part to obtain the audio signal bitstream in each layer; and the corresponding transmission priority is configured for each layer of bitstream, such as the first layer being transmitted with a higher priority, the second layer with a lower priority, and so on.

[0184] In practical applications, the decoding end may only receive the bitstream of one layer, such as... Figure 13 As shown, decoding can be performed using a "one-layer decoding" approach. Based on this, the audio decoding method provided in this application includes: (1) decoding the received one-layer bitstream to obtain the low-frequency signal features and high-frequency signal features of that layer; (2) analyzing the low-frequency signal features based on the first-layer low-frequency synthesis neural network to obtain an estimate of the low-frequency sub-band signal. For example, the quantized value based on the low-frequency signal features... The first layer of the low-frequency synthetic neural network is invoked to generate low-frequency subband signal estimates. (3) Based on the first-layer high-frequency synthetic neural network, the high-frequency signal characteristics are analyzed to obtain the estimated value of the high-frequency sub-band signal. For example, the quantized value based on the high-frequency signal characteristics. The first layer of the high-frequency synthesis neural network is invoked to generate estimates of the high-frequency subband signal. (4) Estimated values ​​based on low-frequency subband signals and high-frequency subband signal estimation value The audio signal at the original sampling frequency is obtained by performing synthesis filtering through a synthesis filter. This completes the decoding process.

[0185] In practical applications, the decoding end may receive the bitstreams from both layers, such as... Figure 13 As shown, a "two-layer decoding" method can be used for decoding. Based on this, the audio decoding method provided in this application includes:

[0186] (1) Decode the received bitstreams of each layer to obtain the low-frequency signal characteristics and high-frequency signal characteristics of each layer.

[0187] (2) Based on the first-layer low-frequency synthetic neural network, the features of the first-layer low-frequency signal are analyzed to obtain the estimated value of the first-layer low-frequency sub-band signal. For example, the quantized value based on the features of the first-layer low-frequency signal. The first-layer low-frequency synthetic neural network is invoked to generate the first-layer low-frequency sub-band signal estimate. .

[0188] (3) Based on the first-layer high-frequency synthetic neural network, the features of the first-layer high-frequency signal are analyzed to obtain the estimated value of the first-layer high-frequency sub-band signal. For example, the quantized value based on the features of the first-layer high-frequency signal. The first-layer high-frequency synthesis neural network is invoked to generate the first-layer high-frequency subband signal estimate. .

[0189] (4) Based on the second-layer low-frequency synthetic neural network, the characteristics of the second-layer low-frequency signal are analyzed to obtain the estimated value of the second-layer low-frequency subband residual signal. For example, the quantized value based on the characteristics of the second-layer low-frequency signal. The second-layer low-frequency synthesis neural network is invoked to generate estimates of the low-frequency subband residual signal. .

[0190] (5) Based on the second-layer high-frequency synthesis neural network, the features of the second-layer high-frequency signal are analyzed to obtain the estimated value of the second-layer high-frequency subband residual signal. For example, the quantized value based on the features of the second-layer high-frequency signal. The second-layer high-frequency synthesis neural network is invoked to generate estimates of the high-frequency subband residual signal. .

[0191] (6) By summing the estimated value of the first-layer low-frequency subband signal with the estimated value of the low-frequency subband residual signal through the low-frequency part, the estimated value of the low-frequency subband signal is obtained. For example, ... and Summing yields the estimated value of the low-frequency subband signal.

[0192] (7) By summing the estimated value of the first-layer high-frequency subband signal with the estimated value of the high-frequency subband residual signal through the high-frequency part, the estimated value of the high-frequency subband signal is obtained. For example, the estimated value of the high-frequency subband signal is obtained by summing the estimated value of the first-layer high-frequency subband signal with the estimated value of the high-frequency subband residual signal. and Summation yields a high-quality estimate of the high-frequency subband signal.

[0193] (8) Based on the estimated values ​​of the low-frequency subband signal and the high-frequency subband signal, a synthesis filter is used to perform synthesis filtering to obtain the final reconstructed audio signal at the original sampling frequency. This completes the decoding process.

[0194] The embodiments of this application can be applied to various audio scenarios, such as remote voice communication. Taking remote voice communication as an example, see... Figure 14 , Figure 14 This is a schematic diagram of a voice communication link provided in an embodiment of this application. Here, taking a Voice over Internet Protocol (VoIP) conferencing system as an example, the voice encoding and decoding technologies involved in this embodiment are deployed in the encoding and decoding parts to solve the basic function of voice compression. The encoder is deployed on the uplink client 1401, and the decoder is deployed on the downlink client 1402. Voice is acquired through the uplink client and pre-processed, enhanced, and encoded. The encoded bitstream is transmitted to the downlink client 1402 via the network. The downlink client 1402 performs decoding and enhancement processing to play the decoded voice.

[0195] To ensure forward compatibility (i.e., compatibility between the new encoder and existing encoders), a transcoder needs to be deployed in the system's backend (i.e., the server) to address the interoperability issue between the new and existing encoders. For example, if the sending end (uplink client) is a new NN encoder, and the receiving end (downlink client) is a Public Switched Telephone Network (PSTN) decoder (e.g., a G.722 decoder), then after receiving the bitstream from the sending end, the server first needs to execute the NN decoder to generate the voice signal, and then call the G.722 encoder to generate a specific bitstream so that the receiving end can decode it correctly. Similar transcoding scenarios will not be elaborated further.

[0196] Before introducing the audio encoding and decoding methods provided in the embodiments of this application, we will first introduce the QMF filter bank and the dilated convolutional network.

[0197] A QMF filter bank is a filter pair that includes analysis and synthesis. For a QMF analysis filter, an input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, representing the QMF low-pass signal and the QMF high-pass signal, respectively. For example... Figure 15 The low-pass portion of the QMF filter shown ( ) and Qualcomm part ( The spectral response of the low-pass and high-pass filters. Based on the relevant theoretical knowledge of QMF analysis filter banks, the correlation between the coefficients of the low-pass and high-pass filters can be easily described, as shown in formula (1):

[0198] (1)

[0199] in, These represent the coefficients of the low-pass filter. This represents the coefficients of the high-pass filter.

[0200] Similarly, based on QMF-related theory, filter banks can be analyzed using QMF. and The QMF synthesis filter bank is described as shown in Equation (2).

[0201]

[0202] (2)

[0203] in, This indicates the recovered low-pass signal. This indicates the recovered high-pass signal.

[0204] The low-pass and high-pass signals recovered at the decoding end are synthesized by a QMF synthesis filter bank, which can then be used to recover the reconstructed signal corresponding to the sampling rate Fs of the input signal.

[0205] See Figure 16A and Figure 16B , Figure 16A This is a schematic diagram of a common convolutional network provided in an embodiment of this application. Figure 16B This is a schematic diagram of the dilated convolutional network provided in an embodiment of this application. Compared to ordinary convolutional networks, dilated convolution can increase the receptive field while maintaining the same feature map size, and can also avoid errors caused by upsampling and downsampling. Although Figure 16A and Figure 16B The kernel sizes shown are all 3×3; however, Figure 16A The receptive field of the ordinary convolution shown is only 3 (901), while Figure 16B The dilated convolution shown achieves a receptive field of 902, which is 5. That is, for a 3×3 convolution kernel, Figure 16A The receptive field of the ordinary convolution shown is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1; while Figure 16B The receptive field of the dilated convolution shown is 5, and the dilation rate is 2.

[0206] Convolution kernels can also be used in similar ways. Figure 16A or Figure 16BThe movement occurs on a plane, which involves the concept of stride rate. For example, if the convolution kernel moves one unit at a time, the corresponding stride rate is 1. In addition, there is the concept of the number of convolution channels, which refers to the number of parameters corresponding to the convolution kernel used for convolution analysis. Theoretically, the more channels, the more comprehensive the signal analysis and the higher the accuracy; however, the more channels, the higher the complexity. For example, a 1×320 tensor can be used for 24-channel convolution operations, resulting in a 24×320 tensor. It should be noted that the size of the dilated convolution kernel (e.g., for speech signals, the kernel size can be set to 1×3), dilation rate, stride rate, and number of channels can be defined according to actual application needs; this application does not impose specific limitations on these aspects.

[0207] The following example uses an audio signal with Fs=32000Hz (this embodiment is also applicable to other sampling frequencies, including but not limited to 8000Hz, 16000Hz, 48000Hz, etc.). The frame length is set to 20ms, which is equivalent to 640 sample points per frame for Fs=32000Hz.

[0208] Next, see [the following text]. Figure 13 The audio encoding method and audio decoding method provided in the embodiments of this application will be described in detail. The audio encoding method provided in the embodiments of this application includes:

[0209] Step 1: Generation of input signal.

[0210] Here, the 640 sample points of the nth frame are denoted as... .

[0211] Step 2, QMF subband signal decomposition.

[0212] Here, a QMF analysis filter (such as a 2-channel QMF filter) is used for filtering, and the filtered signal is downsampled to obtain two sub-band signals, namely the low-frequency sub-band signal. and high-frequency subband signals Among them, low-frequency subband signals The effective bandwidth is 0-8kHz, high-frequency subband signal The effective bandwidth is 8-16kHz, and the number of sample points per frame is 320.

[0213] Step 3, first layer low-frequency analysis.

[0214] Here, the purpose of calling the first layer of the low-frequency analysis neural network is to base it on the low-frequency subband signal. Generate lower-dimensional first-layer low-frequency signal features In this example, The data dimension is 320. The data dimension is 64. Judging from the data volume, it's clear that the first layer of the low-frequency analysis neural network has achieved a "dimensionality reduction" effect, which can be understood as data compression. For an example, see... Figure 17 , Figure 17 This is a schematic diagram of the structure of the first layer low-frequency analysis neural network provided in the embodiments of this application, for low-frequency sub-band signals. The processing flow includes:

[0215] (1) Call a 24-channel causal convolution to convert the input tensor (i.e. ), which is expanded into a 24*320 tensor.

[0216] (2) Preprocess the 24*320 tensor. In practical applications, a pooling operation with a pooling factor of 2 and an activation function of ReLU can be performed to generate a 24*160 tensor.

[0217] (3) Concatenate three encoding blocks with different downsampling factors. Taking an encoding block (Down_factor=4) as an example, one or more dilated convolutions can be performed first, with each convolution kernel size fixed at 1*3 and a stiderate of 1. In addition, the dilation rate of the one or more dilated convolutions can be set as needed, such as 3; of course, this embodiment does not limit different dilation rates for different dilated convolutions. Then, the Down_factors of the three encoding blocks are set to 4, 5, and 8 respectively, which is equivalent to setting different pooling factors, thus achieving downsampling. Finally, the number of channels of the three encoding blocks are set to 48, 96, and 192 respectively. Therefore, after three concatenated encoding blocks, the 24*160 tensor is successively converted into tensors of 48*40, 96*8, and 192*1 respectively.

[0218] (4) For the 192*1 tensor, after similar preprocessing, a causal convolution is performed to output a 64-dimensional feature vector, which is the first layer of low-frequency signal features. .

[0219] Step 4, first-level high-frequency analysis.

[0220] Here, the purpose of calling the first layer of the high-frequency analysis neural network is to base it on the high-frequency subband signal. Generate lower-dimensional first-layer high-frequency signal features In this example, the structure of the first high-frequency analysis neural network can be consistent with that of the first low-frequency analysis neural network, i.e., the input (i.e., The data dimension is 320, and the output (i.e.) The data dimension is 64. Considering that high-frequency subband signals are less important than low-frequency subband signals, the output dimension can be appropriately reduced. This can reduce the complexity of the first-layer high-frequency analysis neural network, but this is not a restriction in this example.

[0221] Step 5, second-layer low-frequency analysis.

[0222] Here, the purpose of calling the second-layer low-frequency analysis neural network is to base it on the low-frequency subband signal. and the characteristics of the first layer of low-frequency signals This yields a lower-dimensional second-layer low-frequency signal feature. The second layer of low-frequency signal characteristics reflects the residual of the reconstructed audio signal at the decoding end relative to the original audio signal, as output by the first layer of the low-frequency analysis neural network. Therefore, at the decoding end, it is possible to determine the residual based on the low-frequency signal characteristics. The residual signal of the predicted low-frequency subband signal is summed with the low-frequency subband signal estimate predicted by the output of the first-layer low-frequency analysis neural network to obtain a more accurate low-frequency subband signal estimate.

[0223] The second-layer low-frequency analysis neural network adopts a structure similar to that of the first-layer low-frequency analysis neural network. (See [link]) Figure 18 , Figure 18 This is a schematic diagram of the structure of the second-layer low-frequency analysis neural network provided in the embodiments of this application. Here, the main differences from the first-layer low-frequency analysis neural network type include: (1) the input of the second-layer low-frequency analysis neural network includes low-frequency subband signals in addition to low-frequency subband signals. It also includes the output of the first layer of the low-frequency analysis neural network. , and Two variables can be concatenated to form a 384-dimensional concatenated feature. (2) Considering that the second layer of low-frequency analysis processes the residual signal, the output of the second layer of low-frequency analysis neural network The dimension is set to 28.

[0224] Step 6, second-level high-frequency analysis.

[0225] Here, the purpose of calling the second-layer high-frequency analysis neural network is to base it on the high-frequency subband signal. and the characteristics of the first layer of high-frequency signals This yields lower-dimensional second-layer high-frequency signal features. The second-layer high-frequency analysis neural network and its structure can be the same as the second-layer low-frequency analysis neural network, i.e., the input ( and The data dimension of the splicing features is 384-dimensional, and the output ( The data dimensions are 28.

[0226] Step 7, quantization encoding.

[0227] The signal characteristics of the second-layer output are quantized by querying a pre-set quantization table, and the quantization result is encoded. Quantization can be performed using scalar quantization (each component is quantized individually), and encoding can be performed using entropy encoding. Furthermore, this application does not limit the combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy encoding techniques.

[0228] In actual implementation, the characteristics of the first layer of low-frequency signals The 64-dimensional features can be encoded using 8kbps, with an average bit rate of 2.5 bits per frame for quantizing one parameter; the first layer is high-frequency signal features. With 64-dimensional features, encoding can be completed at 6kbps, and the average bitrate for quantizing one parameter per frame is 1.875 bits. Therefore, the total encoding time for the first layer is 14kbps.

[0229] In practical implementation, the characteristics of the second-layer low-frequency signal The 28-dimensional features can be encoded using 3.5kbps, with an average bit rate of 2.5 bits per frame for quantizing one parameter; the second layer features high-frequency signals. With 28-dimensional features, encoding can be completed at 3.5kbps, and the average bitrate for quantizing one parameter per frame is 2.5 bits. Therefore, the total encoding time for the second layer is 7kbps.

[0230] Based on this, different feature vectors can be progressively encoded through layered encoding. Depending on the application scenario, this application does not limit the bitrate distribution to other methods; for example, a third or higher layer of encoding can be iteratively introduced. After quantization encoding, a bitstream can be generated. Different transmission strategies can be adopted for bitstreams of different layers to ensure transmission with different priorities. For example, forward error correction (FEC) can be used to improve transmission quality through redundant transmission. The redundancy factor is different for different layers; for example, the redundancy factor for the first layer can be set higher.

[0231] Taking the example that the bitstreams of all layers are received and accurately decoded by the decoding end, the audio encoding method provided in this application includes:

[0232] Step 1, Decode.

[0233] Here, decoding is the inverse process of encoding. The received bitstream is parsed, and low-frequency and high-frequency signal feature estimates are obtained by looking up the quantization table. Specifically, in the first layer, the quantized values ​​of the 64-dimensional signal features of the low-frequency sub-band signal are obtained. , and the quantized values ​​of the 64-dimensional signal characteristics of the high-frequency subband signal. The second layer obtains the quantized values ​​of the 28-dimensional signal characteristics of the low-frequency subband signal. , and the quantized values ​​of the 28-dimensional signal characteristics of the high-frequency subband signal. .

[0234] Step 2, first layer low-frequency synthesis.

[0235] Here, the purpose of calling the first layer of the low-frequency synthetic neural network is to obtain the quantized value of the low-frequency feature vector. Generate the first-level low-frequency subband signal estimate. For example, see Figure 19 , Figure 19 This is a schematic diagram of the first-layer low-frequency synthetic neural network provided in the embodiments of this application. Here, the processing flow of the first-layer low-frequency synthetic neural network is similar to that of the first-layer low-frequency analysis neural network, such as causal convolution; the post-processing structure of the first-layer low-frequency synthetic neural network is similar to the pre-processing structure of the first-layer low-frequency analysis neural network; the decoding block structure and the encoding block structure are symmetrical: the encoding block on the encoding side first performs dilated convolution and then pooling to complete downsampling, and the decoding block on the decoding side first performs pooling to complete upsampling and then dilated convolution.

[0236] Step 3, first-layer high-frequency synthesis.

[0237] Here, the structure of the first high-frequency synthetic neural network is the same as that of the first low-frequency synthetic neural network, and it can be based on the quantized value of the low-frequency signal features of the first layer. The first high-frequency subband signal estimate is obtained. .

[0238] Step 4, second-layer low-frequency synthesis.

[0239] Here, the purpose of calling the second-layer low-frequency synthesis neural network is to obtain the quantized value based on the features of the second-layer low-frequency signal. Generate low-frequency subband residual signal estimates See also Figure 20 , Figure 20 This is a schematic diagram of the structure of the second-layer low-frequency synthetic neural network provided in the embodiment of this application. The structure of the second-layer low-frequency synthetic neural network is similar to that of the first-layer low-frequency synthetic neural network, except that the input data dimension is 28 dimensions.

[0240] Step 5: Second-layer high-frequency synthesis.

[0241] Here, the structure of the second-layer low-frequency synthesis neural network is the same as that of the second-layer low-frequency synthesis neural network, and it can be based on the quantized value of the low-frequency signal features of the second layer. Generate high-frequency subband residual signal estimates .

[0242] Step 6: Synthesis and Filtering.

[0243] Based on the preceding steps, the decoding end obtains the low-frequency subband signal estimate. and high-frequency subband signals and low-frequency subband residual signal estimation value and high-frequency subband signal residual estimates .Will and Add them together to generate a high-precision estimate of the low-frequency subband signal; and The values ​​are summed to generate a high-precision estimate of the high-frequency subband signal. Finally, the low-frequency and high-frequency subband signal estimates are upsampled, and a QMF synthesis filter is applied to synthesize and filter the upsampled results, generating a 640-point reconstructed audio signal. .

[0244] In this embodiment, the relevant neural networks at the encoder and decoder ends can be jointly trained by collecting data to obtain optimal parameters, thereby enabling the trained network model to be put into use. This embodiment only discloses one specific example of network input, network structure, and network output; those skilled in the art can further modify the above configuration as needed.

[0245] By applying the embodiments described above, a low bitrate audio encoding and decoding scheme based on signal processing and deep learning networks can be completed. Through the organic combination of signal decomposition and related signal processing techniques with deep neural networks, the encoding efficiency is significantly improved compared to related techniques, and the encoding quality is also improved while maintaining acceptable complexity. Depending on the encoding content and bandwidth, the encoding end selects different layered transmission strategies for bitstream transmission. The decoding end receives the lower-level bitstream and outputs an audio signal of acceptable quality. If it also receives other higher-level bitstreams, it can output high-quality audio.

[0246] It is understood that in the embodiments of this application, data related to user information (such as audio signals sent by users) is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0247] The following description continues to illustrate the exemplary structure of the audio encoding device 553 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the audio encoding device 553 in the memory 550 may include:

[0248] The first feature extraction module 5531 is used to perform first-level feature extraction processing on the audio signal to obtain the first-level signal features; the second feature extraction module 5532 is used to perform concatenation processing on the audio signal and the signal features of the (i-1)th level for the i-th level of N levels to obtain concatenated features, and perform first-level feature extraction processing on the concatenated features to obtain the signal features of the i-th level, where N and i are integers greater than 1, and i is less than or equal to N; the traversal module 5533 is used to traverse i to obtain the signal features of each level in the N levels, where the data dimension of the signal features is smaller than the data dimension of the audio signal; the encoding module 5534 is used to encode the first-level signal features and the signal features of each level in the N levels respectively to obtain the bitstream of the audio signal at each level.

[0249] In some embodiments, the first feature extraction module 5531 is further configured to perform subband decomposition processing on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal; perform first-level feature extraction processing on the low-frequency subband signals to obtain first-level low-frequency signal features, and perform first-level feature extraction processing on the high-frequency subband signals to obtain first-level high-frequency signal features; and use the low-frequency signal features and the high-frequency signal features as the first-level signal features.

[0250] In some embodiments, the first feature extraction module 5531 is further configured to sample the audio signal according to a first sampling frequency to obtain a sampled signal; perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, and perform downsampling on the low-pass filtered signal to obtain the low-frequency sub-band signal at a second sampling frequency; perform high-pass filtering on the sampled signal to obtain a high-pass filtered signal, and perform downsampling on the high-pass filtered signal to obtain the high-frequency sub-band signal at a second sampling frequency; wherein the second sampling frequency is less than the first sampling frequency.

[0251] In some embodiments, the second feature extraction module 5532 is further configured to perform splicing processing on the low-frequency sub-band signal of the audio signal and the low-frequency signal features of the (i-1)th level to obtain a first spliced ​​feature, and perform feature extraction processing on the first spliced ​​feature at the i-th level to obtain the low-frequency signal features of the i-th level; perform splicing processing on the high-frequency sub-band signal of the audio signal and the high-frequency signal features of the (i-1)th level to obtain a second spliced ​​feature, and perform feature extraction processing on the second spliced ​​feature at the i-th level to obtain the high-frequency signal features of the i-th level; and use the low-frequency signal features of the i-th level and the high-frequency signal features of the i-th level as the signal features of the i-th level.

[0252] In some embodiments, the first feature extraction module 5531 is further configured to perform a first convolution processing on the audio signal to obtain convolutional features of the first level; perform a first pooling processing on the convolutional features to obtain pooling features of the first level; perform a first downsampling processing on the pooling features to obtain downsampled features of the first level; and perform a second convolution processing on the downsampled features to obtain signal features of the first level.

[0253] In some embodiments, the first downsampling process is implemented through M cascaded coding layers. The first feature extraction module 5531 is further configured to perform a first downsampling process on the pooling feature through the first coding layer in the M cascaded coding layers to obtain the downsampling result of the first coding layer; and to perform a first downsampling process on the downsampling result of the (j-1)th coding layer through the j-th coding layer in the M cascaded coding layers to obtain the downsampling result of the j-th coding layer; wherein M and j are integers greater than 1, and j is less than or equal to M; and to iterate through j to obtain the downsampling result of the M-th coding layer, and use the downsampling result of the M-th coding layer as the downsampling feature of the first level.

[0254] In some embodiments, the second feature extraction module 5532 is further configured to perform a third convolution processing on the spliced ​​features to obtain the convolutional features of the i-th level; perform a second pooling processing on the convolutional features to obtain the pooling features of the i-th level; perform a second downsampling processing on the pooling features to obtain the downsampled features of the i-th level; and perform a fourth convolution processing on the downsampled features to obtain the signal features of the i-th level.

[0255] In some embodiments, the encoding module 5534 is further configured to perform quantization processing on the signal features of the first level and the signal features of each of the N levels respectively to obtain the quantization results of the signal features of each level; and to perform entropy encoding processing on the quantization results of the signal features of each level to obtain the bitstream of the audio signal at each level.

[0256] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features. The encoding module 5534 is further configured to encode the low-frequency signal features of the first level and the low-frequency signal features of each of the N levels respectively to obtain the low-frequency code stream of the audio signal at each level; to encode the high-frequency signal features of the first level and the high-frequency signal features of each of the N levels respectively to obtain the high-frequency code stream of the audio signal at each level; and to use the low-frequency code stream and the high-frequency code stream of the audio signal at each level as the code stream of the audio signal at the corresponding level.

[0257] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features. The encoding module 5534 is further configured to encode the low-frequency signal features of the first level according to a first encoding code rate to obtain a first bitstream of the first level, and to encode the high-frequency signal features of the first level according to a second encoding code rate to obtain a second bitstream of the first level. For the signal features of each of the N levels, the following processing is performed respectively: the signal features of the level are encoded according to a third encoding code rate of the level to obtain a second bitstream of each level; the second bitstream of the first level and the second bitstream of each of the N levels are used as the bitstream of the audio signal at each level; wherein, the first encoding code rate is greater than the second encoding code rate, the second encoding code rate is greater than the third encoding code rate of any of the N levels, and the encoding code rate of the level is positively correlated with the decoding quality index of the bitstream of the corresponding level.

[0258] In some embodiments, the encoding module 5534 is further configured to perform the following processing for each of the layers: configuring a corresponding layer transmission priority for the bitstream of the audio signal at the layer; wherein the layer transmission priority is negatively correlated with the number of layers of the layer, and the layer transmission priority is positively correlated with the decoding quality index of the bitstream of the corresponding layer.

[0259] In some embodiments, the signal features include low-frequency signal features and high-frequency signal features, and the audio signal bitstream at each level includes: a low-frequency bitstream encoded based on the low-frequency signal features and a high-frequency bitstream encoded based on the high-frequency signal features; the encoding module 5534 is further configured to perform the following processing for each level: configure a first transmission priority for the low-frequency bitstream of the level and configure a second transmission priority for the high-frequency bitstream of the level; wherein, the first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th level is lower than the first transmission priority of the i-th level, and the transmission priority of the bitstream is positively correlated with the decoding quality index of the corresponding bitstream.

[0260] By applying the above embodiments of this application, layered encoding of audio signals is realized: First, the audio signal is subjected to feature extraction processing of the first level to obtain the signal features of the first level; then, for the i-th level (i is an integer greater than 1, i is less than or equal to N) of the N (N is an integer greater than 1) levels, the audio signal and the signal features of the (i-1)-th level are concatenated to obtain concatenated features, and the concatenated features are subjected to feature extraction processing of the i-th level to obtain the signal features of the i-th level; then, by traversing i, the signal features of each of the N levels are obtained; finally, the signal features of the first level and the signal features of each of the N levels are encoded respectively to obtain the bitstream of the audio signal at each level.

[0261] First, the data dimension of the extracted signal features is smaller than that of the audio signal. This reduces the data dimension of the data processed during audio encoding, thus improving the encoding efficiency of the audio signal.

[0262] Secondly, when extracting signal features from audio signals in a hierarchical manner, the output of each level serves as the input to the next level. This allows each level to combine the signal features extracted from the previous level, resulting in more accurate feature extraction of the audio signal. As the number of levels increases, information loss during feature extraction is minimized. Thus, multiple bitstreams obtained by encoding the signal features extracted in this way contain audio signal information closer to the original audio signal, reducing information loss during encoding and ensuring the quality of audio encoding.

[0263] The following describes the audio decoding apparatus provided in the embodiments of this application. The audio decoding apparatus provided in the embodiments of this application includes: a receiving module, configured to receive multiple bitstreams corresponding to different levels obtained by encoding an audio signal; a decoding module, configured to decode each bitstream of each level to obtain signal features of each level, wherein the data dimension of the signal features is smaller than the data dimension of the audio signal; a feature reconstruction module, configured to reconstruct features from the signal features of each level to obtain a layered audio signal of each level; and an audio synthesis module, configured to synthesize the layered audio signals of the multiple levels to obtain the audio signal.

[0264] In some embodiments, the bitstream includes a low-frequency bitstream and a high-frequency bitstream. The decoding module is further configured to decode the low-frequency bitstream of each layer to obtain the low-frequency signal features of each layer, and to decode the high-frequency bitstream of each layer to obtain the high-frequency signal features of each layer. Correspondingly, the feature reconstruction module is further configured to reconstruct the low-frequency signal features of each layer to obtain the layer-level low-frequency sub-band signal of each layer, and to reconstruct the high-frequency signal features of each layer to obtain the layer-level high-frequency sub-band signal of each layer. The layer-level low-frequency sub-band signal and the layer-level high-frequency sub-band signal are used as the layer-level audio signal of the layer. Correspondingly, the audio synthesis module is further configured to add the layer-level low-frequency sub-band signals of multiple layers to obtain the low-frequency sub-band signal, and add the layer-level high-frequency sub-band signals of multiple layers to obtain the high-frequency sub-band signal. The low-frequency sub-band signal and the high-frequency sub-band signal are synthesized to obtain the audio signal.

[0265] In some embodiments, the audio synthesis module is further configured to upsample the low-frequency subband signal to obtain a low-pass filtered signal; upsample the high-frequency subband signal to obtain a high-frequency filtered signal; and filter and synthesize the low-pass filtered signal and the high-frequency filtered signal to obtain the audio signal.

[0266] In some embodiments, the feature reconstruction module is further configured to perform the following processing on the signal features of each level: perform a first convolution processing on the signal features to obtain the convolution features of the level; perform an upsampling processing on the convolution features to obtain the upsampled features of the level; perform a pooling processing on the upsampled features to obtain the pooled features of the level; and perform a second convolution processing on the pooled features to obtain the level audio signal of the level.

[0267] In some embodiments, the upsampling process is implemented through L cascaded decoding layers. The feature reconstruction module is further configured to perform upsampling processing on the pooled features through the first decoding layer in the L cascaded decoding layers to obtain the upsampling result of the first decoding layer; and to perform upsampling processing on the first upsampling result of the (k-1)th decoding layer through the kth decoding layer in the L cascaded decoding layers to obtain the upsampling result of the kth decoding layer; wherein L and k are integers greater than 1, and k is less than or equal to L; and to iterate through k to obtain the upsampling result of the Lth decoding layer, and use the upsampling result of the Lth decoding layer as the upsampling feature of the layer.

[0268] In some embodiments, the decoding module is further configured to perform the following processing for each of the layers: perform entropy decoding processing on the bitstream of the layer to obtain the quantized value of the bitstream; and perform inverse quantization processing on the quantized value of the bitstream to obtain the signal characteristics of the layer.

[0269] By applying the above embodiments of this application, multiple levels of bitstreams are decoded separately to obtain signal features at each level. Feature reconstruction is then performed on the signal features at each level to obtain the audio signal at each level. Finally, the audio signals at multiple levels are synthesized to obtain the final audio signal. Since the data dimension of the signal features is smaller than the data dimension of the audio signal, the data dimension of the data processed during audio decoding is reduced, thus improving the decoding efficiency of the audio signal.

[0270] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in this application.

[0271] This application also provides a computer-readable storage medium storing executable instructions that, when executed by a processor, will cause the processor to perform the method provided in this application.

[0272] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0273] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0274] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0275] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0276] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An audio encoding method, characterized in that, The method includes: The audio signal is subjected to a first-level feature extraction process to obtain the first-level signal features, which include low-frequency signal features and high-frequency signal features. The audio signal includes low-frequency sub-band signals and high-frequency sub-band signals. For the i-th level other than the first level among the N levels, the low-frequency sub-band signal of the audio signal and the low-frequency signal features of the (i-1)-th level are spliced ​​together to obtain the first spliced ​​feature, and the first spliced ​​feature is subjected to the feature extraction processing of the i-th level to obtain the low-frequency signal features of the i-th level. The high-frequency subband signal of the audio signal and the high-frequency signal features of the (i-1)th level are spliced ​​together to obtain the second spliced ​​feature, and the second spliced ​​feature is subjected to the feature extraction process of the i-th level to obtain the high-frequency signal features of the i-th level. The low-frequency signal features and the high-frequency signal features of the i-th level are used as the signal features of the i-th level, where i is a positive integer that increases sequentially, N is a positive integer greater than 1, and 1 < i ≤ N. By iterating through i, the signal features of each of the N levels, excluding the first level, are obtained; The signal features of the first level and the signal features of each of the N levels other than the first level are quantized and encoded to obtain the bitstream of the audio signal at each level.

2. The method as described in claim 1, characterized in that, The first-level feature extraction process of the audio signal to obtain the first-level signal features includes: The audio signal is subjected to subband decomposition processing to obtain the low-frequency subband signal and the high-frequency subband signal of the audio signal; The low-frequency subband signal is subjected to a first-level feature extraction process to obtain the first-level low-frequency signal features, and the high-frequency subband signal is subjected to a first-level feature extraction process to obtain the first-level high-frequency signal features. The low-frequency signal features and the high-frequency signal features are used as the signal features of the first level.

3. The method as described in claim 1 or 2, characterized in that, The first-level feature extraction process of the audio signal to obtain the first-level signal features includes: The audio signal is subjected to a first convolutional process to obtain the first level of convolutional features; The convolutional features are subjected to a first pooling process to obtain the pooled features of the first level; The pooling features are subjected to a first downsampling process to obtain the downsampled features of the first level; The downsampled features are subjected to a second convolution process to obtain the signal features of the first level.

4. The method as described in claim 3, characterized in that, The first downsampling process is implemented through M cascaded coding layers. The first downsampling process on the pooled features to obtain the downsampling features of the first level includes: The pooling features are downsampled through the first coding layer in the M cascaded coding layers to obtain the downsampling result of the first coding layer; By performing a first downsampling process on the downsampling result of the (j-1)th coding layer in the M cascaded coding layers, the downsampling result of the jth coding layer is obtained, where j is a positive integer that increases sequentially, M is an integer greater than 1, and 1 < j ≤ M; The j-th coding layer is traversed to obtain the downsampling result of the M-th coding layer, and the downsampling result of the M-th coding layer is used as the downsampling feature of the first layer.

5. The method as described in claim 1 or 4, characterized in that, The step of performing feature extraction processing at the i-th level on the first spliced ​​feature to obtain the low-frequency signal feature at the i-th level includes: The first spliced ​​feature is subjected to a third convolution process to obtain the convolutional feature of the i-th level; The convolutional features are subjected to a second pooling process to obtain the pooled features of the i-th layer; The pooling features are subjected to a second downsampling process to obtain the downsampled features of the i-th level; The downsampled features are subjected to a fourth convolution process to obtain the low-frequency signal features of the i-th level.

6. The method as described in claim 1, characterized in that, The step of quantizing and encoding the signal features of the first level and the signal features of each of the N levels other than the first level to obtain the bitstream of the audio signal at each level includes: The low-frequency signal features of the first level and the low-frequency signal features of each of the N levels other than the first level are quantized and encoded to obtain the low-frequency code stream of the audio signal at each level. The high-frequency signal features of the first level and the high-frequency signal features of each of the N levels other than the first level are quantized and encoded to obtain the high-frequency code stream of the audio signal at each level. The low-frequency and high-frequency bitstreams of the audio signal at each level are used as the bitstreams of the audio signal at the corresponding level.

7. The method as described in claim 1, characterized in that, The step of quantizing and encoding the signal features of the first level and the signal features of each of the N levels other than the first level to obtain the bitstream of the audio signal at each level includes: According to the first coding rate, the low-frequency signal features of the first level are quantized and encoded to obtain the first code stream of the first level, and according to the second coding rate, the high-frequency signal features of the first level are quantized and encoded to obtain the second code stream of the first level. For signal features at other levels, the following processing is performed: the signal features at other levels are quantized and encoded according to the third coding rate of the other levels to obtain the second code stream of the other levels, wherein the other levels are the levels other than the first level among N levels; The second bitstream of the first level and the second bitstream of each of the N levels other than the first level are used as the bitstream of the audio signal at each level. Wherein, the first encoding bitrate is greater than the second encoding bitrate, the second encoding bitrate is greater than the third encoding bitrate of any of the N levels except the first level, and the encoding bitrate of each level is positively correlated with the decoding quality index of the bitstream of each level.

8. The method as described in claim 1, characterized in that, After quantizing and encoding the signal features of the first level and the signal features of each of the N levels other than the first level to obtain the bitstream of the audio signal at each level, the method further includes: For each of the aforementioned levels, the following processing is performed: Configure the corresponding layer transmission priority for the audio signal bitstream at the layer level; The hierarchical transmission priority is negatively correlated with the number of levels in the hierarchy, and positively correlated with the decoding quality index of the corresponding level's bitstream.

9. The method as described in claim 1, characterized in that, The audio signal bitstream at each level includes: a low-frequency bitstream obtained by quantization encoding based on the low-frequency signal features, and a high-frequency bitstream obtained by quantization encoding based on the high-frequency signal features; The method further includes: For each of the aforementioned layers, the following processing is performed: a first transmission priority is configured for the low-frequency bitstream of the layer, and a second transmission priority is configured for the high-frequency bitstream of the layer; Wherein, the first transmission priority is higher than the second transmission priority, the second transmission priority of the (i-1)th level is lower than the first transmission priority of the i-th level, and the transmission priority of the bitstream of each level is positively correlated with the decoding quality index of the corresponding bitstream.

10. An audio decoding method, characterized in that, The method includes: Receive multiple layers of bitstreams corresponding to the encoded audio signal, the bitstreams including low-frequency bitstreams and high-frequency bitstreams; The bitstreams of each level are decoded to obtain the signal features of each level. Specifically, the following processing is performed on each level: entropy decoding is performed on the bitstream of the level to obtain the quantized value of the bitstream; inverse quantization is performed on the quantized value of the bitstream to obtain the signal features of the level, which include low-frequency signal features and high-frequency signal features. The signal features of each level are reconstructed to obtain the level audio signal of each level. This reconstructing includes: reconstructing the low-frequency signal features of each level to obtain the level low-frequency sub-band signal of each level; and reconstructing the high-frequency signal features of each level to obtain the level high-frequency sub-band signal of each level. The level low-frequency sub-band signal and the level high-frequency sub-band signal are then used as the level audio signal of each level. Audio synthesis is performed on the hierarchical audio signals of multiple levels to obtain a reconstructed audio signal. The process of synthesizing the hierarchical audio signals of multiple levels to obtain the reconstructed audio signal includes: adding the low-frequency sub-band signals of multiple levels to obtain a low-frequency sub-band signal, and adding the high-frequency sub-band signals of multiple levels to obtain a high-frequency sub-band signal; and synthesizing the low-frequency sub-band signal and the high-frequency sub-band signal to obtain the reconstructed audio signal.

11. The method as described in claim 10, characterized in that, The step of decoding the bitstream at each of the aforementioned levels to obtain the signal characteristics of each level includes: The low-frequency bitstreams of each layer are decoded to obtain the low-frequency signal characteristics of each layer, and the high-frequency bitstreams of each layer are decoded to obtain the high-frequency signal characteristics of each layer.

12. The method as described in claim 10, characterized in that, The process of synthesizing the low-frequency sub-band signal and the high-frequency sub-band signal to obtain the reconstructed audio signal includes: The low-frequency subband signal is upsampled to obtain a low-pass filtered signal; The high-frequency subband signal is upsampled to obtain a high-frequency filtered signal; The low-pass filtered signal and the high-frequency filtered signal are filtered and synthesized to obtain the reconstructed audio signal.

13. The method as described in claim 10, characterized in that, The step of reconstructing the signal features of each of the aforementioned levels to obtain the hierarchical audio signals of each level further includes: For the signal characteristics of each of the aforementioned levels, the following processing is performed respectively: The signal features are subjected to a first convolution process to obtain the convolutional features of the layer. The convolutional features are upsampled to obtain the upsampled features of the layer. The upsampled features are depooled to obtain the depooled features of the level. The unpooling features are subjected to a second convolution process to obtain the hierarchical audio signal of the level.

14. The method as described in claim 13, characterized in that, The upsampling process is implemented through L cascaded decoding layers. The upsampling of the convolutional features to obtain the upsampled features of the specified layer includes: The convolutional features are upsampled through the first decoding layer in the L cascaded decoding layers to obtain the upsampling result of the first decoding layer; By upsampling the first upsampling result of the (k-1)th decoding layer in the L cascaded decoding layers, the upsampling result of the kth decoding layer is obtained, where k is a positive integer that increases sequentially, L is an integer greater than 1, and 1 < k ≤ L; The k is traversed to obtain the upsampling result of the Lth decoding layer, and the upsampling result of the Lth decoding layer is used as the upsampling feature of the layer.

15. An audio encoding device, characterized in that, The device includes: The first feature extraction module is used to perform first-level feature extraction processing on the audio signal to obtain the first-level signal features. The signal features include low-frequency signal features and high-frequency signal features. The audio signal includes low-frequency sub-band signals and high-frequency sub-band signals. The second feature extraction module is used to perform splicing processing on the low-frequency sub-band signal of the audio signal and the low-frequency signal features of the (i-1)th level for the i-th level (excluding the first level) of N levels, to obtain a first spliced ​​feature, and to perform feature extraction processing on the first spliced ​​feature at the i-th level to obtain the low-frequency signal features of the i-th level; to perform splicing processing on the high-frequency sub-band signal of the audio signal and the high-frequency signal features of the (i-1)th level to obtain a second spliced ​​feature, and to perform feature extraction processing on the second spliced ​​feature at the i-th level to obtain the high-frequency signal features of the i-th level; and to use the low-frequency signal features and the high-frequency signal features of the i-th level as the signal features of the i-th level, where i is a positive integer that increases sequentially, N is a positive integer greater than 1, and 1 < i ≤ N; The encoding module is used to traverse i to obtain the signal features of each of the N levels except the first level; and to perform quantization encoding processing on the signal features of the first level and the signal features of each of the N levels except the first level to obtain the bitstream of the audio signal at each level.

16. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the method described in any one of claims 1 to 14.

18. A computer program product comprising executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the method described in any one of claims 1 to 14.

Citation Information

Patent Citations

  • Hierarchical audio coding and decoding methods and systems and transient signal hierarchical coding and decoding methods

    CN102222505A

  • Audio noise reduction processing method, device and equipment

    CN112420065A

  • Voice signal coding and decoding method and device, electronic equipment and storage medium

    CN113470667A