Audio encoding and decoding method, device, equipment, medium and program product

Abstract

The invention provides an audio coding and decoding method and device, equipment, a medium and a program product. The audio coding method comprises the following steps: decomposing an audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal; performing multi-level feature extraction processing based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to the plurality of levels; performing quantization processing on the sub-band signal features corresponding to each level to obtain index values of the sub-band signal features; encoding the index value of the sub-band signal feature to obtain a code stream corresponding to the hierarchy; configuring corresponding transmission priorities for the code streams corresponding to the plurality of layers respectively; wherein the transmission priority is positively correlated with the decoding quality index of the code stream corresponding to the hierarchy. The method can be flexibly applied to application scenes of different network bandwidths, and the audio coding and decoding efficiency and quality are improved.

Description

[0001] This application is a divisional application of a patent application with an application date of June 15, 2022, Chinese patent application number 202210681816.X, and invention name “Audio encoding and decoding methods, devices, equipment, media and program products”. Technical Field

[0002] The present application relates to audio processing technology, and in particular to an audio encoding and decoding method, device, equipment, medium and program product. Background Art

[0003] Audio codec technology is a core technology in communication services including remote audio and video calls. Traditional codec technology is based on the transformation of time domain and frequency domain. For example, various standard voice codec protocols belong to this category. Taking coding as an example, including time domain coding and frequency domain coding, their essence is a compression method based on signal processing. Under the premise of maintaining a certain voice quality, it is difficult to significantly reduce the coding bit rate because the two are mutually constrained.

[0004] Artificial Intelligence (AI) is a comprehensive technology in computer science, and its application in the field of audio coding is increasing. For example, deep learning-based coding and decoding technology aims to achieve higher voice quality than traditional coding and decoding technology at low bit rates. However, due to the high computational complexity of AI coding and decoding technology, the coding efficiency is affected, and it is not suitable for low-bandwidth application scenarios when the bit rate is high.

[0005] In summary, there is no effective solution in the relevant technologies on how to improve the efficiency of encoding and decoding and voice quality by applying different network bandwidth application scenarios. Summary of the invention

[0006] The embodiments of the present application provide an audio encoding and decoding method, device, electronic device, computer-readable storage medium and computer program product, which can be flexibly applied to application scenarios with different network bandwidths and improve the efficiency and quality of audio encoding and decoding.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] The present application provides an audio encoding method, including:

[0009] Decomposing the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal;

[0010] Performing feature extraction processing at multiple levels based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to the multiple levels respectively;

[0011] Quantizing the sub-band signal features corresponding to each of the layers to obtain index values ​​of the sub-band signal features;

[0012] Encoding the index value of the sub-band signal feature to obtain a code stream corresponding to the layer;

[0013] Corresponding transmission priorities are configured for the code streams corresponding to the multiple layers respectively; wherein the transmission priorities are positively correlated with the decoding quality indicators of the code streams corresponding to the layers.

[0014] The present application provides an audio decoding method, including:

[0015] Decoding the code streams corresponding to the multiple layers respectively to obtain the index value of the code stream corresponding to each layer; wherein different layers correspond to different transmission priorities, and the transmission priority is positively correlated with the decoding quality index of the code stream corresponding to the layer;

[0016] Performing inverse quantization processing on the index value of the code stream corresponding to each of the layers to obtain the subband signal features corresponding to each of the layers;

[0017] Performing feature reconstruction processing on the sub-band signal features corresponding to each of the layers to obtain the sub-band signals corresponding to each of the layers;

[0018] The sub-band signals respectively corresponding to the multiple layers are synthesized into an audio signal.

[0019] The present application provides an audio encoding device, including:

[0020] A decomposition module is used to decompose the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal;

[0021] A feature extraction module, configured to perform feature extraction processing at multiple levels based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to the multiple levels;

[0022] A quantization module, used for performing quantization processing on the sub-band signal features corresponding to each of the layers to obtain index values ​​of the sub-band signal features;

[0023] A coding module, used for coding the index value of the sub-band signal feature to obtain a code stream corresponding to the layer;

[0024] A configuration module is used to configure corresponding transmission priorities for the code streams corresponding to the multiple layers respectively; wherein the transmission priorities are positively correlated with the decoding quality indicators of the code streams corresponding to the layers.

[0025] The present application provides an audio decoding device, including:

[0026] A decoding module, used for decoding the code streams corresponding to the multiple layers respectively, and obtaining the index value of the code stream corresponding to each layer respectively; wherein different layers correspond to different transmission priorities, and the transmission priority is positively correlated with the decoding quality index of the code stream corresponding to the layer;

[0027] An inverse quantization module, used for performing inverse quantization processing on the index value of the code stream corresponding to each of the layers to obtain the subband signal characteristics corresponding to each of the layers;

[0028] A feature reconstruction module, used for performing feature reconstruction processing on the sub-band signal features corresponding to each of the layers to obtain the sub-band signals corresponding to each of the layers;

[0029] The synthesis module is used to synthesize the sub-band signals corresponding to the multiple layers into an audio signal.

[0030] An embodiment of the present application provides an electronic device, including:

[0031] A memory for storing executable instructions;

[0032] The processor is used to implement the audio encoding method and audio decoding method provided in the embodiments of the present application when executing the executable instructions stored in the memory.

[0033] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the audio encoding method and audio decoding method provided in the embodiment of the present application when executed by a processor.

[0034] The embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the audio encoding method and audio decoding method described in the embodiment of the present application.

[0035] The embodiments of the present application have the following beneficial effects:

[0036] The sub-band signal characteristics of the audio sub-band signal at each level are obtained by layering, and the sub-band signal characteristics corresponding to each level are encoded in a hierarchical manner. In this way, each level only needs to encode specific sub-band signal characteristics instead of encoding the characteristics of the entire audio signal. This not only improves the efficiency of encoding and decoding, but also reduces the encoding and decoding complexity of each level. According to the importance of code streams at different levels to the decoding quality, different transmission priorities are flexibly configured for code streams at different levels to ensure that more important code streams are transmitted first, which can be suitable for application scenarios with different network bandwidths. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of spectrum comparison at different bit rates provided in an embodiment of the present application;

[0038] Figure 2 1 is a schematic diagram of the architecture of an audio codec system 100 provided in an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of a voice communication link provided in an embodiment of the present application;

[0040] Figure 4A is a schematic diagram of the structure of the terminal 401 provided in an embodiment of the present application;

[0041] Figure 4B is a schematic diagram of the structure of the terminal 402 provided in an embodiment of the present application;

[0042] Figure 5A-5G It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0043] Figure 6A-6G It is a flowchart of the audio decoding method provided in an embodiment of the present application;

[0044] Fig. 7A It is a schematic diagram of performing encoding and decoding at one level provided in an embodiment of the present application;

[0045] Figure 7B It is a schematic diagram of performing two-level encoding and decoding provided in an embodiment of the present application;

[0046] Figure 7C It is a schematic diagram of performing three-level encoding and decoding provided in an embodiment of the present application;

[0047] Fig.7D is a schematic diagram of the spectrum response corresponding to the QMF filter bank provided in an embodiment of the present application;

[0048] Fig. 7E is a schematic diagram of frequency band expansion provided by an embodiment of the present application;

[0049] Fig. 8AIt is a schematic diagram of a common convolutional network and a hole convolutional network provided in an embodiment of the present application;

[0050] Figure 8B is a schematic diagram of the structure of a neural network for performing a first feature extraction process provided in an embodiment of the present application;

[0051] Figure 8C is a schematic diagram of the structure of a neural network for performing a third feature extraction process provided in an embodiment of the present application;

[0052] Fig.8D is a schematic diagram of the structure of a neural network for performing first feature reconstruction processing provided in an embodiment of the present application;

[0053] Fig. 8E It is a schematic diagram of the structure of a neural network for performing third feature reconstruction processing provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0055] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0056] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0059] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0060] 1) Audio Coding: Audio coding is an application of data compression for audio signals containing speech. Speech coding uses speech-specific parameter estimation, uses audio signal processing techniques to model the speech signal, and combines general data compression algorithms to represent the generated modeling parameters in a compact bit stream.

[0061] 2) Quadrature Mirror Filter (QMF): used to decompose the audio signal into two sub-band signals of equal bandwidth, namely, a high-frequency sub-band signal and a low-frequency sub-band signal.

[0062] 3) Scalable Coding: A technology used to be compatible with different terminal devices and link bandwidths. Its characteristic is that the bit stream is layered, the low-layer bit stream can be decoded separately, and the high-layer bit stream can enhance the audio quality.

[0063] 4) Neural Network (NN): An algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. Neural networks rely on the complexity of the system and adjust the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.

[0064] 5) Deep Learning (DL): A type of machine learning that represents attribute categories or features by combining low-level features to form more abstract high-level features, and is used to discover distributed feature representations of data.

[0065] 6) Entropy coding: that is, coding that does not lose any information during the coding process according to the entropy principle. Information entropy is the average amount of information in the source. Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0066] The applicant found that when audio encoding is performed based on signal processing, based on distortion rate analysis and combined with past standardization experience, the following conclusions were drawn: at least a bit rate of 0.75 bits per sample is required to provide ideal voice quality, and if music is encoded, at least a bit rate of 1.5 bits per sample is required. Typically, for a wideband voice signal with a sampling rate of 16000 Hz, the bit rate of the encoded information is 12 kbps; for an ultra-wideband voice signal with a sampling rate of 32000 Hz, the bit rate of the encoded information is 48 kbps.

[0067] See also Figure 1 , Figure 1 is a schematic diagram of spectrum comparison under different bit rates provided in an embodiment of the present application. Figure 1 In FIG. 1 , 101 represents the spectrum of the original audio signal, that is, the spectrum of the uncompressed audio signal; 102 represents the spectrum of the audio signal restored from the data with a bit rate of 20 kbps after encoding; and 103 represents the spectrum of the audio signal restored from the data with a bit rate of 6 kbps after encoding. Figure 1 As shown, the higher the bit rate of the encoded data is, the closer the spectrum of the restored audio signal is to the spectrum of the original audio signal.

[0068] In the audio encoding and decoding method 1 of the related art, the encoding end extracts typical speech features such as line spectral frequency (LSF) based on the traditional signal processing method. For broadband speech, 10-16 LSF coefficients can be extracted per frame, as well as the energy of the speech frame. The encoder will compress and encode these speech features. The decoding end decodes the above-mentioned speech features and calls the speech generation model, such as Wavenet and other generative networks to generate speech signals. The bit rate of this method can be lower than 2kbps. However, it is necessary to call wavenet and other generative networks to generate speech signals, and the computational complexity of the decoding end is very high, which is very challenging to use in mobile terminals. In addition, the audio quality obtained by decoding is poorer than the quality of the audio obtained based on the signal processing method.

[0069] In the audio encoding and decoding method 2 of the related art, the encoder converts the speech signal into a feature vector through a convolutional network or an autoencoder. Then, the feature vector is encoded. The decoding end is the inverse process of the network at the encoder end. The feature vector obtained by decoding is called to generate a speech signal by calling the corresponding network.

[0070] The applicant found that the above two methods mainly generate feature vectors through audio signal analysis or coding networks, and then encode the feature vectors. However, since deep learning networks are used at both ends of the codec, the computational complexity is very high, and huge computing resources are required to achieve audio codec. The codec efficiency is low, and it cannot be applied to application scenarios with different network bandwidths, and lacks flexibility.

[0071] The embodiments of the present application provide an audio encoding and decoding method, device, electronic device, storage medium and program product, which can be flexibly applied to application scenarios of different network bandwidths, and improve the efficiency and quality of audio encoding and decoding. The following describes an exemplary application of an electronic device for performing audio encoding and decoding provided by an embodiment of the present application. The electronic device for performing audio encoding and decoding provided by an embodiment of the present application can be implemented as various types of user terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, a vehicle-mounted terminal), and can also be implemented as a server. Below, an exemplary application when the electronic device is implemented as a server will be described.

[0072] The audio encoding and decoding method provided in the embodiment of the present application can be executed by the terminal or the server alone or in collaboration. Figure 2 , Figure 2 1 is a schematic diagram of the architecture of the audio codec system 100 provided in an embodiment of the present application, including a terminal 401 and a terminal 402. The terminal 401 is connected to the terminal 402 via a network 300, and the network 300 may be a wide area network or a local area network, or a combination of the two.

[0073] In some embodiments, taking a Voice over Internet Protocol (VoIP) conference system as an example, terminal 401 includes a conference client. When a user uses the conference client, an audio signal is generated. The encoder deployed in terminal 401 performs layered encoding processing on the audio signal to be encoded generated by terminal 401 to obtain code streams corresponding to multiple layers. After that, different transmission priorities are configured for the obtained code streams, and the code streams are transmitted to terminal 402 according to the corresponding transmission priorities. The decoder deployed in terminal 402 performs layered decoding and synthesis processing based on the received code stream to obtain an audio signal, and the decoded audio signal is played through the corresponding conference client in terminal 402.

[0074] In addition to being applicable to conference scenarios, the audio codec system 100 in the embodiment of the present application can also be applied to voice and video chats of instant messaging clients, voice chats of game clients, voice and video chats in online live broadcast rooms, and other scenarios.

[0075] In some embodiments, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals 401 and 402 may be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, car terminals, etc., but are not limited thereto.

[0076] In other embodiments, the embodiments of the present application can be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data computing, storage, processing, and sharing.

[0077] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0078] In some embodiments, see Figure 3 , Figure 3 It is a schematic diagram of the voice communication link provided by the embodiment of the present application. The encoder can be deployed on the uplink client, and the decoder can be deployed on the downlink client. The encoder collects the audio signal of the uplink client, and performs pre-processing enhancement on the audio signal, such as feature extraction processing, etc., encodes the enhanced voice features obtained and performs anti-noise performance improvement processing, and after obtaining the code stream, transmits the code stream to the downlink client where the decoder is located. The decoder in the downlink client decodes the received code stream, and performs anti-noise performance improvement processing, and further enhances and adds sound effects to the decoded result, so as to restore the original audio signal as much as possible and play it back on the downlink client.

[0079] It should be noted that, considering forward compatibility, a transcoder (such as a combination of an NN decoder and a G.722 encoder) can be deployed in the background of the audio coding and decoding system to achieve the effect of interconnection between the new encoder and the existing encoder. For example, if the sender is a new NN encoder and the receiver uses a G.722 decoder, the G.722 decoder at the receiver cannot decode the code stream generated by the NN encoder directly sent by the sender. Therefore, in the background (i.e., the server), the code stream encoded by the NN encoder at the sender can be decoded by the NN decoder to generate a voice signal, and then the G.722 encoder is called to generate a specific code stream and sent to the receiver, so that the receiver can decode it correctly.

[0080] Below, see Figure 4A , Figure 4A is a schematic diagram of the structure of the terminal 401 provided in an embodiment of the present application, Figure 4A The terminal 401 shown includes: at least one processor 410, a memory 430, and at least one network interface 420. The various components in the terminal 401 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 4A Various buses are labeled as bus system 440 .

[0081] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0082] The memory 430 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 430 may optionally include one or more storage devices that are physically remote from the processor 410.

[0083] The memory 430 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 430 described in the embodiments of the present application is intended to include any suitable type of memory.

[0084] In some embodiments, memory 430 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0085] The operating system 431 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0086] The network communication module 432 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0087] In some embodiments, the audio encoding device provided in the embodiments of the present application can be implemented in software. Figure 4A An audio encoding device 433 stored in a memory 430 is shown, which may be software in the form of a program and a plug-in, including the following software modules: a decomposition module 4331, a feature extraction module 4332, a quantization module 4333, an encoding module 4334, and a configuration module 4335. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0088] Below, see Figure 4B , Figure 4B is a schematic diagram of the structure of the terminal 402 provided in an embodiment of the present application, Figure 4B The terminal 402 shown includes: at least one processor 450 , a memory 470 , and at least one network interface 460 . The various components in the terminal 402 are coupled together via a bus system 480 .

[0089] In some embodiments, the audio decoding device provided in the embodiments of the present application can be implemented in software. Figure 4B An audio decoding device 473 stored in a memory 470 is shown, which may be software in the form of a program or plug-in, including the following software modules: a decoding module 4731, an inverse quantization module 4734, a feature reconstruction module 4733, and a synthesis module 4734. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0090] The following will describe the audio encoding and decoding method provided by the embodiment of the present application in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. It can be understood that the following method can be executed separately or collaboratively by the terminal or server described above.

[0091] See also Figure 5A , Figure 5A is a flowchart of the audio encoding method provided in the embodiment of the present application, which will be combined with Figure 5A The steps shown are explained.

[0092] In step 101, an audio signal is decomposed to obtain a low-frequency sub-band signal and a high-frequency sub-band signal.

[0093] As an example, after the audio signal is collected, the audio signal is first decomposed to obtain a low-frequency sub-band signal and a high-frequency sub-band signal.

[0094] In some embodiments, decomposing an audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal can be achieved in the following manner: sampling the audio signal at a first sampling frequency to obtain a sampled signal; wherein the sampled signal includes a plurality of sample points sampled from the audio signal; performing low-pass filtering on the sampled signal, performing a first down-sampling process on the obtained low-pass filtering result, to obtain a low-frequency sub-band signal of a second sampling frequency; performing high-pass filtering on the sampled signal, performing a second down-sampling process on the obtained high-pass filtering result, to obtain a high-frequency sub-band signal of the second sampling frequency.

[0095] As an example, a QMF filter bank can be used to implement the decomposition process. Fig.7D , Fig.7D It is a schematic diagram of the spectrum response corresponding to the QMF filter group provided in the embodiment of the present application.

[0096] The QMF filter bank is a filter pair consisting of a QMF analysis filter and a QMF synthesis filter.

[0097] For the QMF analysis filter, the input audio signal with a sampling rate of Fs can be decomposed into two sub-band signals with a sampling rate of Fs / 2, namely, a QMF low-pass signal and a QMF high-pass signal. Fig.7D As shown, 701 represents the low-pass signal H of the QMF analysis filter. Low The spectral response of (z) is shown in FIG. 702 , where 702 represents the high-pass signal H of the QMF analysis filter. High (z) spectral response.

[0098] As an example, the correlation between the QMF low-pass signal filter coefficients and the QMF high-pass signal filter coefficients can be calculated by the following formula:

[0099] h High (k) = (-1) k h Low (k) Formula 1

[0100] Among them, h High (k) represents the coefficient of the kth point of the high-pass signal, h Low (k) represents the coefficient of the kth point of the low-pass signal.

[0101] The following description is made by taking the first sampling frequency Fs=32000 Hz and the duration of a single frame audio signal as 20 ms as an example. The first sampling frequency may also be 8000 Hz, 16000 Hz, 48000 Hz, etc., which is not limited in the present embodiment.

[0102] First, a frame of audio signal x(n) (i.e., a frame of continuous analog signal) is sampled at Fs=32000 Hz to obtain a sampled signal (i.e., a discrete digital signal). As an example, the sampled signal includes 640 sample points, i.e., 640 sample values, sampled from the audio signal.

[0103] Secondly, the sampling signal of the first sampling frequency (i.e., 32000 Hz) is subjected to low-pass filtering through a 2-channel QMF analysis filter. After obtaining the low-pass filtering result, the low-pass filtering result is subjected to a first downsampling process to obtain a low-frequency subband signal x of the second sampling frequency (i.e., 16000 Hz). LB (n).

[0104] Correspondingly, the sampling signal of the first sampling frequency (i.e., 32000 Hz) is subjected to high-pass filtering through a 2-channel QMF analysis filter. After obtaining the high-pass filtering result, the high-pass filtering result is subjected to a second downsampling process to obtain a high-frequency subband signal x of the second sampling frequency (i.e., 16000 Hz). HB (n).

[0105] Among them, low-pass filtering and high-pass filtering can be performed in parallel to improve filtering efficiency. The effective bandwidth range of the low-frequency sub-band signal obtained by downsampling is 0-8kHz, and the effective bandwidth range of the high-frequency sub-band signal is 8-16kHz. Since the bandwidth is the difference between the highest frequency and the lowest frequency contained in the signal, the bandwidth of the low-frequency sub-band signal and the high-frequency sub-band signal is the same, both 8kHz. After decomposition, both sub-band signals include 320 sample points.

[0106] Through the above manner, the low-frequency sub-band signal and the high-frequency sub-band signal corresponding to the audio signal can be accurately obtained, so as to facilitate further processing based on the accurate sub-band signals.

[0107] In step 102, multiple levels of feature extraction processing are performed based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to the multiple levels.

[0108] As an example, after obtaining the low-frequency subband signal and the high-frequency subband signal, in some embodiments, at least one level of feature extraction processing can be performed based on the low-frequency subband signal and the high-frequency subband signal to obtain the subband signal features corresponding to each level.

[0109] As an example, one level (ie, the first level) of feature extraction processing may be performed based on the low frequency subband signal. Fig. 7A , Fig. 7A It is a schematic diagram of performing one level of encoding and decoding provided in an embodiment of the present application.

[0110] like Fig. 7A As shown, the low-frequency subband signal x is obtained LB (n), first, the low-frequency subband signal x LB (n) Perform the first level of analysis (i.e., feature extraction processing) to obtain the first low-frequency subband signal feature F LB (n).

[0111] Secondly, the first low-frequency subband signal feature F LB (n) Perform quantization coding processing to obtain a code stream corresponding to the first level.

[0112] Next, the code stream corresponding to the first level is transmitted to the decoding end, and the decoding end decodes the code stream to obtain the decoding result F′ LB (n), the decoding result F′ LB (n) Perform synthesis processing of the first layer and call the synthesis filter to obtain the synthesis result x′ LB (n) Perform upsampling processing to obtain an estimated value of the low-frequency signal in the original audio signal.

[0113] Correspondingly, a level of encoding and decoding processing can also be performed based on the high frequency sub-band signal. The processing flow is similar to Fig. 7A The process shown is similar and will not be repeated here.

[0114] In some embodiments, two levels of feature extraction processing may be performed based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to each level.

[0115] As an example, the first level feature extraction process can be performed based on the low frequency sub-band signal, and the second level feature extraction process can be performed based on the high frequency sub-band signal. Figure 7B , Figure 7B It is a schematic diagram of two-level encoding and decoding provided in an embodiment of the present application.

[0116] At the encoding end, the low-frequency sub-band signal is analyzed at the first level to obtain the first low-frequency sub-band signal feature F LB (n), and perform a second-level analysis on the high-frequency sub-band signal to obtain the first high-frequency sub-band signal feature F HB (n). Next, LB (n) and F HB (n) Perform quantization encoding processing respectively to obtain a code stream corresponding to the first layer and a code stream corresponding to the second layer, and transmit the code streams of the two layers to a decoding end.

[0117] At the decoding end, first, the received two-level code streams are decoded to obtain the code stream decoding results corresponding to the first level and the code stream decoding results corresponding to the second level respectively; then, the first level synthesis processing is performed based on the code stream decoding results corresponding to the first level to obtain the synthesis result x′ of the first level LB (n), perform synthesis processing on the second level based on the bitstream decoding result corresponding to the second level, and obtain the synthesis result x′ of the second level HB (n); Finally, call the synthesis filter x′ LB (n) and x′ HB (n) performs up-sampling processing, and merges the two up-sampling results to obtain the estimated value x′(n) of the original audio signal.

[0118] The following will describe a method of performing two-level feature extraction processing based on low-frequency sub-band signals and high-frequency sub-band signals in conjunction with the accompanying drawings.

[0119] See also Figure 5B , Figure 5B This is a flowchart of the audio encoding method provided by the embodiment of the present application. Figure 5A , Figure 5B Step 102 in the above example can be implemented by steps 1021 and 1022. Figure 5B Steps 1021 to 1022 are shown for explanation.

[0120] In step 1021, the first level feature extraction process is performed in the following manner: a first feature extraction process is performed based on the low frequency sub-band signal to obtain a first low frequency sub-band signal feature.

[0121] As an example, in the case of performing the feature extraction processing of the first level and the second level, based on the low-frequency subband signal x LB (n) Perform the first feature extraction process at the first level to obtain the first low-frequency subband signal feature F LB (n), wherein the dimension of the first low-frequency sub-band signal feature is smaller than the dimension of the low-frequency sub-band signal.

[0122] Here, the first feature extraction process can be implemented by a signal processing-based method, such as extracting waveform features or parameter features of the low-frequency sub-band signal; the first feature extraction process can also be implemented by a neural network method, and the neural network includes but is not limited to an autoencoder (AutoEncoder, AE), a fully connected network (Full-Connection, FC), a long short-term memory network (Long Short-Term Memory, LSTM), a convolutional network (Convolutional Neural Network, CNN) + a long short-term memory network (LSTM), and a dilated convolutional network (Dilated CNN).

[0123] It should be noted that in the embodiment of the present application, the neural network (such as the hole convolution network) of the encoding end and the decoding end can be jointly trained by collecting data to obtain the optimal parameters. Since there are multiple open source platforms for neural networks and deep learning in the relevant technology, based on these open source platforms, developers only need to prepare data and set the corresponding network structure to train the neural network. After the server completes the training of the neural network, the trained neural network can be put into use. The specific training process is not repeated here. The embodiment of the present application assumes that the optimal parameters of the trained neural network are used for feature extraction processing.

[0124] As an example, the first feature extraction process is described by taking the neural network as a dilated convolutional network. Below, the dilated convolutional network is first described.

[0125] See also Fig. 8A , Fig. 8A Schematic diagram of a common convolutional network and a dilated convolutional network provided in the embodiment of the present application. The purpose of proposing a dilated convolutional network is to increase the receptive field while keeping the size of the feature map unchanged, avoiding errors caused by upsampling and downsampling. Fig. 8A As shown, although the convolution kernel size (Kernelsize) in the ordinary convolution network (i.e., the form shown in 801) and the hollow convolution network (i.e., the form shown in 802) is 3x3, the receptive field of the ordinary convolution network is only 3, and the dilation rate (Dilationrate) is 1; while the receptive field of the hollow convolution network reaches 5, and the dilation rate is 2.

[0126] During the convolution process, the convolution kernel can be Fig. 8AThe movement on the plane in the convolution involves the shift rate (Stride rate). For example, if the convolution kernel shifts 1 grid each time, the shift rate is 1. In addition, the number of channels in the convolution process indicates how many convolution kernels are used to perform the convolution process. Generally, the more channels there are, the more comprehensive the analysis of the signal is and the higher the accuracy is; however, the more channels there are, the higher the computational complexity is. For example, a 1x320 feature vector can be convolved with 24 channels to obtain a 24*320 feature vector. In the convolution process, the size of the appropriate dilated convolution kernel (for example, for audio signals, the dilated convolution kernel is generally set to 1x3), the expansion rate, the shift rate and the number of channels can be defined according to the actual application needs.

[0127] The following describes how to use a dilated convolutional network to perform the first feature extraction process.

[0128] See also Figure 5B , Figure 5B Step 1021 in the above example can be implemented by steps 10211 to 10214. Figure 5B Steps 10211 to 10214 are shown for illustration.

[0129] In step 10211, a first convolution process is performed on the low-frequency sub-band signal to obtain a first convolution feature.

[0130] See also Figure 8B , Figure 8B It is a schematic diagram of the structure of a neural network for performing a first feature extraction process provided in an embodiment of the present application.

[0131] like Figure 8B As shown, first based on the low-frequency subband signal x LB (n) Calling a 24-channel convolutional layer (e.g., a causal convolutional layer) to perform the first convolution process, thereby converting the 1*320 low-frequency subband signal x LB (n), the first convolution feature expanded to 24*320.

[0132] In step 10212, a first pooling process is performed on the first convolution feature to obtain a first pooling feature.

[0133] As an example, see Figure 8B After obtaining the first convolution feature, the pooling layer is called based on the first convolution feature of 24*320 to perform the first pooling process. For example, the pooling factor can be set to 2, and the activation function can be set to the linear rectified unit ReLU function. After the first pooling process, the first pooling feature of 24*160 is obtained.

[0134] In step 10213, a third downsampling process is performed on the first pooled feature to obtain a third downsampling feature, wherein the third downsampling process includes multiple cascaded downsamplings.

[0135] As an example, see Figure 8B After obtaining the first pooling feature, the three cascaded downsampling layers are called to perform the third downsampling process based on the first pooling feature.

[0136] Among them, these three downsampling layers correspond to three different downsampling factors. As an example, the downsampling factor of the first downsampling layer is 4 and the channels are 48; the downsampling factor of the second downsampling layer is 5 and the channels are 96; the downsampling factor of the third downsampling layer is 8 and the channels are 192. Therefore, after the third downsampling process of these three downsampling layers, the first pooling feature of 24*160 is converted into the third downsampling features of 48*40, 96*8, and 192*1 respectively.

[0137] During the third downsampling process, one or more dilated convolution processes can be performed based on the first pooling feature. For example, the size of each dilated convolution kernel can be set to 1*3, the shift rate can be set to 1, and the expansion rate of the dilated convolution process can be set to 3. The embodiment of the present application does not limit this.

[0138] In step 10214, a second convolution process is performed on the third down-sampled feature to obtain a first low-frequency sub-band signal feature.

[0139] As an example, see Figure 8B After obtaining the third down-sampling feature of 192*1, a convolution layer with 64 channels (such as a causal convolution layer) is called based on the third down-sampling feature of 192*1 to perform a second convolution process to obtain a 64-dimensional feature vector, i.e., the first low-frequency subband signal feature F LB (n).

[0140] like Figure 8B As shown, the low-frequency subband signal x LB The dimension of (n) is 320, and the first low-frequency subband signal feature F LB The dimension of (n) is 64. Therefore, the first feature extraction process plays a role of dimensionality reduction, that is, data compression is performed.

[0141] Through the above method, the first low-frequency sub-band signal characteristics can be quickly and efficiently obtained by means of a neural network, which not only can obtain accurate first frequency sub-band signal characteristics, but also can achieve data compression.

[0142] In step 1022, the second level feature extraction process is performed in the following manner: a second feature extraction process is performed based on the high frequency sub-band signal to obtain the first high frequency sub-band signal feature.

[0143] As an example, the second feature extraction process can be implemented based on the frequency band extension method to obtain the first high frequency sub-band signal feature corresponding to the high frequency sub-band signal, wherein the dimension of the first high frequency sub-band signal feature is smaller than the dimension of the high frequency sub-band signal.

[0144] See also Fig. 7E , Fig. 7E Schematic diagram of the frequency band extension provided by the embodiment of the present application. Fig. 7E As shown in the figure, first, a core layer of encoding is implemented on the original ultra-wideband signal at a low sampling frequency to obtain a reconstructed wideband signal; second, the spectrum of the low-frequency part of the reconstructed wideband signal is copied to the high-frequency part of the ultra-wideband; finally, the gain of the copied high-frequency spectrum is controlled according to the boundary information recorded in advance (such as information describing the energy correlation between high frequency and low frequency). Generally, only a bit rate of 1-2kbps is required to achieve the effect of doubling the sampling rate.

[0145] See also Figure 5C , Figure 5C This is a flowchart of the audio encoding method provided by the embodiment of the present application. Figure 5B , Figure 5C Step 1022 in the above example can be implemented by steps 10221 to 10224. Figure 5C Steps 10221-10224 are shown for illustration.

[0146] In step 10221, frequency domain transform processing is performed based on the first number of sample points to obtain a first number of transform coefficients.

[0147] As an example, when performing the second feature extraction process based on the high frequency sub-band signal, the number of sample points included in the high frequency sub-band signal is first determined, for example, the high frequency sub-band signal includes 320 sample points, then the first number is 320. Then, frequency domain transformation processing is performed based on the 320 sample points of the high frequency sub-band signal to obtain 320 transformation coefficients.

[0148] Here, the frequency domain transform processing includes discrete cosine transform (DCT) processing, modified discrete cosine transform (MDCT) processing, and fast Fourier transform (FFT) processing.

[0149] In some embodiments, frequency domain transform processing is performed based on a first number of sample points to obtain a first number of transform coefficients, which can be achieved in the following way: obtaining a reference frame high frequency subband signal corresponding to the reference frame audio signal; based on the first number of sample points in the reference frame high frequency subband signal and the first number of sample points in the high frequency subband signal, discrete cosine transform processing is performed on the first number of sample points included in the high frequency subband signal to obtain transform coefficients corresponding to the first number of sample points included in the high frequency subband signal.

[0150] As an example, first, the next frame audio signal or the previous frame audio signal of the current frame audio signal is obtained, and the obtained audio signal is used as the reference frame audio signal; then, the reference frame audio signal is decomposed to obtain a reference frame high frequency subband signal, and the method for obtaining the reference frame high frequency subband signal is similar to the method for obtaining the current frame high frequency subband signal, which will not be repeated here. Among them, the reference frame high frequency subband signal also includes a first number of sample points, that is, 320 sample points.

[0151] After obtaining 320 sample points of the high frequency sub-band signal of the reference frame, the 320 sample points of the high frequency sub-band signal of the reference frame are merged with the 320 sample points of the high frequency sub-band signal of the current frame to obtain 640 sample points.

[0152] Based on the 640 sample points, MDCT processing is performed. For a 50% time domain overlapping window, the MDCT transform coefficients corresponding to the 320 sample points corresponding to the high frequency sub-band signal of the current frame can be calculated.

[0153] By determining 320 MDCT transform coefficients corresponding to the high frequency sub-band signal of the current frame in a manner based on MDCT processing, key information of the high frequency sub-band signal can be accurately extracted.

[0154] In step 10222, the first number of transform coefficients are divided into a plurality of first sub-bands.

[0155] As an example, after obtaining 320 transform coefficients, the 320 transform coefficients are divided into a plurality of first sub-bands, for example, 8 first sub-bands, where the first sub-band includes a group of transform coefficients corresponding to adjacent sample points.

[0156] Here, the first sub-band can be evenly divided so that each first sub-band includes the same number of transformation coefficients. For example, in the case of uniform division into 8 first sub-bands, each first sub-band includes 320 / 8, that is, 40 transformation coefficients. The first sub-band can also be non-uniformly divided, such as the first sub-band with a lower frequency contains fewer transformation coefficients (that is, the frequency resolution is higher) and the first sub-band with a higher frequency contains more transformation coefficients (that is, the frequency resolution is lower).

[0157] In some embodiments, according to Nyquist sampling theorem (i.e., if the original signal is to be restored from the sampled signal without distortion, the sampling frequency should be greater than 2 times the highest frequency of the original signal. When the sampling frequency is less than 2 times the highest frequency of the original signal, the spectrum of the signal is aliased. When the sampling frequency is greater than 2 times the highest frequency of the original signal, the spectrum of the signal is not aliased), since the 320 transform coefficients corresponding to the high-frequency sub-band signal of the current frame correspond to a spectrum of 8-16kHz, however, general ultra-wideband voice communication does not necessarily require a spectrum of 16kHz. Therefore, if the maximum spectrum is set to 14kHz, only the first 240 transform coefficients need to be considered; correspondingly, if the first sub-band is evenly divided, 6 first sub-bands can be obtained.

[0158] In step 10223, the following processing is performed for each first subband: averaging processing is performed based on the second number of transform coefficients in the first subband to obtain a first average energy corresponding to the first subband, and the first average energy is determined as the first subband spectral envelope corresponding to the first subband.

[0159] As an example, taking 8 first sub-bands obtained by evenly dividing 320 transform coefficients as an example, in this case, the number of transform coefficients included in each first sub-band, that is, the second number, is 40.

[0160] For each of the 8 first subbands, averaging processing is performed based on 40 transform coefficients in the first subband to obtain a first average energy corresponding to the first subband, and the first average energy is used as the first subband spectral envelope corresponding to the first subband.

[0161] As an example, the sum of squares of 40 transform coefficients included in the first sub-band may be calculated, and the ratio of the obtained sum of squares to 40 may be determined as the first average energy, thereby obtaining the first sub-band spectral envelope.

[0162] In this way, first sub-band spectral envelopes corresponding to the eight first sub-bands can be obtained.

[0163] In step 10224, first sub-band spectrum envelopes corresponding to the plurality of first sub-bands are determined as first high-frequency sub-band signal features.

[0164] As an example, after obtaining the first sub-band spectral envelopes corresponding to the eight first sub-bands respectively, the first sub-band spectral envelopes corresponding to the eight first sub-bands respectively are determined as the first high frequency sub-band signal features F HB (n).

[0165] By the above method, the high frequency sub-band signal with a dimension of 320 can be converted into the first high frequency sub-band signal feature F of 8 dimensions. HB(n), so only a small amount of data is needed to represent the high-frequency sub-band signal, which is beneficial to improving the coding efficiency.

[0166] As an example, the first level feature extraction process can be performed based on the low frequency sub-band signal, the second level feature extraction process can be performed based on the high frequency sub-band signal, and the third level feature extraction process can be performed based on the low frequency sub-band signal and the high frequency sub-band signal. Figure 7C , Figure 7C It is a schematic diagram of three-level encoding and decoding provided in an embodiment of the present application.

[0167] At the encoding end, the low-frequency sub-band signal is analyzed at the first level to obtain the first low-frequency sub-band signal feature F LB (n); Perform a second-level analysis on the high-frequency sub-band signal to obtain the first high-frequency sub-band signal feature F HB (n); Based on the low-frequency sub-band signal and the first low-frequency sub-band signal characteristics, the third level of low-frequency analysis is performed to obtain the second low-frequency sub-band signal characteristics F LB，e (n), perform the third level of high frequency analysis based on the high frequency sub-band signal to obtain the second high frequency sub-band signal feature F HB,e (n). Next, LB (n), F HB (n), F LB,e (n) and F HB,e (n) Perform quantization encoding processing respectively to obtain a code stream corresponding to the first level, a code stream corresponding to the second level, and a code stream corresponding to the third level, and transmit the code streams corresponding to the three levels to a decoding end.

[0168] At the decoding end, first, the received three-level code streams are decoded to obtain the code stream decoding results corresponding to the first level, the code stream decoding results corresponding to the second level, and the code stream decoding results corresponding to the third level respectively; then, the first-level synthesis processing is performed on the code stream decoding results corresponding to the first level to obtain the first-level synthesis result x′ LB (n), perform the third-level low-frequency synthesis processing on the bitstream decoding result corresponding to the low-frequency part of the third level to obtain the third-level low-frequency synthesis result x′ LB,e (n), and for x′ LB (n) and x′ LB,e (n) performing a summation process to obtain a summation result of the low-frequency part; performing a high-frequency synthesis process of the third level based on the code stream decoding result corresponding to the second level and the code stream decoding result of the high-frequency part corresponding to the third level to obtain a high-frequency synthesis result x′ HB (n); Finally, call the synthesis filter to sum the low-frequency part and x′ HB(n) performs up-sampling processing, and merges the two up-sampling results to obtain the estimated value x′(n) of the original audio signal.

[0169] The following will describe a method of performing three-level feature extraction processing based on low-frequency sub-band signals and high-frequency sub-band signals in conjunction with the accompanying drawings.

[0170] See also Figure 5D , Figure 5D It is a flowchart of the audio encoding method provided in an embodiment of the present application. Figure 5A Step 102 in the above example can be implemented by steps 1023 and 1024. Figure 5D Steps 1023-1024 are shown for explanation.

[0171] In step 1023, the feature extraction process of the third level is performed in the following manner: a third feature extraction process is performed based on the low frequency sub-band signal and the first low frequency sub-band signal feature to obtain the second low frequency sub-band signal feature.

[0172] As an example, when the multiple levels also include a third level, the feature extraction process of the first level and the second level is the same as the above-mentioned step 1021-step 1022, which will not be repeated here.

[0173] Next, the third feature extraction process is described by taking the implementation of the third feature extraction process through a neural network as an example.

[0174] See also Figure 5D , Figure 5D Step 1023 in the above example can be implemented by steps 10231 to 10235. Figure 5D Steps 10231-10235 are shown for illustration.

[0175] In step 10231, the low-frequency sub-band signal and the first low-frequency sub-band signal feature are spliced ​​to obtain a spliced ​​feature.

[0176] See also Figure 8C , Figure 8C It is a schematic diagram of the structure of a neural network for performing a third feature extraction process provided in an embodiment of the present application.

[0177] like Figure 8C As shown, firstly, the low frequency subband signal x LB (n) (dimension is 320) and the first low-frequency subband signal feature F LB (n) (dimension is 64) is concatenated to obtain a 384-dimensional concatenated feature.

[0178] In step 10232, a third convolution process is performed on the concatenated features to obtain a third convolution feature.

[0179] As an example, after obtaining the splicing feature, a convolution layer with a channel of 24 (eg, a causal convolution layer) is called based on the splicing feature to perform a third convolution process to obtain a 24*384 third convolution feature.

[0180] In step 10233, a second pooling process is performed on the third convolution feature to obtain a second pooling feature.

[0181] As an example, see Figure 8C After obtaining the third convolution feature, the pooling layer is called based on the third convolution feature to perform the second pooling process. When the pooling factor is 2, a 24*192 second pooling feature is obtained.

[0182] In step 10234, the second pooled feature is subjected to a fourth downsampling process to obtain a fourth downsampling feature.

[0183] As an example, see Figure 8C After obtaining the second pooling feature, the three cascaded downsampling layers are called based on the second pooling feature to perform the fourth downsampling process.

[0184] Among them, these three downsampling layers correspond to three different downsampling factors. As an example, the downsampling factor of the first downsampling layer is 4 and the channel is 48; the downsampling factor of the second downsampling layer is 6 and the channel is 96; the downsampling factor of the third downsampling layer is 8 and the channel is 192. Therefore, after the fourth downsampling process of these three downsampling layers, the second pooling feature of 24*192 is converted into the fourth downsampling feature of 48*48, 96*8, and 192*1 respectively.

[0185] In step 10235, a fourth convolution process is performed on the fourth down-sampling feature to obtain a second low-frequency sub-band signal feature.

[0186] As an example, see Figure 8C After obtaining the fourth down-sampling feature of 192*1, a convolution layer with a channel of 28 (such as a causal convolution layer) is called based on the fourth down-sampling feature of 192*1 to perform the fourth convolution process to obtain a 28-dimensional feature vector, i.e., the second low-frequency subband signal feature F LB,e (n).

[0187] Since the second low-frequency subband signal characteristic F LB,e The dimension of (n) is 28, and the first low-frequency subband signal feature F LB The dimension of (n) is 64, so the dimension of the second low-frequency sub-band signal feature is smaller than the dimension of the first low-frequency sub-band signal feature.

[0188] The purpose of the third feature extraction process at the third level is to further extract the features of the low-frequency sub-band signal. It should be noted that the second low-frequency sub-band signal feature obtained by the third feature extraction process reflects the residual between the reconstructed signal of the first low-frequency sub-band signal feature at the decoding end and the original low-frequency sub-band signal, and therefore, the second low-frequency sub-band signal feature can also be called a low-frequency sub-band signal residual feature.

[0189] Through the above method, the second low-frequency sub-band signal characteristics can be quickly and efficiently obtained by means of a neural network, which not only can obtain accurate second frequency sub-band signal characteristics, but also can achieve data compression.

[0190] In step 1024, a fourth feature extraction process is performed based on the first high frequency sub-band signal feature to obtain a second high frequency sub-band signal feature.

[0191] As an example, the fourth feature extraction process of the third level can be implemented based on the second feature extraction process of the second level, that is, the fourth feature extraction process is performed based on the first high frequency sub-band signal feature obtained by the second feature extraction process to obtain the second high frequency sub-band signal feature.

[0192] See also Figure 5E , Figure 5E This is a flowchart of the audio encoding method provided by the embodiment of the present application. Figure 5D , Figure 5E Step 1024 in the above example can be implemented by steps 10241 to 10243. Figure 5E Steps 10241-10243 are shown for illustration.

[0193] In step 10241, a third number of transform parameters is selected from the second number of transform parameters in the first subband, and the third number of transform coefficients is determined as the second subband.

[0194] As an example, in the process of obtaining the first high frequency sub-band signal feature, 8 first sub-bands are obtained, and for each first sub-band, a third number of transformation parameters are selected from the 40 transformation parameters included in the first sub-band, and the selected third number of transformation coefficients are determined as the second sub-band. Wherein, the third number is half of the second number, and when the second number is 40, the third number is 20.

[0195] Here, since there are 8 first sub-bands and one second sub-band is determined according to each first sub-band, 8 second sub-bands can be obtained in the end.

[0196] In step 10242, averaging processing is performed based on the third number of transform coefficients in the second subband to obtain a second average energy corresponding to the second subband, and the second average energy is determined as the second subband spectral envelope corresponding to the second subband.

[0197] As an example, after obtaining 8 second sub-bands, averaging processing is performed based on the 20 transform coefficients in the second sub-band. For example, the sum of the squares of the 20 transform coefficients included in the second sub-band can be calculated, and the ratio of the obtained sum of squares to 20 can be determined as the second average energy, thereby obtaining the second sub-band spectral envelope.

[0198] In this way, second sub-band spectrum envelopes corresponding to the eight second sub-bands can be obtained.

[0199] In step 10243, the second sub-band spectrum envelope corresponding to each second sub-band is determined as the second high-frequency sub-band signal feature.

[0200] As an example, after obtaining the second sub-band spectrum envelopes corresponding to the eight second sub-bands respectively, the second sub-band spectrum envelopes corresponding to the eight second sub-bands respectively are determined as the second high frequency sub-band signal features F HB，e (n).

[0201] Since the second feature extraction process maps the 320-dimensional high-frequency sub-band signal to an 8-dimensional first high-frequency sub-band signal feature, that is, the transform coefficients corresponding to every 40 sample points share a first sub-band, this mapping method has a certain information loss. Therefore, in the fourth feature extraction process, the spectral envelope of the high-frequency sub-band signal is further transmitted for each first sub-band, that is, it is configured that the transform coefficients corresponding to every 20 sample points share a second sub-band. In this way, the spectral energy of the transform coefficients corresponding to every 20 sample points can be adjusted once, so that the obtained second high-frequency sub-band signal feature has a higher resolution and can more accurately reflect the characteristics of the high-frequency sub-band signal.

[0202] In step 103, the sub-band signal features corresponding to each layer are quantized to obtain index values ​​of the sub-band signal features.

[0203] As an example, after obtaining the sub-band signal features corresponding to each layer, the sub-band signal features corresponding to each layer are quantized to obtain index values ​​of the sub-band signal features corresponding to each layer.

[0204] Here, the quantization process is used to digitize the subband signal characteristics on the amplitude axis, and generally includes a vector quantization process or a scalar quantization process.

[0205] The principle of vector quantization processing is to combine multiple scalar data into a vector, divide the vector space as a complete area into multiple sub-areas, and determine a representative vector in each sub-area. If the sub-band signal feature falls into a sub-area during quantization, the representative vector corresponding to the sub-area is used to replace the sub-band signal feature, that is, the sub-band signal feature is quantized into the representative vector.

[0206] The principle of scalar quantization processing is to take the entire dynamic range as an interval and divide it into multiple sub-intervals. A representative value is determined for each sub-interval. If the sub-band signal feature falls into a sub-interval during quantization, the representative value corresponding to the sub-interval is used to replace the sub-band signal feature, that is, the sub-band signal feature is quantized to the representative value.

[0207] As an example, in order to improve the quantization efficiency, the quantization process can be completed by querying a set quantization table for the sub-band signal characteristics corresponding to each level.

[0208] In step 104, the index value of the sub-band signal feature is encoded to obtain a code stream corresponding to the layer.

[0209] As an example, after obtaining the index value of the sub-band signal feature corresponding to each layer, the index value of the sub-band signal feature is encoded, for example, entropy encoded, to obtain a code stream corresponding to each layer.

[0210] See also Fig. 5F , Fig. 5F This is a flowchart of the audio encoding method provided by the embodiment of the present application. Figure 5A , Fig. 5F Step 104 in the above example can be implemented by steps 1041 to 1044. Fig. 5F Steps 1041 to 1044 are shown for illustration.

[0211] In step 1041, the index value of the first low-frequency signal feature is encoded to obtain a first code stream corresponding to the first level.

[0212] As an example, if feature extraction processing is performed on the first and second levels, and the subband signal feature corresponding to the first level is the first low-frequency subband signal feature, then the index value of the first low-frequency signal feature is encoded to obtain the first code stream corresponding to the first level.

[0213] As an example, in the first level, assuming that the average bit rate of quantizing one parameter per frame of audio signal is 2.5 bits, since the duration of one frame is 20ms, which is equivalent to 0.125 bits / ms, after unit conversion, it is 125 bits / t / s. Since the first low-frequency subband signal feature is a 64-dimensional feature, it is equivalent to containing 64 parameters. Therefore, the average bit rate for encoding these 64 parameters is 64*125 bits / t / s, that is, 8000 bits / t / s, that is, 8kbps. Therefore, the first bit rate corresponding to the first code stream is 8kbps.

[0214] In step 1042, the index value of the first high-frequency signal feature is encoded to obtain a second code stream corresponding to the second level.

[0215] As an example, since the subband signal feature corresponding to the second level is the first high frequency subband signal feature, the index value of the first high frequency signal feature is encoded to obtain the second code stream corresponding to the second level.

[0216] As an example, in the second level, assuming that the average bit rate of quantizing one parameter per frame of audio signal is 5 bits, since the first high frequency subband signal feature is an 8-dimensional feature, which is equivalent to containing 8 parameters, the average bit rate for encoding these 8 parameters is 2 kbps. Therefore, the second bit rate corresponding to the second code stream is 2 kbps.

[0217] Therefore, the bit rate of the first code stream is greater than the bit rate corresponding to the second code stream. Since the bit rate is positively correlated with the decoding quality index of the corresponding code stream, the decoding quality index of the first code stream corresponding to the first level is higher, and the decoding quality index of the second code stream corresponding to the second level is relatively lower.

[0218] In step 1043, the index value of the second low-frequency signal feature is encoded to obtain a third code stream corresponding to the third level.

[0219] As an example, if feature extraction processing of the third level is also performed, since the subband signal features corresponding to the third level include the second low-frequency subband signal features, the index value of the second low-frequency signal features is encoded to obtain a third code stream corresponding to the third level.

[0220] As an example, in the third level, assuming that the average bit rate of quantizing a low-frequency subband signal feature parameter per frame of audio signal is 2.5 bits, since the second high-frequency subband signal feature is a 28-dimensional feature, which is equivalent to containing 28 parameters, the average bit rate for encoding these 28 parameters is 3.5 kbps. Therefore, the third bit rate corresponding to the third bit stream is 3.5 kbps.

[0221] In step 1044, the index value of the second high-frequency signal feature is encoded to obtain a fourth code stream corresponding to the third level.

[0222] As an example, since the subband signal characteristics corresponding to the third level also include the second high frequency subband signal characteristics, the index value of the second high frequency signal characteristics is encoded to obtain a fourth code stream corresponding to the third level.

[0223] As an example, in the third level, assuming that the average bit rate of quantizing a high-frequency sub-band signal feature parameter per frame of audio signal is 5 bits, since the second high-frequency sub-band signal feature is an 8-dimensional feature, which is equivalent to containing 8 parameters, the average bit rate for encoding these 8 parameters is 2 kbps. Therefore, the fourth bit rate corresponding to the fourth bit stream is 2 kbps.

[0224] Therefore, the first bit rate of the first code stream (i.e., 8 kbps) is greater than the third bit rate of the third code stream (i.e., 3.5 kbps), the third bit rate of the third code stream (i.e., 3.5 kbps) is greater than the second bit rate of the second code stream (i.e., 2 kbps), and the second bit rate of the second code stream (i.e., 2 kbps) is equal to the fourth bit rate of the fourth code stream (i.e., 2 kbpps).

[0225] Since the bit rate is positively correlated with the decoding quality index of the corresponding bit stream, the decoding quality index of the first bit stream corresponding to the first level is higher, the decoding quality index of the third bit stream corresponding to the third level is second, and the decoding quality index of the second bit stream corresponding to the second level and the fourth bit stream corresponding to the third level are relatively low.

[0226] Through the above-mentioned layered encoding method, different code rates can be used to progressively encode the characteristics corresponding to different levels. Since each level only needs to encode part of the sub-band signal characteristics, the coding efficiency can be improved, the coding complexity can be reduced, and the flexibility of selecting different coding methods according to different sub-band signal characteristics is provided.

[0227] In step 105, corresponding transmission priorities are configured for the code streams corresponding to the multiple layers.

[0228] As an example, after obtaining the code stream corresponding to each layer, the corresponding transmission priority is configured for the code streams of different layers. The transmission priority is positively correlated with the decoding quality index of the code stream corresponding to the layer, that is, the higher the decoding quality index of the code stream, the higher the corresponding transmission priority is, and it can be transmitted first when the bandwidth is limited.

[0229] As an example, the transmission priority can be configured through the forward error correction (FEC) mechanism, which improves the stability and quality of the transmitted data through redundant transmission. Here, a code stream with a higher transmission priority can be configured with a larger FEC redundancy multiple, and a code stream with a lower transmission priority can be configured with a smaller FEC redundancy multiple.

[0230] See also Figure 5G , Figure 5G This is a flowchart of the audio encoding method provided by the embodiment of the present application. Figure 5A , Figure 5G Step 105 in the above example can be implemented by steps 1051 to 1054. Figure 5G Steps 1051 to 1054 are shown for illustration.

[0231] In step 1051, a first transmission priority is configured for a first code stream corresponding to a first layer.

[0232] As an example, if the first and second layers are coded, and the first code stream corresponding to the first layer is obtained by coding the index value of the first low-frequency signal feature, then the first transmission priority is configured for the first code stream corresponding to the first layer.

[0233] In step 1052, a second transmission priority is configured for the second code stream corresponding to the second layer.

[0234] As an example, if the second level encoding process is performed, and the second code stream corresponding to the second level is obtained by encoding the index value of the first high-frequency signal feature, then the second transmission priority is configured for the second code stream corresponding to the second level.

[0235] Since the first bit rate corresponding to the first bit stream is greater than the second bit rate corresponding to the second bit stream, and the bit rate is positively correlated with the decoding quality index corresponding to the bit stream, a higher first transmission priority is configured for the first bit stream, and a second transmission priority lower than the first transmission priority is configured for the second bit stream.

[0236] In step 1053, a third transmission priority is configured for the third code stream corresponding to the third layer.

[0237] As an example, if a third level of encoding is performed, and the third level corresponds to a third bitstream, which is obtained by encoding the index value of the second low-frequency signal feature, then a third transmission priority is configured for the third bitstream corresponding to the third level.

[0238] In step 1054, a fourth transmission priority is configured for the fourth code stream corresponding to the third layer.

[0239] As an example, the third level also corresponds to a fourth code stream, which is obtained by encoding the index value of the second high-frequency signal feature. Then, the fourth transmission priority is configured for the fourth code stream corresponding to the third level.

[0240] Since the first code rate of the first code stream is greater than the third code rate of the third code stream, the third code rate of the third code stream is greater than the second code rate of the second code stream, the second code rate of the second code stream is equal to the fourth code rate of the fourth code stream, and the code rate is positively correlated with the decoding quality index corresponding to the code stream. Therefore, correspondingly, the first transmission priority is higher than the third transmission priority, the third transmission priority is higher than the second transmission priority, and the second transmission priority is the same as the fourth transmission priority.

[0241] By configuring the corresponding transmission priorities for the code streams corresponding to different levels, it can be ensured that the more important code streams can be transmitted first when the bandwidth is limited, thereby improving the flexibility of data transmission.

[0242] In the embodiment of the present application, the subband signal characteristics of the audio subband signal at each layer are obtained in a hierarchical manner, and the subband signal characteristics corresponding to each layer are encoded in a hierarchical manner. In this way, each layer only needs to encode specific subband signal characteristics, and there is no need to encode the characteristics of the entire audio signal. This not only improves the efficiency of encoding and decoding, but also reduces the encoding and decoding complexity of each layer. According to the importance of code streams at different layers to the decoding quality, different transmission priorities are flexibly configured for code streams at different layers to ensure that more important code streams are transmitted first, which can be suitable for application scenarios with different network bandwidths.

[0243] The audio decoding method provided by the embodiment of the present application will be described below in conjunction with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. It can be understood that the following method can be executed separately or collaboratively by the terminal or server described above.

[0244] See also Fig. 6A , Fig. 6A is a flowchart of the audio decoding method provided in the embodiment of the present application, which will be combined with Fig. 6A The steps shown are explained.

[0245] In step 201, code streams corresponding to multiple layers are decoded to obtain index values ​​of the code streams corresponding to each layer.

[0246] As an example, after receiving the code streams corresponding to the multiple layers respectively, the decoding end performs decoding processing on the code streams corresponding to the multiple layers respectively, so as to obtain the index value of the code stream corresponding to each layer.

[0247] Here, if the decoding end receives a code stream corresponding to one layer, the decoding obtains the index value of the code stream corresponding to one layer. If the decoding end receives code streams corresponding to multiple layers, the decoding obtains the index values ​​of the code streams corresponding to the multiple layers respectively.

[0248] Different levels correspond to different transmission priorities, and the transmission priority is positively correlated with the decoding quality index of the code stream corresponding to the level.

[0249] In step 202, the index values ​​of the code stream corresponding to each layer are inversely quantized to obtain the subband signal features corresponding to each layer.

[0250] As an example, after obtaining the index value of the code stream corresponding to each layer, the sub-band signal feature corresponding to the index value of the code stream can be obtained by querying the quantization table, thereby implementing inverse quantization processing.

[0251] As an example, if the index value of the code stream corresponding to each level from the first level to the third level is obtained, then after inverse quantization processing, the first low-frequency subband signal feature F′ corresponding to the first level is obtained. LB (n), the first high frequency subband signal feature F′ corresponding to the second level HB (n), the second low-frequency subband signal feature F′ corresponding to the third level LB,e (n), the second high frequency subband signal feature F′ corresponding to the third level HB,e (n).

[0252] In step 203, feature reconstruction processing is performed on the sub-band signal features corresponding to each layer to obtain the sub-band signals corresponding to each layer.

[0253] As an example, after obtaining the sub-band signal features corresponding to each layer, feature reconstruction processing is performed on the sub-band signal features corresponding to each layer, so as to obtain the sub-band signals corresponding to each layer.

[0254] As an example, if the encoding end only performs feature extraction processing on one level, and finally encodes a code stream corresponding to one level, then after the decoding end receives the code stream corresponding to the one level, the code stream corresponding to the one level is decoded to obtain the index value of the corresponding code stream. Then, the index value of the code stream is inversely quantized to obtain the subband signal feature corresponding to the one level. Feature reconstruction processing is then performed based on the subband signal feature to obtain the corresponding subband signal.

[0255] As an example, see Fig. 7AIf the encoding end only performs feature extraction processing on the first level, the first bitstream corresponding to the first level is finally obtained. The decoding end processes based on the first bitstream to obtain the first low-frequency subband signal feature corresponding to the first level. Afterwards, feature reconstruction processing is performed based on the first low-frequency subband signal feature to obtain the first low-frequency subband signal.

[0256] Correspondingly, if the encoding end only performs the second-level feature extraction processing, the corresponding processing flow is similar to the above method and will not be repeated here.

[0257] See also Figure 6B , Figure 6B This is a flowchart of the audio decoding method provided by the embodiment of the present application. Fig. 6A , Figure 6B Step 203 in the above example can be implemented by steps 2031 and 2032. Figure 6B Steps 2031-2032 are shown for explanation.

[0258] In step 2031, the first level feature reconstruction processing is performed in the following manner: a first feature reconstruction processing is performed based on the first low frequency sub-band signal feature to obtain a first low frequency sub-band signal.

[0259] As an example, if the subband signal features corresponding to the first level and the second level are obtained, and the first level corresponds to the first low-frequency subband signal feature, then the first feature reconstruction processing is performed based on the first low-frequency subband signal feature to obtain the first low-frequency subband signal x′ LB (n). The dimension of the first low-frequency sub-band signal is greater than the dimension of the first low-frequency sub-band signal feature.

[0260] See also Figure 6B , step 2031 can be implemented by steps 20311 to 20314. Figure 6B Explain steps 20311 to 20314.

[0261] In step 20311, a first convolution process is performed on the first low-frequency sub-band signal feature to obtain a first convolution feature.

[0262] See also Fig.8D , Fig.8D It is a schematic diagram of the structure of a neural network for performing first feature reconstruction processing provided in an embodiment of the present application.

[0263] like Fig.8D As shown, first, based on the first low-frequency sub-band signal feature, a convolution layer with a channel of 192 (eg, a causal convolution layer) is called to perform a first convolution process to obtain a first convolution feature of 192*1.

[0264] In step 20312, a first upsampling process is performed on the first convolution feature to obtain a first upsampling feature.

[0265] As an example, see Fig.8D After obtaining the first convolution feature, the three cascaded upsampling layers are called to perform the first upsampling process based on the first convolution feature.

[0266] Among them, these three upsampling layers correspond to three different upsampling factors. As an example, the upsampling factor of the first upsampling layer is 8 and the channels are 96; the upsampling factor of the second upsampling layer is 5 and the channels are 48; the upsampling factor of the third upsampling layer is 4 and the channels are 24. Therefore, after the first upsampling process of these three upsampling layers, the first convolution feature of 192*1 is converted into the first upsampling features of 96*8, 48*40, and 24*160 respectively.

[0267] In step 20313, a first pooling process is performed on the first up-sampled feature to obtain a first pooled feature.

[0268] As an example, after obtaining the first up-sampling feature, the pooling layer is called to perform the first pooling process based on the first up-sampling feature, and when the pooling factor is 2, a first pooling feature of 24*320 is obtained.

[0269] In step 20314, a second convolution process is performed on the first pooled feature to obtain a first low-frequency sub-band signal.

[0270] As an example, after obtaining the first pooling feature, a convolution layer with a channel of 1 (e.g., a causal convolution layer) is called based on the first pooling feature to perform a second convolution process to obtain a first low-frequency subband signal of 1*320. It should be noted that the first low-frequency subband signal here is an estimated value of the low-frequency subband signal at the encoding end.

[0271] Through the above method, the first low-frequency sub-band signal can be obtained quickly and efficiently by means of a neural network.

[0272] In step 2032, the second-level feature reconstruction process is performed in the following manner: a second feature reconstruction process is performed based on the first high-frequency sub-band signal feature to obtain a first high-frequency sub-band signal.

[0273] As an example, if the subband signal features corresponding to the first level and the second level are obtained, and the second level corresponds to the first high-frequency subband signal feature, then the second feature reconstruction processing is performed based on the first high-frequency subband signal feature to obtain the first high-frequency subband signal x′ HB (n). The dimension of the first high frequency sub-band signal is greater than the dimension of the first high frequency sub-band signal feature.

[0274] See also Figure 6C , Figure 6C This is a flowchart of the audio decoding method provided by the embodiment of the present application. Figure 6B , Figure 6C Step 2032 in the above example can be implemented by steps 20321 to 20324. Figure 6C Steps 20321-20324 are shown for illustration.

[0275] In step 20321, frequency domain transform processing is performed on each of the first number of sample points included in the first low-frequency sub-band signal to obtain a first number of low-frequency transform coefficients.

[0276] As an example, when performing the second feature reconstruction process based on the first high-frequency sub-band signal feature, firstly obtain the first low-frequency sub-band signal based on the first feature reconstruction process, and then perform frequency domain transformation process on the first number (e.g., 320) of sample points included in the first low-frequency sub-band signal to obtain 320 low-frequency transformation coefficients. Here, the frequency domain transformation process can be FFT, DCT, MDCT, etc.

[0277] In step 20322, two spectrum replication processes are performed based on the low-frequency transform coefficients in the second half of the first number of low-frequency transform coefficients to obtain a first number of reference transform coefficients included in the first high-frequency reference subband signal.

[0278] As an example, after obtaining 320 low-frequency transform coefficients, the low-frequency transform coefficients of the second half, i.e., the last 160 low-frequency transform coefficients, are selected from the 320 low-frequency transform coefficients, and the selected last 160 low-frequency transform coefficients are spectrally replicated twice to obtain 320 new low-frequency transform coefficients, and the new 320 low-frequency transform coefficients are used as the 320 reference transform coefficients included in the first high-frequency reference subband signal.

[0279] Among them, the frequency of the sample points corresponding to the low-frequency transform coefficients in the second half is higher than the frequency of the sample points corresponding to the low-frequency transform coefficients in the first half.

[0280] It should be noted that the audio signal has the following characteristics: the harmonics of the spectrum in the low-frequency part are relatively more, and the harmonics of the spectrum in the high-frequency part are relatively less. Therefore, in order to avoid the generated spectrum containing too many harmonics due to simple copying, the spectrum of the low-frequency transform coefficients corresponding to the sample points with higher frequencies is copied here, that is, the spectrum of the low-frequency transform coefficients corresponding to the last 160 sample points with higher frequencies.

[0281] In step 20323, based on multiple first subband spectral envelopes corresponding to the first high frequency subband signal characteristics, a first gain control is performed on a first number of reference transform coefficients included in the first high frequency reference subband signal to obtain a first number of first gain reference transform coefficients.

[0282] As an example, after obtaining 320 reference transformation coefficients corresponding to the first high-frequency reference subband signal, a first gain control is performed on the 320 reference transformation coefficients included in the first high-frequency reference subband signal based on 8 first subband spectral envelopes corresponding to the first high-frequency subband signal characteristics, thereby obtaining 320 first gain reference transformation coefficients.

[0283] See also Fig.6D , Fig.6D This is a flowchart of the audio decoding method provided by the embodiment of the present application. Figure 6C , Fig.6D Step 20323 in can be implemented by steps 203231 to 203236. Fig.6D Steps 203231-203236 are shown for illustration.

[0284] In step 203231, a first number of reference transform coefficients are divided into a plurality of first reference sub-bands.

[0285] As an example, when performing the first gain control, 320 reference transform coefficients are first divided into a plurality of first reference sub-bands which is the same as the number of first sub-band spectral envelopes. That is, when the number of first sub-band spectral envelopes is 8, 8 first reference sub-bands are obtained here, and each first reference sub-band includes 40 reference transform coefficients.

[0286] In step 203232, a plurality of different first combinations are generated, each first combination comprising a first reference subband and a first subband spectral envelope, and for each first combination, the following processing is performed: the first subband spectral envelope is determined as a first average energy.

[0287] As an example, after obtaining 8 first reference subbands, 8 first combinations are generated for the 8 first reference subbands and 8 first subband spectral envelopes, each first combination includes a first reference subband and a first subband spectral envelope, and different first combinations include different first reference subbands and first subband spectral envelopes.

[0288] For each of the eight first combinations, a first sub-band spectral envelope is first determined as a first average energy.

[0289] In step 203233, averaging processing is performed based on the second number of reference transform coefficients in the first reference sub-band to obtain a first reference average energy corresponding to the first reference sub-band.

[0290] As an example, the second number here is a ratio of the first number to the number of the plurality of first reference sub-bands. When the first number is 320 and the number of the first reference sub-bands is 8, the second number is 40.

[0291] The 40 reference transform coefficients included in the first reference subband are averaged to obtain a first reference average energy. The calculation method of the first reference average energy is similar to the calculation method of the first average energy described above, and will not be repeated here.

[0292] In step 203234, the ratio of the first average energy to the first reference average energy is determined, and the ratio is squared to obtain a first proportional factor.

[0293] As an example, after obtaining the first average energy and the first reference average energy corresponding to the first combination, the ratio of the first average energy to the first reference average energy is calculated, and the ratio is squared to obtain the first proportional factor.

[0294] In step 203235, the second number of reference transform coefficients in the first reference sub-band are multiplied by the first scale factor respectively to obtain the second number of first gain reference transform coefficients in the first reference sub-band.

[0295] As an example, after obtaining the first scale factor, 40 reference transform coefficients in the first reference subband are all multiplied by the first scale factor, thereby obtaining 40 first gain reference transform coefficients.

[0296] In step 203236, the second number of first gain reference transform coefficients in each first reference sub-band are merged to obtain a first number of first gain reference transform coefficients.

[0297] As an example, since 8 first combinations are obtained in step 203232, and each first combination calculates 40 first gain reference transformation coefficients of a first reference subband, the 40 first gain reference transformation coefficients of the first reference subband corresponding to these 8 first combinations are merged to obtain 320 first gain reference transformation coefficients.

[0298] By performing gain control in the above manner, the spectrum energy of the reference transform coefficients virtually generated by the decoding end can be made closer to the spectrum energy of the original transform coefficients at the encoding end, thereby facilitating accurate generation of an estimated value of the high frequency subband signal.

[0299] In step 20324, an inverse frequency domain transform process is performed on the first number of first gain reference transform coefficients to obtain a first high frequency sub-band signal.

[0300] As an example, after obtaining 320 first gain reference transform coefficients, the 320 first gain reference transform coefficients are subjected to inverse frequency domain transform processing to obtain a first high frequency sub-band signal. It should be noted that the first high frequency sub-band signal here is an estimated value of the high frequency sub-band signal at the encoding end.

[0301] Since the inverse frequency domain transform process is performed based on the transform coefficient after gain control, the obtained first high frequency subband signal can be closer to the high frequency subband signal at the encoding end, thereby improving the decoding quality of the high frequency subband signal.

[0302] See also Fig. 6E , Fig. 6E This is a flowchart of the audio decoding method provided by the embodiment of the present application. Fig. 6A , Fig. 6E Step 203 in the above example can be implemented by steps 2033-2034. Fig. 6E Steps 2033-2034 are shown for explanation.

[0303] In step 2033, the third level feature reconstruction processing is performed in the following manner: the third feature reconstruction processing is performed on the second low frequency sub-band signal feature to obtain the second low frequency sub-band signal.

[0304] As an example, if the subband signal feature corresponding to the third level is also obtained, and the third level corresponds to the second low-frequency subband signal feature, then the third feature reconstruction processing is performed based on the second low-frequency subband signal feature to obtain the second low-frequency subband signal x′ LB，e (n).

[0305] See also Fig. 6E , step 2033 can be implemented by steps 20331 to 20334. Fig. 6E Steps 20331 to 20334 are explained.

[0306] In step 20331, a third convolution process is performed on the second low-frequency sub-band signal feature to obtain a third convolution feature.

[0307] See also Fig. 8E , Fig. 8E It is a schematic diagram of the structure of a neural network for performing third feature reconstruction processing provided in an embodiment of the present application.

[0308] like Fig. 8E As shown, based on the second low-frequency sub-band signal feature, a convolution layer with a channel of 192 (eg, a causal convolution layer) is called to perform a third convolution process to obtain a 192*1 third convolution feature.

[0309] In step 20332, a second upsampling process is performed on the third convolution feature to obtain a second upsampling feature.

[0310] As an example, see Fig. 8E After obtaining the third convolution feature, the three cascaded upsampling layers are called to perform the second upsampling process based on the third convolution feature.

[0311] Among them, these three upsampling layers correspond to three different upsampling factors. As an example, the upsampling factor of the first upsampling layer is 8 and the channels are 96; the upsampling factor of the second upsampling layer is 5 and the channels are 48; the upsampling factor of the third upsampling layer is 4 and the channels are 24. Therefore, after the second upsampling process of these three upsampling layers, the first convolution feature of 192*1 is converted into the second upsampling features of 96*8, 48*40, and 24*160 respectively.

[0312] In step 20333, a second pooling process is performed on the second up-sampled feature to obtain a second pooled feature.

[0313] As an example, after obtaining the second up-sampling feature, the pooling layer is called to perform a second pooling process based on the second up-sampling feature, and when the pooling factor is 2, a second pooling feature of 24*320 is obtained.

[0314] In step 20334, a fourth convolution process is performed on the second pooled feature to obtain a second low-frequency sub-band signal.

[0315] As an example, see Fig. 8E After obtaining the second pooling feature, a convolution layer with a channel of 1 (e.g., a causal convolution layer) is called based on the second pooling feature to perform a fourth convolution process to obtain a second low-frequency subband signal of 1*320. It should be noted that the second low-frequency subband signal here is a residual signal of the first low-frequency subband signal.

[0316] Through the above method, the second low-frequency sub-band signal can be obtained quickly and efficiently by means of a neural network.

[0317] In step 2034, a fourth feature reconstruction process is performed based on the first high frequency sub-band signal feature and the second high frequency sub-band signal feature to obtain a second high frequency sub-band signal.

[0318] As an example, if the subband signal characteristics corresponding to the third level are also obtained, and the third level corresponds to the second high-frequency subband signal characteristics, then the fourth characteristic reconstruction processing is performed based on the first high-frequency subband signal characteristics and the second high-frequency subband signal characteristics to obtain the second high-frequency subband signal.

[0319] See also Fig. 6F , Fig. 6FThis is a flowchart of the audio decoding method provided by the embodiment of the present application. Fig. 6E , Fig. 6F Step 2034 in the above example can be implemented by steps 20341 to 20344. Fig. 6F Steps 20341-20344 are shown for illustration.

[0320] In step 20341, a plurality of first sub-bands corresponding to the first high frequency sub-band signal characteristic are determined, and a plurality of second sub-bands corresponding to the second high frequency sub-band signal characteristic are determined.

[0321] As an example, since during the encoding process, the first high-frequency subband signal feature is determined based on multiple first subbands, and the second high-frequency subband signal feature is determined based on multiple second subbands, when performing the fourth feature reconstruction process, first determine the multiple first subbands corresponding to the first high-frequency subband signal feature, and determine the multiple second subbands corresponding to the second high-frequency subband signal feature. The number of the multiple first subbands is the same as the number of the multiple second subbands. For example, the number of the first subband and the number of the second subband are both 8.

[0322] In step 20342, third subband spectral envelopes corresponding to the plurality of third subbands are determined based on the first subband spectral envelopes corresponding to the plurality of first subbands and the second subband spectral envelopes corresponding to the plurality of second subbands.

[0323] As an example, after obtaining 8 first subbands and 8 second subbands, based on the first subband spectral envelope corresponding to the first subband and the second subband spectral envelope corresponding to the second subband, the third subband spectral envelopes corresponding to the 8 third subbands are determined. The number of the third subbands is the same as the number of the second subbands and also the same as the number of the first subbands, for example, both are 8.

[0324] In some embodiments, third subband spectral envelopes corresponding to multiple third subbands are determined in the following manner: multiple different second combinations are generated, each second combination includes a first subband and a second subband corresponding to the first subband, and for each second combination, the following processing is performed: the first subband spectral envelope corresponding to the first subband is determined as the first average energy, and the second subband spectral envelope corresponding to the second subband is determined as the second average energy; the first average energy is multiplied by the second quantity to obtain a first multiplication result; the second average energy is multiplied by the third quantity to obtain a second multiplication result; the first multiplication result is subtracted from the second multiplication result, the ratio of the subtraction result to the third quantity is determined as the third average energy corresponding to the third subband, and the third average energy is determined as the third subband spectral envelope corresponding to the third subband.

[0325] As an example, first, eight second combinations are generated, each of which includes a first subband and a second subband corresponding to the first subband, and different second combinations include different first subbands and different second subbands.

[0326] Secondly, for each of the 8 second combinations, the first subband spectral envelope corresponding to the first subband is taken as the first average energy, and the first average energy is multiplied by the second number of high-frequency transformation coefficients included in the first subband (e.g., 40) to obtain a first multiplication result.

[0327] Correspondingly, the second subband spectrum envelope corresponding to the second subband is used as the second average energy, and the second average energy is multiplied by the third number of high-frequency transform coefficients included in the second subband (e.g., 20) to obtain a second multiplication result. The third number is the number of high-frequency transform coefficients included in the second subband, and the third number is half of the second number.

[0328] Finally, the first multiplication result and the second multiplication result are subtracted, and the ratio of the obtained subtraction result to the third number is determined as the third average energy corresponding to the third subband, and the third average energy is determined as the third subband spectral envelope corresponding to the third subband. The second number of high-frequency transform coefficients included in the first subband is the union of the high-frequency transform coefficients included in the second subband and the high-frequency transform coefficients included in the third subband. That is, the 40 high-frequency transform coefficients included in the first subband are the union of the 20 high-frequency transform coefficients included in the second subband and the 20 high-frequency transform coefficients included in the third subband.

[0329] As an example, the third average energy is calculated as follows:

[0330]

[0331] Among them, F′ HB (n) represents the first average energy corresponding to the first subband in each second combination, F′ HB,e (n) represents the second average energy corresponding to the second subband in each second combination.

[0332] As an example, since 8 second combinations are obtained, and each second combination calculates a third sub-band spectrum envelope, 8 third sub-band spectrum envelopes can be obtained based on the 8 second combinations.

[0333] In the above manner, multiple third sub-band spectral envelopes can be accurately obtained.

[0334] In step 20343, the plurality of second sub-band spectral envelopes and the plurality of third sub-band spectral envelopes are determined as a fourth sub-band spectral envelope.

[0335] As an example, after obtaining 8 third sub-band spectral envelopes, 8 second sub-band spectral envelopes and 8 third sub-band spectral envelopes are both determined as fourth sub-band spectral envelopes, that is, 16 fourth sub-band spectral envelopes are obtained.

[0336] In step 20344, based on a plurality of fourth sub-band spectral envelopes, a second gain control is performed on a first number of reference transform coefficients included in the first high-frequency reference sub-band signal to obtain a first number of second gain reference transform coefficients.

[0337] As an example, after obtaining 16 fourth subband spectral envelopes, second gain control is performed on 320 reference transform coefficients included in the first high-frequency reference subband signal based on the 16 fourth subband spectral envelopes, thereby obtaining 320 second gain reference transform coefficients.

[0338] See also Figure 6G , Figure 6G This is a flowchart of the audio decoding method provided by the embodiment of the present application. Fig. 6F , Figure 6G Step 20344 in can be implemented by steps 203441 to 203446. Figure 6G Steps 203441-203446 are shown for illustration.

[0339] In step 203441, the first number of reference transform coefficients is divided into a plurality of second reference subbands.

[0340] As an example, when performing the second gain control, 320 reference transform coefficients are first divided into the same number of second reference subbands as the fourth subband spectral envelope, that is, divided into 16 second reference subbands, each second reference subband including 20 reference transform coefficients.

[0341] In step 203442, a plurality of different third combinations are generated, each third combination comprising a second reference subband and a fourth subband spectral envelope, and for each third combination, the following processing is performed: the fourth subband spectral envelope is determined as a fourth average energy.

[0342] As an example, after obtaining 16 second reference subbands, 16 third combinations are generated for the 16 second reference subbands and 16 fourth subband spectral envelopes, each third combination includes a second reference subband and a fourth subband spectral envelope, and different third combinations include different second reference subbands and fourth subband spectral envelopes.

[0343] For each of the 16 third combinations, the fourth sub-band spectral envelope is first determined as the fourth average energy.

[0344] In step 203443, an averaging process is performed based on the third number of reference transform coefficients in the second reference sub-band to obtain a second reference average energy corresponding to the second reference sub-band.

[0345] As an example, the third number here is a ratio of the first number to the number of the plurality of second reference sub-bands. When the first number is 320 and the number of the second reference sub-bands is 16, the third number is 20.

[0346] The second reference average energy is obtained by performing an average processing based on the 20 reference transform coefficients included in the second reference subband. The calculation method of the second reference average energy is similar to the calculation method of the second average energy described above, and will not be repeated here.

[0347] In step 203444, the ratio of the fourth average energy to the second reference average energy is determined, and the ratio is squared to obtain a second proportional factor.

[0348] As an example, after obtaining the fourth average energy and the second reference average energy corresponding to the third combination, the ratio of the fourth average energy to the second reference average energy is calculated, and the ratio is squared to obtain the second proportional factor.

[0349] In step 203445, the third number of reference transform coefficients in the second reference sub-band are multiplied by the second scale factor respectively to obtain the third number of second gain reference transform coefficients in the second reference sub-band.

[0350] As an example, after obtaining the second scale factor, all 20 reference transform coefficients in the second reference subband are multiplied by the second scale factor, thereby obtaining 20 second gain reference transform coefficients.

[0351] In step 203446, the third number of second gain reference transform coefficients in each second reference sub-band are merged to obtain the first number of second gain reference transform coefficients.

[0352] As an example, since 16 third combinations are obtained in step 203442, and each third combination is calculated to obtain 20 second gain reference transformation coefficients of a second reference subband, the 20 second gain reference transformation coefficients of the second reference subband corresponding to these 16 third combinations are merged to obtain 320 second gain reference transformation coefficients.

[0353] By performing gain control in the above manner, the spectrum energy of the reference transform coefficients virtually generated by the decoding end can be made closer to the spectrum energy of the original transform coefficients at the encoding end, thereby facilitating accurate generation of an estimated value of the high frequency subband signal.

[0354] In step 20345, an inverse frequency domain transform process is performed on the first number of second gain reference transform coefficients to obtain a second high frequency sub-band signal.

[0355] As an example, after obtaining 320 second gain reference transform coefficients, the 320 second gain reference transform coefficients are subjected to inverse frequency domain transform processing to obtain a second high frequency subband signal. It should be noted that the second high frequency subband signal here is an estimated value of the high frequency subband signal at the encoding end.

[0356] Since the inverse frequency domain transform processing is performed based on the transform coefficient after gain control, the obtained second high frequency subband signal can be closer to the high frequency subband signal at the encoding end, thereby improving the decoding quality of the high frequency subband signal.

[0357] In step 204, the sub-band signals corresponding to the multiple layers are synthesized into an audio signal.

[0358] As an example, after obtaining the sub-band signals corresponding to the multiple levels, the sub-band signals corresponding to the multiple levels are synthesized into an audio signal. As an example, the synthesis process can be implemented by calling a QMF synthesis filter.

[0359] As an example, the H corresponding to the QMF analysis filter can be Low (z) and H High (z), describes the QMF synthesis filter bank, and is calculated as follows:

[0360] G Low (z) = H Low (z) Formula 3

[0361] G High (z)=(-1)*H High (z) Formula 4

[0362] Among them, G Low (z) represents the spectral response of the low-pass signal of the QMF synthesis filter, H Low (z) represents the spectral response of the low-pass signal of the QMF analysis filter, G High (z) represents the spectral response of the high-pass signal of the QMF synthesis filter, H High (z) represents the spectral response of the high-pass signal of the QMF analysis filter.

[0363] The low-pass signal (i.e., low-frequency subband signal) and high-pass signal (i.e., high-frequency subband signal) with a sampling rate of Fs / 2 recovered at the decoding end can be processed by the QMF synthesis filter group to restore a reconstructed signal with the same sampling rate Fs as the original audio signal.

[0364] In some embodiments, after obtaining the sub-band signals corresponding to the first level and the second level respectively, the sub-band signals corresponding to the two levels respectively are synthesized into an audio signal in the following manner: performing a third up-sampling process on the first low-frequency sub-band signal of the second sampling frequency to obtain a third up-sampling result of the first sampling frequency; performing a fourth up-sampling process on the first high-frequency sub-band signal of the second sampling frequency to obtain a fourth up-sampling result of the first sampling frequency; and synthesizing the third up-sampling result and the fourth up-sampling result to obtain an audio signal.

[0365] As an example, the first level corresponds to the first low-frequency subband signal, and the first low-frequency subband signal of the second sampling rate (e.g., 16000 Hz) is subjected to a third upsampling process by a QMF synthesis filter to obtain a third upsampling result of the first sampling frequency (e.g., 32000 Hz), wherein the second sampling frequency is half of the first sampling frequency.

[0366] Correspondingly, the second level corresponds to the first high frequency subband signal, and the first high frequency subband signal of the second sampling rate (eg, 16000 Hz) is subjected to a fourth upsampling process through a QMF synthesis filter to obtain a fourth upsampling result of the first sampling frequency (eg, 32000 Hz).

[0367] Afterwards, the third up-sampling result and the fourth up-sampling result are synthesized to obtain an audio signal. It should be noted that the audio signal here is an estimated value of the original audio signal at the encoding end.

[0368] Through the above method, an accurate audio signal can be obtained.

[0369] In some embodiments, after obtaining the sub-band signals corresponding to the first level, the second level and the third level respectively, the sub-band signals corresponding to the three levels are synthesized into an audio signal in the following manner: a first low-frequency sub-band signal and a second low-frequency sub-band signal are summed to obtain a first summed result; a fifth up-sampling process is performed on the first summed result of the second sampling frequency to obtain a fifth up-sampling result of the first sampling frequency; a sixth up-sampling process is performed on the second high-frequency sub-band signal of the second sampling frequency to obtain a sixth up-sampling result of the first sampling frequency; and the fifth up-sampling result and the sixth up-sampling result are synthesized to obtain an audio signal.

[0370] As an example, the third level corresponds to the second low-frequency subband signal. First, the first low-frequency subband signal corresponding to the first level and the second low-frequency subband signal corresponding to the third level are summed to obtain a first summation result, that is, a high-precision estimated value of the low-frequency subband signal. Secondly, the first summation result of the second sampling rate (e.g., 16000 Hz) is subjected to a fifth upsampling process through a QMF synthesis filter to obtain a fifth upsampling result of the first sampling frequency (e.g., 32000 Hz).

[0371] Since the third level also corresponds to the second high frequency sub-band signal, and the second high frequency sub-band signal has higher resolution and accuracy than the first high frequency sub-band signal, synthesis processing is performed based on the second high frequency sub-band signal. For example, the second high frequency sub-band signal of the second sampling rate (e.g., 16000 Hz) is subjected to the sixth up-sampling processing through the QMF synthesis filter to obtain the sixth up-sampling result of the first sampling frequency (e.g., 32000 Hz).

[0372] Afterwards, the fifth up-sampling result and the sixth up-sampling result are synthesized to obtain an audio signal. It should be noted that the audio signal here is an estimated value of the original audio signal at the encoding end.

[0373] Through the above method, an accurate audio signal can be obtained.

[0374] In the embodiment of the present application, decoding and feature reconstruction are performed in layers, and synthesis processing is performed based on the subband signals corresponding to each layer. Since each layer only needs to process part of the subband signals, the decoding efficiency can be improved and the decoding complexity can be reduced.

[0375] The following further describes an exemplary structure of the audio encoding device 433 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 4A As shown, the software modules stored in the audio encoding device 433 of the memory 430 may include: a decomposition module 4331, which is used to decompose the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal; a feature extraction module 4332, which is used to perform feature extraction processing at multiple levels based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to the multiple levels; a quantization module 4333, which is used to quantize the sub-band signal features corresponding to each level to obtain the index value of the sub-band signal features; an encoding module 4334, which is used to encode the index value of the sub-band signal features to obtain the code stream corresponding to the level; a configuration module 4335, which is used to configure the corresponding transmission priority for the code streams corresponding to the multiple levels; wherein the transmission priority is positively correlated with the decoding quality index of the code stream corresponding to the level.

[0376] In the above scheme, the decomposition module 4331 is used to sample the audio signal at a first sampling frequency to obtain a sampled signal; wherein the sampled signal includes multiple sample points sampled from the audio signal; the sampled signal is low-pass filtered, and the obtained low-pass filtered result is subjected to a first down-sampling process to obtain a low-frequency sub-band signal of a second sampling frequency; the sampled signal is high-pass filtered, and the obtained high-pass filtered result is subjected to a second down-sampling process to obtain a high-frequency sub-band signal of the second sampling frequency; wherein the second sampling frequency is half of the first sampling frequency, and the bandwidth of the low-frequency sub-band signal and the high-frequency sub-band signal are the same.

[0377] In the above scheme, the multiple levels include a first level and a second level; the feature extraction module 4332 is used to perform feature extraction processing of the first level in the following manner: performing a first feature extraction processing based on the low-frequency sub-band signal to obtain a first low-frequency sub-band signal feature; wherein the dimension of the first low-frequency sub-band signal feature is smaller than the dimension of the low-frequency sub-band signal; performing feature extraction processing of the second level in the following manner: performing a second feature extraction processing based on the high-frequency sub-band signal to obtain a first high-frequency sub-band signal feature; wherein the dimension of the first high-frequency sub-band signal feature is smaller than the dimension of the high-frequency sub-band signal.

[0378] In the above scheme, the feature extraction module 4332 is used to perform a first convolution process on the low-frequency sub-band signal to obtain a first convolution feature; perform a first pooling process on the first convolution feature to obtain a first pooling feature; perform a third downsampling process on the first pooling feature to obtain a third downsampling feature; wherein the third downsampling process includes multiple cascaded downsampling; perform a second convolution process on the third downsampling feature to obtain a first low-frequency sub-band signal feature.

[0379] In the above scheme, the high-frequency sub-band signal includes a first number of sample points, and the first number is an integer greater than 2; the feature extraction module 4332 is used to perform frequency domain transform processing based on the first number of sample points respectively to obtain a first number of transform coefficients; divide the first number of transform coefficients into multiple first sub-bands; perform the following processing for each first sub-band: perform averaging processing based on the second number of transform coefficients in the first sub-band to obtain the first average energy corresponding to the first sub-band, and determine the first average energy as the first sub-band spectral envelope corresponding to the first sub-band; wherein the second number is the ratio of the first number to the number of multiple first sub-bands; and determine the first sub-band spectral envelopes corresponding to the multiple first sub-bands as the first high-frequency sub-band signal features.

[0380] In the above scheme, the feature extraction module 4332 is used to obtain a reference frame high-frequency subband signal corresponding to a reference frame audio signal; wherein the reference frame audio signal is the previous frame or the next frame of the audio signal, and the reference frame high-frequency subband signal includes a first number of sample points; based on the first number of sample points in the reference frame high-frequency subband signal and the first number of sample points in the high-frequency subband signal, discrete cosine transform processing is performed on the first number of sample points included in the high-frequency subband signal to obtain transform coefficients corresponding to the first number of sample points included in the high-frequency subband signal.

[0381] In the above scheme, the multiple levels also include a third level; the feature extraction module 4332 is used to perform feature extraction processing of the third level in the following manner: performing a third feature extraction processing based on the low-frequency subband signal and the first low-frequency subband signal feature to obtain the second low-frequency subband signal feature; performing a fourth feature extraction processing based on the first high-frequency subband signal feature to obtain the second high-frequency subband signal feature.

[0382] In the above scheme, the feature extraction module 4332 is used to perform splicing processing on the low-frequency sub-band signal and the first low-frequency sub-band signal feature to obtain a splicing feature; perform a third convolution processing on the splicing feature to obtain a third convolution feature; perform a second pooling processing on the third convolution feature to obtain a second pooling feature; perform a fourth downsampling processing on the second pooling feature to obtain a fourth downsampling feature; wherein the fourth downsampling processing includes multiple cascaded downsampling; perform a fourth convolution processing on the fourth downsampling feature to obtain a second low-frequency sub-band signal feature; wherein the dimension of the second low-frequency sub-band signal feature is smaller than the dimension of the first low-frequency sub-band signal feature.

[0383] In the above scheme, the feature extraction module 4332 is used to perform the following processing for each first sub-band corresponding to the first high-frequency sub-band signal feature: select a third number of transformation parameters from the second number of transformation parameters in the first sub-band, and determine the third number of transformation coefficients as the second sub-band; wherein the third number is half of the second number; perform averaging processing based on the third number of transformation coefficients in the second sub-band to obtain the second average energy corresponding to the second sub-band, and determine the second average energy as the second sub-band spectral envelope corresponding to the second sub-band; determine the second sub-band spectral envelope corresponding to each second sub-band as the second high-frequency sub-band signal feature.

[0384] In the above scheme, the multiple levels include a first level and a second level, and the subband signal characteristics corresponding to the first level are first low-frequency subband signal characteristics, and the subband signal characteristics corresponding to the second level are first high-frequency subband signal characteristics; the encoding module 4334 is used to encode the index value of the first low-frequency signal characteristic to obtain a first code stream corresponding to the first level; encode the index value of the first high-frequency signal characteristic to obtain a second code stream corresponding to the second level; wherein the code rate of the first code stream is greater than the code rate corresponding to the second code stream, and the code rate is positively correlated with the decoding quality index of the corresponding code stream.

[0385] In the above scheme, the multiple levels also include the third level, and the subband signal characteristics corresponding to the third level are the second low-frequency subband signal characteristics and the second high-frequency subband signal characteristics; the encoding module 4334 is used to encode the index value of the second low-frequency signal characteristics to obtain a third code stream corresponding to the third level; encode the index value of the second high-frequency signal characteristics to obtain a fourth code stream corresponding to the third level; wherein the bit rate of the first code stream is greater than the bit rate of the third code stream, the bit rate of the third code stream is greater than the bit rate of the second code stream, the bit rate of the second code stream is equal to the bit rate of the fourth code stream, and the bit rate is positively correlated with the decoding quality index of the corresponding code stream.

[0386] In the above scheme, the multiple levels include a first level and a second level, and the first code stream corresponding to the first level is obtained by encoding the index value of the first low-frequency signal feature, and the second code stream corresponding to the second level is obtained by encoding the index value of the first high-frequency signal feature; the configuration module 4335 is used to configure the first transmission priority for the first code stream corresponding to the first level; and configure the second transmission priority for the second code stream corresponding to the second level; wherein the first transmission priority is higher than the second transmission priority.

[0387] In the above scheme, the multiple levels also include the third level, and the third level corresponds to the third code stream and the fourth code stream, the third code stream is obtained by encoding the index value of the second low-frequency signal feature, and the fourth code stream is obtained by encoding the index value of the second high-frequency signal feature; the configuration module 4335 is used to configure the third transmission priority for the third code stream corresponding to the third level; and configure the fourth transmission priority for the fourth code stream corresponding to the third level; wherein the first transmission priority is higher than the third transmission priority, the third transmission priority is higher than the second transmission priority, and the second transmission priority is the same as the fourth transmission priority.

[0388] The following further describes an exemplary structure of the audio decoding device 473 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 4BAs shown, the software modules stored in the audio encoding device 473 of the memory 470 may include: a decoding module 4731, which is used to decode the code streams corresponding to multiple levels to obtain the index value of the code stream corresponding to each level; wherein different levels correspond to different transmission priorities, and the transmission priority is positively correlated with the decoding quality index of the code stream corresponding to the level; an inverse quantization module 4732, which is used to inverse quantize the index value of the code stream corresponding to each level to obtain the sub-band signal characteristics corresponding to each level; a feature reconstruction module 4733, which is used to reconstruct the sub-band signal characteristics corresponding to each level to obtain the sub-band signal corresponding to each level; and a synthesis module 4734, which is used to synthesize the sub-band signals corresponding to multiple levels into an audio signal.

[0389] In the above scheme, the multiple levels include a first level and a second level, the first level corresponds to the first low-frequency sub-band signal characteristics, and the second level corresponds to the first high-frequency sub-band signal characteristics; the feature reconstruction module 4733 is used to perform the feature reconstruction processing of the first level in the following manner: performing a first feature reconstruction processing based on the first low-frequency sub-band signal characteristics to obtain a first low-frequency sub-band signal; wherein the dimension of the first low-frequency sub-band signal is greater than the dimension of the first low-frequency sub-band signal characteristics; performing the feature reconstruction processing of the second level in the following manner: performing a second feature reconstruction processing based on the first high-frequency sub-band signal characteristics to obtain a first high-frequency sub-band signal; wherein the dimension of the first high-frequency sub-band signal is greater than the dimension of the first high-frequency sub-band signal characteristics.

[0390] In the above scheme, the feature reconstruction module 4733 is used to perform a first convolution process on the first low-frequency sub-band signal feature to obtain a first convolution feature; perform a first upsampling process on the first convolution feature to obtain a first upsampling feature; wherein the first upsampling process includes multiple cascaded upsamplings; perform a first pooling process on the first upsampling feature to obtain a first pooling feature; perform a second convolution process on the first pooling feature to obtain a first low-frequency sub-band signal.

[0391] In the above scheme, the feature reconstruction module 4733 is used to perform frequency domain transformation processing based on the first number of sample points included in the first low-frequency sub-band signal to obtain a first number of low-frequency transformation coefficients; based on the low-frequency transformation coefficients of the second half of the first number of low-frequency transformation coefficients, perform two spectrum replication processes to obtain the first number of reference transformation coefficients included in the first high-frequency reference sub-band signal; wherein the frequency of the sample points corresponding to the low-frequency transformation coefficients in the second half is higher than the frequency of the sample points corresponding to the low-frequency transformation coefficients in the first half; based on multiple first sub-band spectral envelopes corresponding to the characteristics of the first high-frequency sub-band signal, perform first gain control on the first number of reference transformation coefficients included in the first high-frequency reference sub-band signal to obtain a first number of first gain reference transformation coefficients; perform inverse frequency domain transformation processing on the first number of first gain reference transformation coefficients to obtain a first high-frequency sub-band signal.

[0392] In the above scheme, the feature reconstruction module 4733 is used to divide a first number of reference transform coefficients into multiple first reference sub-bands; wherein the number of first reference sub-bands is the same as the number of first sub-band spectral envelopes; generate multiple different first combinations, each first combination includes a first reference sub-band and a first sub-band spectral envelope, and for each first combination, perform the following processing: determine the first sub-band spectral envelope as a first average energy; perform averaging processing based on the second number of reference transform coefficients in the first reference sub-band to obtain a first reference average energy corresponding to the first reference sub-band; wherein the second number is the ratio of the first number to the number of multiple first reference sub-bands; determine the ratio of the first average energy to the first reference average energy, and perform square root processing on the ratio to obtain a first proportional factor; multiply the second number of reference transform coefficients in the first reference sub-band by the first proportional factor respectively to obtain a second number of first gain reference transform coefficients in the first reference sub-band; merge the second number of first gain reference transform coefficients in each first reference sub-band to obtain a first number of first gain reference transform coefficients.

[0393] In the above scheme, the multiple levels also include a third level, and the third level corresponds to the second low-frequency sub-band signal characteristics and the second high-frequency sub-band signal characteristics; the feature reconstruction module 4733 is used to perform the feature reconstruction processing of the third level in the following manner: performing the third feature reconstruction processing on the second low-frequency sub-band signal characteristics to obtain the second low-frequency sub-band signal; performing the fourth feature reconstruction processing based on the first high-frequency sub-band signal characteristics and the second high-frequency sub-band signal characteristics to obtain the second high-frequency sub-band signal.

[0394] In the above scheme, the feature reconstruction module 4733 is used to perform a third convolution process on the second low-frequency sub-band signal feature to obtain a third convolution feature; perform a second upsampling process on the third convolution feature to obtain a second upsampling feature; wherein the second upsampling process includes multiple cascaded upsampling; perform a second pooling process on the second upsampling feature to obtain a second pooling feature; perform a fourth convolution process on the second pooling feature to obtain a second low-frequency sub-band signal; wherein the second low-frequency sub-band signal is a residual signal of the first low-frequency sub-band signal.

[0395] In the above scheme, the feature reconstruction module 4733 is used to determine multiple first sub-bands corresponding to the first high-frequency sub-band signal feature, and determine multiple second sub-bands corresponding to the second high-frequency sub-band signal feature; wherein the number of the multiple first sub-bands is the same as the number of the multiple second sub-bands; based on the first sub-band spectral envelopes corresponding to the multiple first sub-bands and the second sub-band spectral envelopes corresponding to the multiple second sub-bands, determine the third sub-band spectral envelopes corresponding to the multiple third sub-bands; wherein the number of the multiple third sub-bands is the same as the number of the multiple second sub-bands; determine the multiple second sub-band spectral envelopes and the multiple third sub-band spectral envelopes as the fourth sub-band spectral envelope; based on the multiple fourth sub-band spectral envelopes, perform a second gain control on the first number of reference transformation coefficients included in the first high-frequency reference sub-band signal to obtain a first number of second gain reference transformation coefficients; perform inverse frequency domain transformation processing on the first number of second gain reference transformation coefficients to obtain a second high-frequency sub-band signal.

[0396] In the above scheme, the feature reconstruction module 4733 is used to generate multiple different second combinations, each second combination includes a first subband and a second subband corresponding to the first subband, and for each second combination, the following processing is performed: the first subband spectral envelope corresponding to the first subband is determined as the first average energy, and the second subband spectral envelope corresponding to the second subband is determined as the second average energy; the first average energy is multiplied by the second number to obtain a first multiplication result; wherein the second number is the number of high-frequency transformation coefficients included in the first subband; the second average energy is multiplied by the third number to obtain a second multiplication result; wherein the third number is the number of high-frequency transformation coefficients included in the second subband, and the third number is half of the second number; the first multiplication result is subtracted from the second multiplication result, and the ratio of the obtained subtraction result to the third number is determined as the third average energy corresponding to the third subband, and the third average energy is determined as the third subband spectral envelope corresponding to the third subband; wherein the second number of high-frequency transformation coefficients included in the first subband is the union of the following two: the high-frequency transformation coefficients included in the second subband and the high-frequency transformation coefficients included in the third subband.

[0397] In the above solution, the feature reconstruction module 4733 is used to divide the first number of reference transform coefficients into a plurality of second reference sub-bands; wherein the number of the second reference sub-bands is the same as the number of the fourth sub-band spectral envelopes;

[0398] Generate multiple different third combinations, each third combination includes a second reference subband and a fourth subband spectral envelope, and for each third combination, perform the following processing: determine the fourth subband spectral envelope as a fourth average energy; perform averaging processing based on a third number of reference transformation coefficients in the second reference subband to obtain a second reference average energy corresponding to the second reference subband; wherein the third number is the ratio of the first number to the number of multiple second reference subbands; determine the ratio of the fourth average energy to the second reference average energy, and perform square root processing on the ratio to obtain a second proportional factor; multiply the third number of reference transformation coefficients in the second reference subband by the second proportional factor respectively to obtain a third number of second gain reference transformation coefficients in the second reference subband; merge the third number of second gain reference transformation coefficients in each second reference subband to obtain a first number of second gain reference transformation coefficients.

[0399] In the above scheme, the multiple levels include a first level and a second level, the first level corresponds to the first low-frequency sub-band signal, and the second level corresponds to the first high-frequency sub-band signal; the synthesis module 4734 is used to perform a third up-sampling process on the first low-frequency sub-band signal of the second sampling frequency to obtain a third up-sampling result of the first sampling frequency; wherein the second sampling frequency is half of the first sampling frequency; perform a fourth up-sampling process on the first high-frequency sub-band signal of the second sampling frequency to obtain a fourth up-sampling result of the first sampling frequency; and synthesize the third up-sampling result and the fourth up-sampling result to obtain an audio signal.

[0400] In the above scheme, the multiple levels also include a third level, and the third level corresponds to the second low-frequency subband signal and the second high-frequency subband signal; the synthesis module 4734 is used to sum the first low-frequency subband signal and the second low-frequency subband signal to obtain a first summation result; perform a fifth upsampling process on the first summation result of the second sampling frequency to obtain a fifth upsampling result of the first sampling frequency; perform a sixth upsampling process on the second high-frequency subband signal of the second sampling frequency to obtain a sixth upsampling result of the first sampling frequency; and perform a synthesis process on the fifth upsampling result and the sixth upsampling result to obtain an audio signal.

[0401] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the audio encoding method and the audio decoding method described in the embodiment of the present application.

[0402] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the audio encoding method and the audio decoding method provided in the embodiment of the present application.

[0403] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0404] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0405] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0406] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0407] To summarize, in the embodiments of the present application, the subband signal characteristics of the audio subband signal at each layer are obtained in layers, and the subband signal characteristics corresponding to each layer are encoded in layers. In this way, each layer only needs to encode specific subband signal characteristics, and there is no need to encode the characteristics of the entire audio signal. This not only improves the efficiency of encoding and decoding, but also reduces the encoding and decoding complexity of each layer. According to the importance of the code streams at different layers to the decoding quality, different transmission priorities are flexibly configured for the code streams at different layers to ensure that the more important code streams are transmitted first, which can be suitable for application scenarios with different network bandwidths.

[0408] In addition, in the embodiments of the present application, decoding and feature reconstruction are performed in layers, and synthesis processing is performed based on the subband signals corresponding to each layer. Since each layer only needs to process part of the subband signals, the decoding efficiency can be improved and the decoding complexity can be reduced.

[0409] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An audio encoding method, characterized in that: The method comprises: Decomposing the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal; Performing feature extraction processing at multiple levels based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to the multiple levels; Based on the subband signal characteristics corresponding to the multiple levels, respectively, determining the code streams corresponding to the multiple levels, wherein the code stream of the first level includes a first code stream corresponding to the low-frequency subband signal, the code stream of the second level includes a second code stream corresponding to the high-frequency subband signal, and the code stream of the third level includes a third code stream corresponding to the low-frequency subband signal and a fourth code stream corresponding to the high-frequency subband signal; Corresponding transmission priorities are configured for the code streams corresponding to the multiple layers respectively.

2. The method according to claim 1, characterized in that The transmission priority is positively correlated with a decoding quality index of a code stream corresponding to the layer.

3. The method according to claim 1, characterized in that The plurality of levels include the first level, the second level, and the third level; The performing of multiple levels of feature extraction processing based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to the multiple levels respectively includes: The feature extraction process of the first level is performed in the following manner: performing a first feature extraction process based on the low frequency sub-band signal to obtain a first low frequency sub-band signal feature; wherein the dimension of the first low frequency sub-band signal feature is smaller than the dimension of the low frequency sub-band signal; The feature extraction process of the second level is performed in the following manner: performing a second feature extraction process based on the high frequency sub-band signal to obtain a first high frequency sub-band signal feature; wherein the dimension of the first high frequency sub-band signal feature is smaller than the dimension of the high frequency sub-band signal; The feature extraction process of the third level is performed by: performing a third feature extraction process based on the low frequency sub-band signal and the first low frequency sub-band signal feature to obtain a second low frequency sub-band signal feature; A fourth feature extraction process is performed based on the first high frequency sub-band signal feature to obtain a second high frequency sub-band signal feature.

4. The method according to claim 3, characterized in that The performing a first feature extraction process based on the low-frequency sub-band signal to obtain a first low-frequency sub-band signal feature includes: Performing a first convolution process on the low-frequency sub-band signal to obtain a first convolution feature; Performing a first pooling process on the first convolutional feature to obtain a first pooling feature; Performing a third downsampling process on the first pooled feature to obtain a third downsampling feature; wherein the third downsampling process includes multiple cascaded downsamplings; A second convolution process is performed on the third down-sampling feature to obtain the first low-frequency sub-band signal feature.

5. The method according to claim 3 or 4, characterized in that: The high frequency sub-band signal includes a first number of sample points, where the first number is an integer greater than 2; The performing a second feature extraction process based on the high frequency sub-band signal to obtain a first high frequency sub-band signal feature comprises: Performing frequency domain transformation processing on the first number of sample points respectively to obtain the first number of transformation coefficients; dividing the first number of transform coefficients into a plurality of first subbands; For each of the first sub-bands, the following processing is performed: performing an averaging process based on a second number of transform coefficients in the first sub-band to obtain a first average energy corresponding to the first sub-band, and determining the first average energy as a first sub-band spectrum envelope corresponding to the first sub-band; wherein the second number is a ratio of the first number to the number of the plurality of first sub-bands; The first sub-band spectrum envelopes respectively corresponding to the multiple first sub-bands are determined as the first high-frequency sub-band signal features.

6. The method according to claim 5, characterized in that The performing frequency domain transformation processing based on the first number of sample points respectively to obtain the first number of transformation coefficients includes: Acquire a reference frame high frequency subband signal corresponding to a reference frame audio signal; wherein the reference frame audio signal is a previous frame or a next frame of the audio signal, and the reference frame high frequency subband signal includes the first number of sample points; Based on the first number of sample points in the high-frequency sub-band signal of the reference frame and the first number of sample points in the high-frequency sub-band signal, discrete cosine transform processing is performed on the first number of sample points included in the high-frequency sub-band signal to obtain transform coefficients corresponding to the first number of sample points included in the high-frequency sub-band signal.

7. The method according to claim 3, characterized in that The performing a third feature extraction process based on the low-frequency sub-band signal and the first low-frequency sub-band signal feature to obtain a second low-frequency sub-band signal feature includes: Performing splicing processing on the low-frequency sub-band signal and the first low-frequency sub-band signal feature to obtain a splicing feature; Performing a third convolution process on the concatenated features to obtain a third convolution feature; Performing a second pooling process on the third convolutional feature to obtain a second pooling feature; Performing a fourth downsampling process on the second pooling feature to obtain a fourth downsampling feature; wherein the fourth downsampling process includes multiple cascaded downsamplings; Performing a fourth convolution process on the fourth down-sampling feature to obtain the second low-frequency sub-band signal feature; wherein the dimension of the second low-frequency sub-band signal feature is smaller than the dimension of the first low-frequency sub-band signal feature.

8. The method according to claim 3, characterized in that The performing a fourth feature extraction process based on the first high frequency sub-band signal feature to obtain a second high frequency sub-band signal feature includes: The following processing is performed for each of the first sub-bands corresponding to the first high frequency sub-band signal feature: Selecting a third number of transform parameters from the second number of transform parameters in the first subband, and determining the third number of transform coefficients as a second subband; wherein the third number is half of the second number; Performing averaging processing based on the third number of transform coefficients in the second sub-band to obtain a second average energy corresponding to the second sub-band, and determining the second average energy as a second sub-band spectrum envelope corresponding to the second sub-band; The second sub-band spectrum envelope corresponding to each of the second sub-bands is determined as the second high-frequency sub-band signal feature.

9. The method according to claim 1, characterized in that: The multiple levels include the first level, the second level and the third level, and the subband signal characteristics corresponding to the first level are first low-frequency subband signal characteristics, the subband signal characteristics corresponding to the second level are first high-frequency subband signal characteristics, and the subband signal characteristics corresponding to the third level are second low-frequency subband signal characteristics and second high-frequency subband signal characteristics; The determining, based on the subband signal characteristics respectively corresponding to the multiple layers, the code streams respectively corresponding to the multiple layers comprises: Performing encoding processing on the index value of the first low-frequency signal feature to obtain the first code stream corresponding to the first layer; Encoding the index value of the first high-frequency signal feature to obtain the second code stream corresponding to the second layer; The bit rate of the first bit stream is greater than the bit rate corresponding to the second bit stream, and the bit rate is positively correlated with the decoding quality index of the corresponding bit stream; Performing encoding processing on the index value of the second low-frequency signal feature to obtain the third code stream corresponding to the third level; Encoding the index value of the second high-frequency signal feature to obtain the fourth code stream corresponding to the third level; The bit rate of the first code stream is greater than the bit rate of the third code stream, the bit rate of the third code stream is greater than the bit rate of the second code stream, the bit rate of the second code stream is equal to the bit rate of the fourth code stream, and the bit rate is positively correlated with the decoding quality index of the corresponding code stream.

10. The method according to claim 1, characterized in that The determining, based on the subband signal characteristics respectively corresponding to the multiple layers, the code streams respectively corresponding to the multiple layers comprises: Quantizing the sub-band signal features corresponding to each of the layers to obtain index values ​​of the sub-band signal features; The index value of the sub-band signal feature is encoded to obtain a code stream corresponding to the layer.

11. The method according to claim 1, characterized in that: The multiple levels include the first level, the second level and the third level, and the first code stream corresponding to the first level is obtained by encoding the index value of the first low-frequency signal feature, the second code stream corresponding to the second level is obtained by encoding the index value of the first high-frequency signal feature, the third level corresponds to the third code stream and the fourth code stream, the third code stream is obtained by encoding the index value of the second low-frequency signal feature, and the fourth code stream is obtained by encoding the index value of the second high-frequency signal feature; The configuring of transmission priorities corresponding to the code streams corresponding to the multiple layers respectively includes: configuring a first transmission priority for the first code stream corresponding to the first layer; configuring a second transmission priority for the second code stream corresponding to the second layer; wherein the first transmission priority is higher than the second transmission priority; configuring a third transmission priority for the third code stream corresponding to the third layer; configuring a fourth transmission priority for the fourth code stream corresponding to the third layer; The first transmission priority is higher than the third transmission priority, the third transmission priority is higher than the second transmission priority, and the second transmission priority is the same as the fourth transmission priority.

12. The method according to claim 1, characterized in that The step of decomposing the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal includes: Sampling the audio signal at a first sampling frequency to obtain a sampled signal; wherein the sampled signal includes a plurality of sample points sampled from the audio signal; Performing low-pass filtering on the sampled signal, and performing a first down-sampling process on the obtained low-pass filtering result to obtain the low-frequency sub-band signal of the second sampling frequency; Performing high-pass filtering on the sampled signal, and performing a second down-sampling process on the obtained high-pass filtering result to obtain the high-frequency sub-band signal of the second sampling frequency; The second sampling frequency is half of the first sampling frequency, and the bandwidths of the low-frequency sub-band signal and the high-frequency sub-band signal are the same.

13. An audio decoding method, characterized in that: The method comprises: Based on the code streams corresponding to the multiple layers, respectively, determining the subband signal characteristics corresponding to each layer, wherein different layers correspond to different transmission priorities, the code stream of the first layer includes a first code stream corresponding to the low-frequency subband signal, the code stream of the second layer includes a second code stream corresponding to the high-frequency subband signal, and the code stream of the third layer includes a third code stream corresponding to the low-frequency subband signal and a fourth code stream corresponding to the high-frequency subband signal; Performing feature reconstruction processing on the sub-band signal features corresponding to each of the layers to obtain the sub-band signals corresponding to each of the layers; The sub-band signals respectively corresponding to the multiple layers are synthesized into an audio signal.

14. The method according to claim 13, characterized in that The transmission priority is positively correlated with a decoding quality index of a code stream corresponding to the layer.

15. The method according to claim 13, characterized in that The multiple levels include the first level, the second level and the third level, the first level corresponds to the first low-frequency sub-band signal characteristics, the second level corresponds to the first high-frequency sub-band signal characteristics, and the third level corresponds to the second low-frequency sub-band signal characteristics and the second high-frequency sub-band signal characteristics; The performing feature reconstruction processing on the sub-band signal features corresponding to each of the layers to obtain the sub-band signals corresponding to each of the layers comprises: The feature reconstruction processing of the first level is performed in the following manner: performing a first feature reconstruction processing based on the first low-frequency sub-band signal feature to obtain a first low-frequency sub-band signal; wherein the dimension of the first low-frequency sub-band signal is greater than the dimension of the first low-frequency sub-band signal feature; The feature reconstruction processing of the second level is performed in the following manner: performing a second feature reconstruction processing based on the first high frequency sub-band signal feature to obtain a first high frequency sub-band signal; wherein the dimension of the first high frequency sub-band signal is greater than the dimension of the first high frequency sub-band signal feature; The feature reconstruction processing of the third level is performed by: performing a third feature reconstruction processing on the feature of the second low-frequency sub-band signal to obtain a second low-frequency sub-band signal; A fourth feature reconstruction process is performed based on the first high frequency sub-band signal feature and the second high frequency sub-band signal feature to obtain a second high frequency sub-band signal.

16. The method according to claim 15, characterized in that The performing a first feature reconstruction process based on the first low-frequency sub-band signal feature to obtain a first low-frequency sub-band signal includes: Performing a first convolution process on the first low-frequency sub-band signal feature to obtain a first convolution feature; Performing a first upsampling process on the first convolution feature to obtain a first upsampling feature; wherein the first upsampling process includes multiple cascade upsampling; Performing a first pooling process on the first up-sampled features to obtain first pooled features; A second convolution process is performed on the first pooled features to obtain the first low-frequency sub-band signal.

17. The method according to claim 15 or 16, characterized in that The performing a second feature reconstruction process based on the first high frequency sub-band signal feature to obtain a first high frequency sub-band signal comprises: Performing frequency domain transform processing on a first number of sample points included in the first low-frequency subband signal to obtain the first number of low-frequency transform coefficients; Based on the low-frequency transform coefficients of the second half of the first number of low-frequency transform coefficients, performing spectrum replication processing twice to obtain the first number of reference transform coefficients included in the first high-frequency reference sub-band signal; wherein the frequency of the sample points corresponding to the low-frequency transform coefficients of the second half is higher than the frequency of the sample points corresponding to the low-frequency transform coefficients of the first half; Based on a plurality of first sub-band spectral envelopes corresponding to the first high-frequency sub-band signal feature, performing a first gain control on the first number of reference transform coefficients included in the first high-frequency reference sub-band signal to obtain the first number of first gain reference transform coefficients; An inverse frequency domain transform process is performed on the first number of first gain reference transform coefficients to obtain a first high frequency sub-band signal.

18. The method according to claim 17, characterized in that The step of performing a first gain control on the first number of reference transform coefficients included in the first high-frequency reference sub-band signal based on a plurality of first sub-band spectral envelopes corresponding to the first high-frequency sub-band signal feature to obtain the first number of first gain reference transform coefficients comprises: Dividing the first number of reference transform coefficients into a plurality of first reference sub-bands; wherein the number of the first reference sub-bands is the same as the number of the first sub-band spectral envelopes; Generate a plurality of different first combinations, each of which includes a first reference subband and a first subband spectral envelope, and perform the following processing on each of the first combinations: determining the first subband spectral envelope as a first average energy; Performing an averaging process based on a second number of reference transform coefficients in the first reference sub-band to obtain a first reference average energy corresponding to the first reference sub-band; wherein the second number is a ratio of the first number to the number of the plurality of first reference sub-bands; determining a ratio of the first average energy to the first reference average energy, and performing a square root processing on the ratio to obtain a first proportional factor; Multiplying the second number of reference transform coefficients in the first reference sub-band by the first scale factor respectively to obtain the second number of first gain reference transform coefficients in the first reference sub-band; The second number of first gain reference transform coefficients in each of the first reference sub-bands are combined to obtain the first number of first gain reference transform coefficients.

19. The method according to claim 15, characterized in that The performing third feature reconstruction processing on the second low-frequency sub-band signal feature to obtain the second low-frequency sub-band signal includes: Performing a third convolution process on the second low-frequency sub-band signal feature to obtain a third convolution feature; Performing a second upsampling process on the third convolution feature to obtain a second upsampling feature; wherein the second upsampling process includes multiple cascade upsampling; Performing a second pooling process on the second up-sampled features to obtain second pooling features; Perform a fourth convolution process on the second pooling feature to obtain a second low-frequency sub-band signal; wherein the second low-frequency sub-band signal is a residual signal of the first low-frequency sub-band signal.

20. The method according to claim 15, characterized in that The performing fourth feature reconstruction processing based on the first high frequency sub-band signal feature and the second high frequency sub-band signal feature to obtain a second high frequency sub-band signal comprises: Determine a plurality of first sub-bands corresponding to the first high frequency sub-band signal characteristic, and determine a plurality of second sub-bands corresponding to the second high frequency sub-band signal characteristic; wherein the number of the plurality of first sub-bands is the same as the number of the plurality of second sub-bands; Determine third subband spectral envelopes corresponding to a plurality of third subbands respectively based on the first subband spectral envelopes corresponding to the plurality of first subbands respectively and the second subband spectral envelopes corresponding to the plurality of second subbands respectively; wherein the number of the plurality of third subbands is the same as the number of the plurality of second subbands; Determine a plurality of the second sub-band spectral envelopes and a plurality of the third sub-band spectral envelopes as a fourth sub-band spectral envelope; Based on the plurality of fourth sub-band spectral envelopes, performing second gain control on the first number of reference transform coefficients included in the first high-frequency reference sub-band signal to obtain the first number of second gain reference transform coefficients; An inverse frequency domain transform process is performed on the first number of second gain reference transform coefficients to obtain a second high frequency sub-band signal.

21. The method according to claim 20, characterized in that The determining, based on the first subband spectral envelopes respectively corresponding to the plurality of first subbands and the second subband spectral envelopes respectively corresponding to the plurality of second subbands, third subband spectral envelopes respectively corresponding to the plurality of third subbands comprises: Generate a plurality of different second combinations, each of which includes one of the first subbands and the second subband corresponding to the first subband, and perform the following processing on each of the second combinations: determining the first sub-band spectrum envelope corresponding to the first sub-band as a first average energy, and determining the second sub-band spectrum envelope corresponding to the second sub-band as a second average energy; Multiplying the first average energy by a second number to obtain a first multiplication result; wherein the second number is the number of high-frequency transform coefficients included in the first sub-band; Multiplying the second average energy by a third number to obtain a second multiplication result; wherein the third number is the number of the high-frequency transform coefficients included in the second sub-band, and the third number is half of the second number; performing a subtraction process on the first multiplication result and the second multiplication result, determining a ratio of the obtained subtraction result to the third quantity as a third average energy corresponding to the third subband, and determining the third average energy as a third subband spectrum envelope corresponding to the third subband; The second number of high-frequency transform coefficients included in the first sub-band is a union of the following two: the high-frequency transform coefficients included in the second sub-band and the high-frequency transform coefficients included in the third sub-band.

22. The method according to claim 20, characterized in that The step of performing a second gain control on the first number of reference transform coefficients included in the first high-frequency reference sub-band signal based on the plurality of fourth sub-band spectral envelopes to obtain the first number of second gain reference transform coefficients comprises: Dividing the first number of reference transform coefficients into a plurality of second reference sub-bands; wherein the number of the second reference sub-bands is the same as the number of the fourth sub-band spectral envelopes; Generate a plurality of different third combinations, each of the third combinations comprising one of the second reference sub-bands and one of the fourth sub-band spectral envelopes, and perform the following processing for each of the third combinations: Determining the fourth sub-band spectral envelope as a fourth average energy; Performing averaging processing based on a third number of reference transform coefficients in the second reference sub-band to obtain a second reference average energy corresponding to the second reference sub-band; wherein the third number is a ratio of the first number to the number of the plurality of second reference sub-bands; determining a ratio of the fourth average energy to the second reference average energy, and performing a square root processing on the ratio to obtain a second proportional factor; Multiplying the third number of reference transform coefficients in the second reference sub-band by the second scale factor respectively to obtain the third number of second gain reference transform coefficients in the second reference sub-band; The third number of second gain reference transform coefficients in each of the second reference sub-bands are combined to obtain the first number of second gain reference transform coefficients.

23. The method according to claim 13, characterized in that The multiple levels include the first level, the second level and the third level, the first level corresponds to the first low-frequency sub-band signal, the second level corresponds to the first high-frequency sub-band signal, and the third level corresponds to the second low-frequency sub-band signal feature and the second high-frequency sub-band signal feature; The step of synthesizing the sub-band signals respectively corresponding to the plurality of layers into an audio signal comprises: performing a summation process on the first low-frequency sub-band signal and the second low-frequency sub-band signal to obtain a first summation result; Performing a fifth upsampling process on the first summation result of the second sampling frequency to obtain a fifth upsampling result of the first sampling frequency; Performing a sixth upsampling process on the second high frequency subband signal of the second sampling frequency to obtain a sixth upsampling result of the first sampling frequency; The fifth up-sampling result and the sixth up-sampling result are synthesized to obtain the audio signal.

24. The method according to claim 13, characterized in that The determining, based on the code streams corresponding to the multiple layers, the subband signal characteristics corresponding to each layer respectively, comprises: Decoding the code streams corresponding to the multiple layers to obtain index values ​​of the code streams corresponding to each layer; The index value of the code stream corresponding to each of the layers is inversely quantized to obtain the subband signal feature corresponding to each of the layers.

25. An audio encoding device, characterized in that: The device comprises: A decomposition module is used to decompose the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal; A feature extraction module, configured to perform feature extraction processing at multiple levels based on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain sub-band signal features corresponding to the multiple levels; A coding module, configured to determine the code streams corresponding to the multiple layers respectively based on the subband signal characteristics corresponding to the multiple layers respectively, wherein the code stream of the first layer includes a first code stream corresponding to the low-frequency subband signal, the code stream of the second layer includes a second code stream corresponding to the high-frequency subband signal, and the code stream of the third layer includes a third code stream corresponding to the low-frequency subband signal and a fourth code stream corresponding to the high-frequency subband signal; The configuration module is used to configure corresponding transmission priorities for the code streams corresponding to the multiple layers.

26. An audio decoding device, characterized in that: The device comprises: A decoding module, configured to determine the subband signal characteristics corresponding to each layer based on the code streams corresponding to the multiple layers, wherein different layers correspond to different transmission priorities, the code stream of the first layer includes a first code stream corresponding to the low-frequency subband signal, the code stream of the second layer includes a second code stream corresponding to the high-frequency subband signal, and the code stream of the third layer includes a third code stream corresponding to the low-frequency subband signal and a fourth code stream corresponding to the high-frequency subband signal; A feature reconstruction module, used for performing feature reconstruction processing on the sub-band signal features corresponding to each of the layers to obtain the sub-band signals corresponding to each of the layers; The synthesis module is used to synthesize the sub-band signals corresponding to the multiple layers into an audio signal.

27. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, used to implement the audio encoding method described in any one of claims 1 to 12 or the audio decoding method described in any one of claims 13-24 when executing the executable instructions stored in the memory.

28. A computer-readable storage medium, characterized in that: Executable instructions are stored, and the executable instructions are used to implement the audio encoding method described in any one of claims 1 to 12 or the audio decoding method described in any one of claims 13-24 when executed by a processor.

29. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the audio encoding method according to any one of claims 1 to 12 or the audio decoding method according to any one of claims 13 to 24 is implemented.

30. A method for processing a code stream, characterized in that: A code stream is stored on a non-volatile computer-readable storage medium, wherein the code stream is generated according to the audio encoding method according to any one of claims 1 to 12, or is decoded based on the audio decoding method according to any one of claims 13 to 24.

Images