Audio processing method and device, equipment, storage medium and computer program product

By performing multi-channel decomposition and quantization encoding processing on the audio signal, the problem of low audio encoding efficiency in the prior art is solved, and efficient audio signal compression and recovery is achieved.

CN118942471APending Publication Date: 2024-11-12TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411353107.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-06-15
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art is difficult to effectively improve audio encoding efficiency while ensuring audio quality.

Method used

By performing multi-channel signal decomposition processing on the audio signal, multiple subband signals are obtained, and each subband signal is compressed and quantized, and then quantized and decompressed at the decoding end to recover the audio signal.

Benefits of technology

While ensuring audio quality, the audio encoding efficiency is significantly improved, the amount of data is reduced, and high-quality audio signal recovery is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942471A_ABST
    Figure CN118942471A_ABST
Patent Text Reader

Abstract

The invention provides an audio processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps that multi-channel signal decomposition processing is carried out on an audio signal, N sub-band signals of the audio signal are obtained, N is an integer larger than 2, and frequency bands of the N sub-band signals are sequentially increased; performing signal compression processing on each sub-band signal to obtain a sub-band signal feature of each sub-band signal; and performing quantization coding processing on the sub-band signal feature of each sub-band signal to obtain a code stream of each sub-band signal. According to the invention, the audio coding efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the patent application with application number 202210681037.X, application date June 15, 2022, and invention name “audio processing method, device, equipment, storage medium and computer program product”. Technical Field

[0002] The present application relates to data processing technology, and in particular to an audio processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0003] Audio coding technology is a core technology in communication services including remote audio and video calls. To put it simply, speech coding technology is to use less network bandwidth resources to transmit as much voice information as possible. From the perspective of Shannon's information theory, speech coding is a kind of source coding. The purpose of source coding is to compress the amount of data we want to transmit as much as possible at the encoding end, remove the redundancy in the information, and at the same time be able to restore it losslessly (or nearly losslessly) at the decoding end.

[0004] However, there is no effective solution in the related technology on how to effectively improve the efficiency of audio encoding while ensuring the audio quality. Summary of the invention

[0005] The embodiments of the present application provide an audio processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve audio encoding efficiency while ensuring audio quality.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application provides an audio processing method, including:

[0008] Performing multi-channel signal decomposition processing on the audio signal to obtain N sub-band signals of the audio signal, wherein N is an integer greater than 2, and frequency bands of the N sub-band signals are increased in sequence;

[0009] Performing signal compression processing on each of the sub-band signals to obtain a sub-band signal feature of each of the sub-band signals;

[0010] The sub-band signal characteristics of each of the sub-band signals are quantized and encoded to obtain a code stream of each of the sub-band signals.

[0011] The present application provides an audio processing method, including:

[0012] Performing quantization decoding processing on N code streams to obtain sub-band signal characteristics corresponding to each of the code streams;

[0013] Wherein, N is an integer greater than 2, and the N code streams are obtained by respectively encoding N sub-band signals obtained after the audio signal is subjected to multi-channel signal decomposition processing;

[0014] Performing signal decompression processing on each of the sub-band signal features to obtain an estimated sub-band signal corresponding to each of the sub-band signal features;

[0015] Signal synthesis processing is performed on the multiple estimated sub-band signals to obtain synthesized audio signals corresponding to the multiple code streams.

[0016] The present application provides an audio processing device, including:

[0017] A decomposition module, configured to perform multi-channel signal decomposition processing on the audio signal to obtain N sub-band signals of the audio signal, wherein N is an integer greater than 2, and the frequency bands of the N sub-band signals are increased in sequence;

[0018] A compression module, used for performing signal compression processing on each of the sub-band signals to obtain sub-band signal characteristics of each of the sub-band signals;

[0019] The encoding module is used to perform quantization encoding processing on the sub-band signal characteristics of each of the sub-band signals to obtain a code stream of each of the sub-band signals.

[0020] The present application provides an audio processing device, including:

[0021] A decoding module, used to perform quantization decoding processing on N code streams to obtain sub-band signal characteristics corresponding to each of the code streams;

[0022] Wherein, N is an integer greater than 2, and the N code streams are obtained by respectively encoding N sub-band signals obtained after the audio signal is subjected to multi-channel signal decomposition processing;

[0023] A decompression module, used to perform signal decompression processing on each of the sub-band signal features to obtain an estimated sub-band signal corresponding to each of the sub-band signal features;

[0024] The synthesis module is used to perform signal synthesis processing on the multiple estimated sub-band signals to obtain synthesized audio signals corresponding to the multiple code streams.

[0025] An embodiment of the present application provides an electronic device for audio processing, the electronic device comprising:

[0026] A memory for storing executable instructions;

[0027] The processor is used to implement the audio processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0028] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the audio processing method provided in the embodiment of the present application.

[0029] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the audio processing method provided in the embodiment of the present application.

[0030] The embodiments of the present application have the following beneficial effects:

[0031] By decomposing an audio signal into multiple sub-band signals, performing differentiated signal processing on the sub-band signals, and quantizing and encoding the sub-band signal features with reduced feature dimensions, the audio coding efficiency is improved while ensuring the audio quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of spectrum comparison at different bit rates provided in an embodiment of the present application;

[0033] Figure 2 It is a schematic diagram of the architecture of the audio codec system provided in an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 4 is a flowchart of an audio processing method provided in an embodiment of the present application;

[0036] Figure 5 is a flowchart of an audio processing method provided in an embodiment of the present application;

[0037] Figure 6 is a schematic diagram of an end-to-end voice communication link provided by an embodiment of the present application;

[0038] Figure 7 It is a flowchart of a speech encoding and decoding method based on subband decomposition and neural network provided in an embodiment of the present application;

[0039] Fig. 8A is a schematic diagram of a filter bank provided in an embodiment of the present application;

[0040] Figure 8B It is a schematic diagram of the principle of obtaining 4-channel sub-band signals based on the filter bank provided in an embodiment of the present application;

[0041] Figure 8C It is a schematic diagram of the principle of obtaining sub-band signals of three channels based on a filter bank provided in an embodiment of the present application;

[0042] Fig.9A is a schematic diagram of a common convolutional network provided in an embodiment of the present application;

[0043] Fig. 9B is a schematic diagram of a dilated convolutional network provided in an embodiment of the present application;

[0044] Fig.10 is a schematic diagram of frequency band expansion provided by an embodiment of the present application;

[0045] Fig.11 is a network structure diagram of channel analysis provided by an embodiment of the present application;

[0046] Fig.12 This is the network structure of channel synthesis provided in the embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0048] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0051] 1) Neural Network (NN): It is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.

[0052] 2) Deep Learning (DL): It is a new research direction in the field of Machine Learning (ML). Deep learning is to learn the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0053] 3) Quantization: It refers to the process of approximating the continuous value (or a large number of discrete values) of a signal to a finite number (or a small number) of discrete values. Quantization includes vector quantization (VQ) and scalar quantization.

[0054] Among them, vector quantization is an effective lossy compression technology, and its theoretical basis is Shannon's rate-distortion theory. The basic principle of vector quantization is to replace the input vector with the index of the codeword that best matches the input vector in the code book for transmission and storage, while decoding only requires a simple table lookup operation. For example, a number of scalar data are combined into a vector space, and the vector space is divided into several small areas. When quantizing, the vectors that fall into the small areas are replaced by the corresponding indexes instead of the input vectors.

[0055] Scalar quantization is the quantization of scalars, that is, one-dimensional vector quantization, which divides the dynamic range into several small intervals, each of which has a representative value (i.e., index). When the input signal falls into a certain interval, the input signal is quantized into the representative value.

[0056] 4) Entropy coding: A lossless coding method that does not lose any information according to the entropy principle during the encoding process. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exp-Golomb coding, and arithmetic coding.

[0057] 5) Quadrature Mirror Filters (QMF): It is a filter pair that includes analysis and synthesis. The QMF analysis filter group is used to decompose the sub-band signal to reduce the signal bandwidth so that each sub-band signal can be processed smoothly through its own channel; the QMF synthesis filter group is used to synthesize the sub-band signals restored by the decoding end, for example, to reconstruct the original audio signal through zero-value interpolation and bandpass filtering.

[0058] Voice coding technology is to use less network bandwidth resources to transmit as much voice information as possible. The compression rate of voice codec can reach more than 10 times, that is, after the original 10MB of voice data is compressed by the encoder, only 1MB is needed to transmit, which greatly reduces the bandwidth resources consumed by transmitting information. For example, for a wideband voice signal with a sampling rate of 16000Hz, if a 16-bit sampling depth (the degree of precision of recording voice intensity during sampling) is used, the bit rate (the amount of data transmitted per unit time) of the uncompressed version is 256kbps; if voice coding technology is used, even if it is lossy coding, within the bit rate range of 10-20kbps, the quality of the reconstructed voice signal can be close to the uncompressed version, and even the auditory perception is no different. If a higher sampling rate service is required, such as 32000Hz ultra-wideband voice, the bit rate range must be at least 30kbps.

[0059] In the communication system, in order to ensure smooth communication, the industry deploys standard voice codec protocols, such as standards from international and domestic standard organizations such as ITU-T, 3GPP, IETF, AVS, CCSA, G.711, G.722, AMR series, EVS, OPUS and other standards. Figure 1 A spectrum comparison diagram at different bit rates is given to demonstrate the relationship between compression bit rate and quality. Curve 101 is the spectrum curve of the original speech, that is, the signal without compression; Curve 102 is the spectrum curve of the OPUS encoder at a bit rate of 20kbps; Curve 103 is the spectrum curve of the OPUS encoder at a bit rate of 6kbps. Figure 1 It can be seen that as the encoding bit rate increases, the compressed signal is closer to the original signal.

[0060] In the related art, speech coding is roughly based on the following coding principles: speech coding can directly encode speech waveform samples sample by sample; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0061] The above coding principles all come from voice signal modeling, that is, a compression method based on signal processing. In order to improve the coding efficiency while ensuring the voice quality relative to the compression method based on signal processing. The embodiments of the present application provide an audio processing method, a device, an electronic device, a computer-readable storage medium and a computer program product, which can improve the coding efficiency. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a terminal device, or can be implemented as a server, or can be implemented by a terminal device and a server in collaboration. The following is an example of an electronic device implemented as a terminal device.

[0062] For example, see Figure 2 , Figure 2 It is a schematic diagram of the architecture of the audio encoding and decoding system 100 provided in an embodiment of the present application. The audio decoding system 100 includes: a server 200, a network 300, a terminal device 400 (i.e., an encoding end) and a terminal device 500 (i.e., a decoding end), wherein the network 300 can be a local area network, a wide area network, or a combination of the two.

[0063] In some embodiments, a client 410 is running on the terminal device 400, and the client 410 can be various types of clients, such as an instant messaging client, a web conference client, a live broadcast client, a browser, etc. In response to an audio collection instruction triggered by a sender (such as an initiator of a web conference, a host, an initiator of a voice call, etc.), the client 410 calls the microphone of the terminal device 400 to collect audio signals, and encodes the collected audio signals to obtain a bit stream.

[0064] For example, the client 410 calls the audio processing method provided in the embodiment of the present application to encode the collected audio signal, that is, performs multi-channel signal decomposition processing on the audio signal to obtain N sub-band signals of the audio signal, performs signal compression processing on each sub-band signal to obtain the sub-band signal characteristics of each sub-band signal; performs quantization encoding processing on the sub-band signal characteristics of each sub-band signal to obtain a code stream for each sub-band signal.

[0065] The client 410 can send the code stream (i.e., the low-frequency code stream and the high-frequency code stream) to the server 200 through the network 300, so that the server 200 sends the code stream to the terminal device 500 associated with the recipient (such as the participants, audience, recipient of the voice call, etc. of the online conference).

[0066] After receiving the code stream sent by the server 200, the client 510 (such as an instant messaging client, a web conference client, a live broadcast client, a browser, etc.) can decode the code stream to obtain an audio signal, thereby realizing audio communication.

[0067] For example, the client 510 calls the audio processing method provided in the embodiment of the present application to decode the received code stream, that is, performs quantization decoding processing on N code streams to obtain sub-band signal characteristics corresponding to each code stream; performs signal decompression processing on each sub-band signal characteristic to obtain an estimated sub-band signal corresponding to each sub-band signal characteristic; performs signal synthesis processing on multiple estimated sub-band signals to obtain a decoded audio signal.

[0068] In some embodiments, the embodiments of the present application can be implemented with the help of cloud technology (Cloud Technology). Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0069] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The service interaction function between the above servers 200 can be realized through cloud technology.

[0070] For example, Figure 2 The server 200 shown in the figure can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. Figure 2 The terminal devices 400 and 500 shown in the figure can be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, vehicle-mounted terminals, etc., but are not limited thereto. The terminal devices (such as the terminal devices 400 and 500) and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0071] In some embodiments, the terminal device or server 200 can also implement the audio processing method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP, a web conference APP, or an instant messaging APP, etc.; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0072] In some embodiments, multiple servers may form a blockchain, and server 200 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the audio processing method provided in the embodiment of the present application (such as audio processing logic and code stream) may be stored on the blockchain.

[0073] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, and takes the electronic device 500 as a terminal device as an example. Figure 3 The electronic device 500 shown includes: at least one processor 520, a memory 550, at least one network interface 530 and a user interface 540. The various components in the electronic device 500 are coupled together via a bus system 550. It is understood that the bus system 550 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 550 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 550 is not described in detail. Figure 3 Various buses are labeled as bus system 550 .

[0074] The processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0075] The user interface 540 includes one or more output devices 541 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 540 also includes one or more input devices 542, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0076] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 520.

[0077] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0078] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0079] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0080] A network communication module 552, used to reach other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0081] a presentation module 553 for enabling presentation of information via one or more output devices 541 (e.g., display screen, speaker, etc.) associated with the user interface 540 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0082] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 542 and translate the detected inputs or interactions.

[0083] In some embodiments, the audio processing device provided in the embodiments of the present application can be implemented in software. Figure 3 An audio processing device 555 stored in a memory 550 is shown, which may be software in the form of a program or plug-in, including the following software modules: a decomposition module 5551, a compression module 5552, an encoding module 5553, or a decoding module 5554, a decompression module 5555, and a synthesis module 5556, wherein the decomposition module 5551, the compression module 5552, and the encoding module 5553 are used to implement an audio encoding function, and the decoding module 5554, the decompression module 5555, and the synthesis module 5556 are used to implement an audio decoding function. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented.

[0084] As mentioned above, the audio processing method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 4 , Figure 4 The flowchart of the audio processing method provided in the embodiment of the present application is as follows: Figure 4 The steps shown are explained.

[0085] In step 101, a multi-channel signal decomposition process is performed on an audio signal to obtain N sub-band signals of the audio signal, wherein N is an integer greater than 2, and the frequency bands of the N sub-band signals are increased in sequence.

[0086] As an example of obtaining an audio signal, the encoding end responds to an audio capture instruction triggered by a sender (such as the initiator of a web conference, a host, the initiator of a voice call, etc.), and calls the microphone of the terminal device of the encoding end to collect the audio signal to obtain the audio signal (also called the input signal).

[0087] After acquiring the audio signal, the audio signal is decomposed into multiple sub-band signals through the QMF analysis filter. Since the sub-band signals with low frequencies have a greater impact on audio coding, the sub-band signals are subsequently subjected to differentiated signal processing.

[0088] In some embodiments, multi-channel signal decomposition processing is implemented by multi-layer two-channel sub-band decomposition; multi-channel signal decomposition processing is performed on the audio signal to obtain N sub-band signals of the audio signal, including: performing two-channel sub-band decomposition processing on the audio signal of the first layer to obtain a low-frequency sub-band signal of the first layer and a high-frequency sub-band signal of the first layer; performing two-channel sub-band decomposition processing on the i-th layer of the sub-band signal to obtain a low-frequency sub-band signal of the i+1th layer and a high-frequency sub-band signal of the i+1th layer; wherein the sub-band signal of the i-th layer is the low-frequency sub-band signal of the i-th layer, or the high-frequency sub-band signal of the i-th layer and the low-frequency sub-band signal of the i-th layer, i is an increasing natural number and has a value range of 1≤i<N; the sub-band signal of the last layer and the high-frequency sub-band signal of each layer that has not been subjected to two-channel sub-band decomposition processing are used as the sub-band signal of the audio signal.

[0089] The subband signal includes a plurality of sample points obtained by sampling the audio signal. Figure 8B As shown, the audio signal is subjected to an iterative two-layer 2-channel QMF analysis filter, that is, the audio signal is iteratively decomposed into two layers of two-channel sub-bands to obtain a 4-channel sub-band signal (x k (n), n=1,2,3,4), that is, the audio signal is subjected to the first layer of two-channel sub-band decomposition processing to obtain the first layer of low-frequency sub-band signal and the first layer of high-frequency sub-band signal; the first layer of low-frequency sub-band signal is subjected to the second layer of two-channel sub-band decomposition processing to obtain the first layer of low-frequency sub-band signal in the second layer of low-frequency sub-band signal x1(n) and the first layer of low-frequency sub-band signal in the second layer of high-frequency sub-band signal x2(n); the first layer of high-frequency sub-band signal is subjected to the second layer of two-channel sub-band decomposition processing to obtain the first layer of high-frequency sub-band signal in the second layer of low-frequency sub-band signal x3(n) and the first layer of high-frequency sub-band signal in the second layer of high-frequency sub-band signal x4(n), thereby obtaining the 4-channel sub-band signal x k (n),n=1,2,3,4.

[0090] Figure 8CAs shown, the audio signal is subjected to two-layer iterative 2-channel QMF analysis filter, that is, the audio signal is iteratively decomposed into two layers of two-channel sub-bands to obtain a 3-channel sub-band signal (x 2,1 (n), x 2,2 (n), x 1,2 (n)), that is, the audio signal is decomposed into two channels at the first layer to obtain the low-frequency sub-band signal at the first layer and the high-frequency sub-band signal at the first layer x 1,2 (n); Perform two-channel subband decomposition processing on the low-frequency subband signal of the first layer, and obtain the low-frequency subband signal of the first layer in the low-frequency subband signal of the second layer x 2,1 (n) and the low-frequency subband signal of the first layer is the high-frequency subband signal of the second layer x 2,2 (n); not for the high frequency subband signal x of the first layer 1,2 (n) Perform two-channel sub-band decomposition to obtain a three-channel sub-band signal x 2,1 (n), x 2,2 (n), x 1,2 (n).

[0091] In some embodiments, a first-layer two-channel sub-band decomposition process is performed on an audio signal to obtain a first-layer low-frequency sub-band signal and a first-layer high-frequency sub-band signal, including: sampling the audio signal to obtain a sampled signal, wherein the sampled signal includes a plurality of sample points obtained by sampling; performing a first-layer low-pass filtering process on the sampled signal to obtain a first-layer low-pass filtered signal; down-sampling the first-layer low-pass filtered signal to obtain a first-layer low-frequency sub-band signal; performing a first-layer high-pass filtering process on the sampled signal to obtain a first-layer high-pass filtered signal; and down-sampling the first-layer high-pass filtered signal to obtain a first-layer high-frequency sub-band signal.

[0092] It should be noted that the audio signal is a continuous analog signal, the sampling signal is a discrete digital signal, and the sampling point is a sampling value obtained by sampling the audio signal.

[0093] As an example, take the audio signal as an input signal with a sampling rate of Fs=32000 Hz as an example, sample the audio signal to obtain a sampled signal x(n) including 640 sample points. Call the analysis filter (2 channels) in the QMF filter bank, perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, perform high-pass filtering on the sampled signal to obtain a high-pass filtered signal, perform down-sampling on the low-pass filtered signal to obtain a low-frequency subband signal x of the first layer. LB (n), downsample the high-pass filtered signal to obtain the high-frequency sub-band signal of the first layer. LB (n) and x HBThe effective bandwidth of (n) is 0-8kHz and 8-16kHz respectively, x LB (n) and x HB The number of sample points of (n) is 320.

[0094] It should be noted that the QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, which represent the QMF low-pass signal and the QMF high-pass signal respectively. After the low-pass signal and high-pass signal are restored by the decoding end, they are synthesized by the QMF synthesis filter, and the reconstructed signal with a sampling rate of Fs corresponding to the input signal can be restored.

[0095] In some embodiments, a multi-channel signal decomposition process is performed on an audio signal to obtain N sub-band signals of the audio signal, including: sampling the audio signal to obtain a sampled signal, wherein the sampled signal includes multiple sample points obtained by sampling; filtering the sampled signal on the jth channel to obtain the jth filtered signal; down-sampling the jth filtered signal to obtain the jth sub-band signal of the audio signal; wherein j is an increasing natural number and has a value range of 1≤j≤N.

[0096] For example, the QMF analysis filter group can be divided into multiple channels in advance, and the jth channel filter processing is performed on the sampled signal through the filter of the jth channel to obtain the jth filtered signal; the jth filtered signal is downsampled to obtain the jth subband signal of the audio signal.

[0097] In step 102, signal compression processing is performed on each sub-band signal to obtain a sub-band signal feature of each sub-band signal.

[0098] Among them, the characteristic dimension of the subband signal feature of each subband signal is not positively correlated with the frequency band of each subband signal, and the characteristic dimension of the subband signal feature of the Nth subband signal is smaller than the characteristic dimension of the subband signal feature of the first subband signal, wherein the non-positive correlation means that the characteristic dimension of the subband signal feature decreases or remains unchanged as the frequency band of the subband signal increases, that is, the characteristic dimension of a certain subband signal feature is less than or equal to the characteristic dimension of the previous subband signal feature. The subband signal can be compressed by signal compression (i.e., channel analysis) to reduce the data volume of the subband signal, that is, the dimension of the subband signal feature of the subband signal is smaller than the dimension of the subband signal.

[0099] For example, since the lower the frequency of a sub-band signal, the greater the impact on audio coding, differential signal processing is performed on the sub-band signal so that the feature dimension of the sub-band signal feature of the sub-band signal biased towards high frequency is lower.

[0100] In some embodiments, signal compression processing is performed on each subband signal to obtain subband signal features of each subband signal, including: performing the following processing on any subband signal: calling a first neural network model corresponding to the subband signal; performing feature extraction processing on the subband signal through the first neural network model to obtain subband signal features of the subband signal; wherein the structural complexity of the first neural network model is positively correlated with the dimension of the subband signal features of the subband signal.

[0101] For example, the subband signal is subjected to feature extraction processing by the first neural network model to obtain subband signal features, so as to minimize the feature dimension of the subband signal features while ensuring the completeness of the subband signal features. The embodiment of the present application is not limited to the structure of the first neural network model.

[0102] In some embodiments, feature extraction processing is performed on the subband signal through the first neural network model to obtain subband signal features of the subband signal, including: performing the following processing on the subband signal through the first neural network model: convolution processing on the subband signal to obtain convolution features of the subband signal; pooling processing on the convolution features to obtain pooling features of the subband signal; downsampling processing on the pooling features to obtain downsampled features of the subband signal; convolution processing on the downsampled features to obtain subband signal features of the subband signal.

[0103] like Fig.11 As shown, the neural network model of the first channel is called based on the subband signal x1(n) to generate a lower dimensional feature vector F1(n), that is, the subband signal feature. First, the input subband signal x1(n) is convolved by causal convolution to obtain a 24×160 convolution feature. Then, the 24×160 convolution feature is pooled by a factor of 2 (i.e., preprocessed) to obtain a 24×80 pooled feature. Next, the 24×80 pooled feature is downsampled to obtain a 192×1 downsampled feature. Finally, the 192×1 downsampled feature is convolved by causal convolution to obtain a 32-dimensional feature vector F1(n).

[0104] In some embodiments, the downsampling process is implemented through multiple cascaded coding layers; the pooling features are downsampled to obtain the downsampled features of the subband signal, including: downsampling the pooling features through the first coding layer in the multiple cascaded coding layers; outputting the downsampling result of the first coding layer to the subsequent cascaded coding layers, continuing the downsampling process and outputting the downsampling results through the subsequent cascaded coding layers until the downsampling process is output to the last coding layer; and using the downsampling result output by the last coding layer as the downsampling feature of the subband signal.

[0105] like Fig.11As shown, cascade three coding blocks (i.e., coding layers) with different downsampling factors (Down_factor) to downsample the pooled features, that is, first downsample the 24×80 pooled features through the coding block with the downsampling factor (Down_factor=2) to obtain a 48×40 downsampled result, then downsample the 48×40 downsampled result through the coding block with the downsampling factor (Down_factor=5) to obtain a 96×8 downsampled result, and finally downsample the 96×8 downsampled result through the coding block with the downsampling factor (Down_factor=8) to obtain a 192×1 downsampled feature. Taking the coding block (Down_factor=4) as an example, one or more dilated convolutions can be performed first, and pooling can be performed based on the Down_factor to achieve the downsampling effect.

[0106] It should be noted that after processing through one coding layer, the understanding of downsampling features is deepened. After learning through multiple coding layers, the downsampling features of low-frequency sub-band signals can be gradually and accurately learned. Through the cascaded coding layers, the downsampling features of low-frequency sub-band signals with progressive accuracy can be obtained.

[0107] In some embodiments, signal compression processing is performed on each subband signal to obtain subband signal features of each subband signal, including: performing feature extraction processing on the first k subband signals respectively to obtain subband signal features corresponding to the first k subband signals respectively; performing frequency band expansion processing on the last Nk subband signals respectively to obtain subband signal features corresponding to the last Nk subband signals respectively; wherein k is an integer and its value range is 1<k<N.

[0108] Among them, k is a multiple of 2. Since the lower the frequency of the sub-band signal, the greater the impact on audio coding, the sub-band signal is subjected to differentiated signal processing, and the compression strength of the higher frequency sub-band signal is greater, that is, the sub-band signal is compressed by another method, namely, band expansion (recovering the broadband speech signal from the band-limited narrowband speech signal) to quickly compress the sub-band signal and extract the high-frequency features of the sub-band signal. Among them, the high-frequency expansion processing is used to reduce the dimension of the sub-band signal to achieve the function of data compression.

[0109] As an example, call the QMF analysis filter (2-channel QMF) and perform downsampling. Figure 8C As shown, the 3-channel decomposition is realized by QMF analysis filter, and finally 3 sub-band signals x can be obtained HB (n), x 2,1 (n) and x 2,2 (n). Figure 8C As shown, x 2,1 (n) and x 2,2(n) are the 0-4kHz and 4-8kHz spectra, respectively, which are generated by two iterations of 2-channel QMF analysis filtering, equivalent to x1(n) and x2(n) in the first implementation. 2,1 (n) and x 2,2 (n) contains 160 sample points respectively. Figure 8C As shown, it corresponds to the 8-16kHz frequency band, and no detailed analysis is required. Therefore, only one QMF high-pass filter is required to generate a high-frequency sub-band signal x for the original 32kHz sampled input signal. HB (n), each frame contains 320 sample points.

[0110] For x 2,1 (n) and x 2,2 (n) The corresponding two sub-band signals can call the neural network model ( Fig.11 The first channel and the second channel of the sub-band signal are extracted. As a result, the feature vectors F1(n) and F2(n) of the corresponding sub-band signals are generated, and the dimensions are 32 and 16 respectively. For the high-frequency sub-band signal x including 320 points HB (n) Through frequency band expansion, the feature vector F of the corresponding sub-band signal is generated HB (n).

[0111] In some embodiments, frequency band extension processing is performed on the last Nk sub-band signals respectively to obtain sub-band signal features corresponding to the last Nk sub-band signals respectively, including: performing the following processing on any sub-band signal among the last Nk sub-band signals: performing frequency domain transform processing based on multiple sample points included in the sub-band signal to obtain transform coefficients corresponding to the multiple sample points respectively; dividing the transform coefficients corresponding to the multiple sample points respectively into multiple sub-bands; performing mean processing on the transform coefficients included in each sub-band to obtain the average energy corresponding to each sub-band, and using the average energy as the sub-band spectral envelope corresponding to each sub-band; determining the sub-band spectral envelopes corresponding to the multiple sub-bands as the sub-band signal features corresponding to the sub-band signal.

[0112] It should be noted that the frequency domain transformation method of the embodiment of the present application includes modified discrete cosine transform (MDCT, Modified Discrete Cosine Transform), discrete cosine transform (DCT, Discrete Cosine Transform), fast Fourier transform (FFT, Fast Fourier Transform), etc., and the embodiment of the present application is not limited to the method of frequency domain transformation. The mean processing of the embodiment of the present application includes arithmetic mean and geometric mean, and the embodiment of the present application is not limited to the method of mean processing.

[0113] In some embodiments, frequency domain transform processing is performed based on multiple sample points included in the subband signal to obtain transform coefficients corresponding to the multiple sample points, including: obtaining a reference subband signal of a reference audio signal, wherein the reference audio signal is an audio signal adjacent to the audio signal, and the reference subband signal has the same frequency band as the subband signal; based on the multiple sample points included in the reference subband signal and the multiple sample points included in the subband signal, discrete cosine transform processing is performed on the multiple sample points included in the subband signal to obtain transform coefficients corresponding to the multiple sample points included in the subband signal.

[0114] In some embodiments, the process of performing geometric mean processing on the transform coefficients included in each sub-band is as follows: determining the sum of squares of the transform coefficients corresponding to the sample points included in each sub-band; and determining the ratio of the sum of squares to the number of sample points included in the sub-band to obtain the average energy corresponding to each sub-band.

[0115] As an example, for a high frequency subband signal x comprising 320 points HB (n), call the modified discrete cosine transform (MDCT) to generate 320-point MDCT coefficients (i.e., the transform coefficients corresponding to the multiple sample points included in the high-frequency subband signal). Specifically, if there is a 50% overlap, the n+1th frame of high-frequency data (i.e., the reference audio signal) and the nth frame of high-frequency data (i.e., the audio signal) can be merged (joined), and the 640-point MDCT is calculated to obtain 320-point MDCT coefficients.

[0116] The 320-point MDCT coefficients are divided into N sub-bands (i.e., the transform coefficients corresponding to the multiple sample points are divided into multiple sub-bands). The sub-bands here are multiple adjacent MDCT coefficients that form a group. The 320-point MDCT coefficients can be divided into 8 sub-bands. For example, the 320 points can be evenly distributed, that is, the number of points included in each sub-band is the same. Of course, the embodiment of the present application cannot perform non-uniform division of the 320 points, such as the sub-bands that are biased towards low frequencies include fewer MDCT coefficients (higher frequency resolution), and the sub-bands that are biased towards high frequencies include more MDCT coefficients (lower frequency resolution).

[0117] According to Nyquist sampling theorem (to restore the original signal from the sampled signal without distortion, the sampling frequency should be greater than 2 times the highest frequency of the original signal. When the sampling frequency is less than 2 times the highest frequency of the spectrum, the spectrum of the signal is aliased. When the sampling frequency is greater than 2 times the highest frequency of the spectrum, the spectrum of the signal is not aliased), the 320-point MDCT coefficients above represent the spectrum of 8-16kHz. However, ultra-wideband voice communication does not necessarily require the spectrum to be 16kHz. For example, if the spectrum is set to 14kHz, only the MDCT coefficients of the first 240 points need to be considered. Correspondingly, the number of subbands can be controlled to 6.

[0118] For each subband, the average energy of all MDCT coefficients in the current subband is calculated (i.e., the transform coefficients included in each subband are averaged) as the subband spectrum envelope (the spectrum envelope is a smooth curve passing through the main peak points of the spectrum). For example, the MDCT coefficients included in the current subband are x(n), n = 1, 2, ..., 40, then the average energy Y is calculated by the geometric mean = ((x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40). When the 320-point MDCT coefficients are divided into 8 sub-bands, 8 sub-band spectrum envelopes can be obtained. These 8 sub-band spectrum envelopes are the feature vectors F of the generated high-frequency sub-band signal. HB (n), i.e., high-frequency features.

[0119] In step 103, quantization encoding is performed on the sub-band signal characteristics of each sub-band signal to obtain a code stream of each sub-band signal.

[0120] For example, differentiated signal processing is performed on sub-band signals so that the feature dimension of sub-band signal characteristics of sub-band signals biased towards high frequencies is lower, the sub-band signal characteristics with reduced feature dimensions are quantized and encoded, and the bit stream is transmitted to the decoding end, and the bit stream is decoded by the decoding end to restore the audio signal, thereby improving the audio coding efficiency while ensuring the audio quality.

[0121] In some embodiments, subband signal characteristics of each subband signal are quantized and encoded to obtain a code stream of each subband signal, including: quantizing the subband signal characteristics of each subband signal to obtain an index value of the subband signal characteristics; and entropy encoding the index value of the subband signal characteristics to obtain a code stream of the subband signal.

[0122] For example, for the sub-band signal characteristics of the sub-band signal, scalar quantization (each component is quantized separately) and entropy coding methods can be performed. In addition, the embodiment of the present application does not limit the technical combination of vector quantization (adjacent multiple components are combined into a vector for joint quantization) and entropy coding, and the encoded code stream is transmitted to the decoding end, and the code stream is decoded by the decoding end.

[0123] As mentioned above, the audio processing method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 5 , Figure 5 The flowchart of the audio processing method provided in the embodiment of the present application is as follows: Figure 5 The steps shown are explained.

[0124] In step 201, N code streams are quantized and decoded to obtain sub-band signal characteristics corresponding to each code stream, wherein N is an integer greater than 2, and the N code streams are obtained by respectively encoding N sub-band signals obtained after the audio signal is subjected to multi-channel signal decomposition processing.

[0125] For example, by Figure 4 After the audio processing method shown encodes the code stream of the sub-band signal, the encoded code stream of the sub-band signal is transmitted to the decoding end. After the decoding end receives the code stream of the sub-band signal, it quantizes and decodes the code stream of the sub-band signal to obtain the sub-band signal characteristics corresponding to the code stream.

[0126] It should be noted that quantization decoding is the inverse process of quantization encoding. For the received bitstream, entropy decoding is first performed, and the subband signal characteristics are obtained by looking up the quantization table (i.e., inverse quantization, the quantization table is a mapping table generated by quantization during the encoding process). It should be noted that the process of decoding the received bitstream by the decoding end is the inverse process of the encoding process by the encoding end. Therefore, the value generated in the decoding process is an estimate relative to the value in the encoding process. For example, the subband signal characteristics generated in the decoding process are an estimate relative to the subband signal characteristics in the encoding process.

[0127] For example, performing quantization decoding processing on N code streams to obtain sub-band signal characteristics corresponding to each code stream includes: performing the following processing on any code stream among the N code streams: performing entropy decoding processing on the code stream to obtain an index value corresponding to the code stream; performing inverse quantization processing on the index value corresponding to the code stream to obtain a sub-band signal characteristic corresponding to the code stream.

[0128] In step 202, signal decompression processing is performed on each sub-band signal feature to obtain an estimated sub-band signal corresponding to each sub-band signal feature.

[0129] For example, signal decompression (also known as channel synthesis) is the inverse process of signal compression, and signal decompression processing is performed on sub-band signal characteristics to obtain an estimated sub-band signal (an estimated value) corresponding to each sub-band signal characteristic.

[0130] In some embodiments, signal decompression processing is performed on each subband signal feature to obtain an estimated subband signal corresponding to each subband signal feature, including: performing the following processing on any subband signal feature: calling a second neural network model corresponding to the subband signal feature; reconstructing the subband signal feature through the second neural network model to obtain an estimated subband signal corresponding to the subband signal feature; wherein the structural complexity of the second neural network model is positively correlated with the dimension of the subband signal feature.

[0131] For example, when the encoding end performs feature extraction on all sub-band signals to obtain sub-band signal features, the decoding end performs feature reconstruction processing on the sub-band signal features to obtain high-frequency sub-band signals corresponding to the high-frequency features.

[0132] As an example, when the decoding end receives 4 code streams, the 4 code streams are quantized and decoded to obtain the estimated values ​​F′ of the subband signal vectors (i.e., feature vectors) of the 4 channels. k (n), k = 1, 2, 3, 4, based on the estimated value F′ of the eigenvector k (n), k = 1, 2, 3, 4, calling a deep neural network (such as Fig.12 As shown), generate the estimated value x′ of the subband signal k (n), k = 1, 2, 3, 4, that is, estimating the sub-band signal.

[0133] like Fig.12 As shown in the figure, the flowchart of the signal compression network structure is similar to the signal decompression network structure, for example, the causal convolution and the post-processing structure in the signal decompression network structure are similar to the pre-processing in the signal compression network structure. The structure of the decoding block is symmetrical to the encoding block at the encoding end: the encoding block at the encoding end first performs dilated convolution and then performs pooling to complete downsampling, while the decoding block at the decoding end first performs pooling to complete upsampling and then performs dilated convolution.

[0134] In some embodiments, the subband signal features are reconstructed through a second neural network model to obtain an estimated subband signal corresponding to the subband signal features, including: performing the following processing on the subband signal features through the second neural network model: convolution processing on the subband signal features to obtain convolution features of the subband signal features; upsampling processing on the convolution features to obtain upsampled features of the subband signal features; pooling processing on the upsampled features to obtain pooled features of the subband signal features; convolution processing on the pooled features to obtain an estimated subband signal corresponding to the subband signal features.

[0135] like Fig.12 As shown, based on the subband signal feature F′1(n), call Fig.12 The neural network model shown (channel 1) generates a low-frequency sub-band signal x′1(n). First, the input low-frequency feature vector F′1(n) is convolved by causal convolution to obtain a 192×1 convolution feature. Then, the 192×1 convolution feature is upsampled to obtain a 24×80 upsampled feature. Next, the 24×80 upsampled feature is pooled (i.e., post-processed) to obtain a 24×160 pooled feature. Finally, the pooled feature is convolved by causal convolution to obtain a 160-dimensional sub-band signal x′1(n).

[0136] In some embodiments, upsampling processing is implemented through multiple cascaded decoding layers; upsampling processing is performed on the convolution features to obtain upsampled features of low subband signal features, including: upsampling processing is performed on the convolution features through the first decoding layer in the multiple cascaded decoding layers; the upsampling result of the first decoding layer is output to the subsequent cascaded decoding layers, and the upsampling processing and upsampling result output are continued through the subsequent cascaded decoding layers until it is output to the last decoding layer; the upsampling result output by the last decoding layer is used as the upsampled feature of the subband signal feature.

[0137] like Fig.12 As shown in the figure, three decoding blocks (i.e., decoding layers) with different upsampling factors (Up_factor) are cascaded to upsample the convolution features, i.e., the 192×1 convolution features are first upsampled by the decoding block with upsampling factor (Up_factor=8) to obtain a 96×8 upsampled result, and then the 96×8 upsampled result is upsampled by the decoding block with upsampling factor (Up_factor=5) to obtain a 48×40 upsampled result, and finally the 48×40 upsampled result is upsampled by the decoding block with upsampling factor (Up_factor=4) to obtain a 24×80 upsampled feature. Taking the decoding block (Up_factor=4) as an example, pooling can be performed based on Up_factor first, and then one or more dilated convolutions can be performed to achieve the upsampling effect.

[0138] It should be noted that after processing through a layer of decoding, the understanding of upsampling features is deepened. After learning through multiple layers of decoding, upsampling features can be gradually and accurately learned. Through cascaded decoding layers, upsampling features with progressive accuracy can be obtained.

[0139] In some embodiments, signal decompression processing is performed on each subband signal feature to obtain an estimated subband signal corresponding to each subband signal feature, including: performing feature reconstruction processing on the first k subband signal features to obtain estimated subband signals corresponding to the first k subband signal features; performing inverse processing of frequency band expansion on the last Nk subband signal features to obtain estimated subband signals corresponding to the last Nk subband signal features; wherein k is an integer and its value range is 1<k<N.

[0140] For example, when the encoding end performs feature extraction on the first k sub-band signals to obtain sub-band signal features, and performs frequency band expansion processing on the last Nk sub-band signals, the decoding end performs feature reconstruction processing on the first k sub-band signal features respectively to obtain estimated sub-band signals corresponding to the first k sub-band signal features respectively; and performs inverse processing of frequency band expansion on the last Nk sub-band signal features respectively to obtain estimated sub-band signals corresponding to the last Nk sub-band signal features respectively.

[0141] In some embodiments, the inverse processing of frequency band expansion is performed on the last Nk sub-band signal features respectively to obtain estimated sub-band signals corresponding to the last Nk sub-band signal features respectively, including: performing the following processing on any sub-band signal feature among the last Nk sub-band signal features: performing signal synthesis processing on the estimated sub-band signal associated with the sub-band signal feature in the first k estimated sub-band signals to obtain a low-frequency sub-band signal corresponding to the sub-band signal feature; performing frequency domain transform processing based on multiple sample points included in the low-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points respectively; performing spectrum replication processing on the transform coefficients of the second half of the transform coefficients corresponding to the multiple sample points respectively to obtain reference transform coefficients of the reference sub-band signal; performing gain processing on the reference transform coefficients of the reference sub-band signal based on the sub-band spectrum envelope corresponding to the sub-band signal feature to obtain the gained reference transform coefficients; performing inverse frequency domain transform processing on the gained reference transform coefficients to obtain the estimated sub-band signal corresponding to the sub-band signal feature.

[0142] For example, when the estimated subband signal associated with the subband signal feature among the first k estimated subband signals is the estimated subband signal of the next layer corresponding to the subband signal feature, the estimated subband signal associated with the subband signal feature among the first k estimated subband signals is subjected to signal synthesis processing to obtain a low-frequency subband signal corresponding to the subband signal feature, that is, when the encoding end performs multi-layer two-channel subband decomposition on the audio signal to generate a subband signal of the corresponding level, and performs signal compression on the subband signal of the corresponding level to obtain the corresponding subband signal feature, then when the estimated subband signal associated with the subband signal feature is subjected to signal synthesis processing on the estimated subband signal associated with the subband signal feature among the first k estimated subband signals, to obtain a low-frequency subband signal corresponding to the subband signal feature, so that the low-frequency subband signal and the subband signal feature are at the same level.

[0143] When the low-frequency subband signal and the subband signal characteristics are at the same level, frequency domain transform processing is performed based on the multiple sample points included in the low-frequency subband signal to obtain transform coefficients corresponding to the multiple sample points; spectrum replication processing is performed on the transform coefficients of the second half of the transform coefficients corresponding to the multiple sample points to obtain reference transform coefficients of the reference subband signal; based on the subband spectrum envelope corresponding to the subband signal characteristics, gain processing is performed on the reference transform coefficients of the reference subband signal to obtain the gained reference transform coefficients; inverse frequency domain transform processing is performed on the gained reference transform coefficients to obtain an estimated subband signal corresponding to the subband signal characteristics.

[0144] It should be noted that the frequency domain transformation method of the embodiments of the present application includes modified discrete cosine transform (MDCT), discrete cosine transform (DCT), fast Fourier transform (FFT), etc. The embodiments of the present application are not limited to the frequency domain transformation method.

[0145] In some embodiments, based on the subband spectral envelope corresponding to the subband signal characteristic, the reference transform coefficient of the reference subband signal is gain processed to obtain the reference transform coefficient after gain, including: based on the subband spectral envelope corresponding to the subband signal characteristic, the reference transform coefficient of the reference subband signal is divided into multiple subbands; for any subband among the multiple subbands, the following processing is performed: determining a first average energy corresponding to the subband in the subband spectral envelope, and determining a second average energy corresponding to the subband; determining a gain factor based on the ratio of the first average energy to the second average energy; multiplying the gain factor by each reference transform coefficient included in the subband to obtain the reference transform coefficient after gain.

[0146] As an example, for the received bitstream, entropy decoding is first performed, and the feature vectors F′ of the three channels are obtained by looking up the quantization table. k (n), k=1,2 and F′ HB (n), that is, the sub-band signal characteristics, F′ k (n), k=1,2 and F′ HB (n) is based on Figure 8C The binary tree is shown, where F′ k (n), k=1,2 is F′ HB (n) is at the next level. Based on decoding, the feature vector F′ is obtained k (n),k=1,2Ref Fig.12 The first channel and the second channel in the subband signal are obtained by 2,1 (n) and x′ 2,2(n), x′ 2,1 (n) and x′ 2,2 The dimension of (n) is 160, x′ 2,1 (n) and x′ 2,2 (n) in F′ HB (n) The next level.

[0147] Based on x′ 2,1 (n) and x′ 2,2 (n), calling a 2-channel QMF synthesis filter can generate an estimate x′ corresponding to the low-frequency subband signal of 0-8kHz LB (n), referred to as low-frequency subband signal, has a dimension of 320, and the low-frequency subband signal x′ LB (n), and the sub-band signal characteristics F′ HB (n) are at the same level. x′ LB (n) is for subsequent band expansion of 8-16kHz.

[0148] For the 8-16kHz band extension process, it is based on the 8 sub-band spectrum envelopes (i.e. F′) decoded from the bitstream. HB (n)) and the estimated value x′ of the low-frequency subband signal of 0-8kHz generated locally at the decoding end LB (n) is realized. The inverse process of the specific frequency band expansion process is as follows:

[0149] First, the low-frequency subband signal x′ generated by the decoding end LB (n) A 640-point MDCT transform similar to that at the encoding end is also performed to generate 320-point MDCT coefficients (i.e., MDCT coefficients of the low-frequency part), that is, frequency domain transform processing is performed based on multiple sample points included in the low-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points.

[0150] Then, x′ LB (n) The 320-point MDCT coefficients generated are copied to generate the MDCT coefficients of the high-frequency part (i.e., the reference transform coefficients of the reference subband signal). Referring to the basic characteristics of the speech signal, the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid simple copying, which causes the artificially generated MDCT spectrum of the high-frequency part to contain too many harmonics, the last 160 points of the 320-point MDCT coefficients that the low-frequency subband depends on can be used as the master, and the spectrum can be copied twice to generate the reference value of the 320-point MDCT coefficients of the reference subband signal (i.e., the reference transform coefficients of the reference subband signal), that is, the transform coefficients of the second half of the transform coefficients corresponding to multiple sample points are processed by spectrum copying to obtain the reference transform coefficients of the reference subband signal.

[0151] Next, call the 8 sub-band spectral envelopes obtained previously (that is, the 8 sub-band spectral envelopes obtained by querying the quantization table, that is, the sub-band spectral envelope F′ corresponding to the sub-band signal feature HB (n)), these 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands, and the reference values ​​of the MDCT coefficients of the generated 320-point reference sub-band signal are divided into 8 reference sub-bands (that is, based on the sub-band spectral envelopes corresponding to the sub-band signal characteristics, the reference transform coefficients of the reference sub-band signal are divided into multiple sub-bands), and based on a high-frequency sub-band and the corresponding reference sub-band, the reference values ​​of the MDCT coefficients of the generated 320-point reference sub-band signal are gain controlled (multiplication is performed in the frequency domain). For example, the gain factor is calculated based on the average energy of the high-frequency sub-band (that is, the first average energy) and the average energy of the corresponding reference sub-band signal (the second average energy), and the MDCT coefficient corresponding to each point in the corresponding reference sub-band signal is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to the original coefficient energy at the encoding end.

[0152] For example, assuming that the average energy of the reference subband of the reference subband signal generated by copying the past is Y_L, and the average energy of the high-frequency subband to be gain-controlled (i.e., the high-frequency subband corresponding to the subband spectrum envelope decoded based on the bitstream) is Y_H, then a gain factor a=sqrt(Y_H / Y_L) is calculated. With the gain factor a, each point in the reference subband signal generated by copying is directly multiplied by a.

[0153] Finally, the inverse MDCT transform is called to generate the estimated value x′ of the subband signal HB (n)(i.e., sub-band signal characteristic F′ HB (n) The estimated subband signal corresponding to the gain is obtained by performing an inverse MDCT transform on the 320-point MDCT coefficients after gain to generate 640-point estimated values. By overlapping, the first 320 effective estimated values ​​are taken as x′ HB (n).

[0154] It should be noted that when the first k sub-band signal features and the last Nk sub-band signal features are at the same level, the last Nk sub-band signal features can be directly band-expanded based on the estimated sub-band signals corresponding to the first k sub-band signal features.

[0155] As an example, for the received bitstream, entropy decoding is first performed, and the feature vectors F′ of the four channels are obtained by looking up the quantization table. k (n), k = 1, 2, 3, 4, that is, sub-band signal characteristics, F′ k (n), k = 1, 2, 3, 4 is based on Figure 8B The binary tree is shown, where F′ k(n), k=1,2,3,4 are at the same level. Based on decoding, the feature vector F′ is obtained k (n),k=1,2Ref Fig.12 The first and second channels in the , two sub-band signals x'1(n) and x'2(n) are obtained. Referring to the basic characteristics of the speech signal, the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid simple copying, resulting in the artificially generated high-frequency MDCT spectrum containing too many harmonics, x'2(n) can be selected to k (n), k = 3, 4 for frequency band expansion. k The inverse process of band extension with (n), k = 3, 4 is based on the 8 sub-band spectral envelopes (i.e., F′) decoded from the bitstream. k The inverse process of specific frequency band expansion is similar to the inverse process of the above-mentioned frequency band expansion.

[0156] In step 203, a signal synthesis process is performed on the multiple estimated sub-band signals to obtain synthesized audio signals corresponding to multiple code streams.

[0157] For example, the signal synthesis process is the inverse process of the signal decomposition process. The decoding end performs sub-band synthesis processing on multiple estimated sub-band signals to restore the audio signal, wherein the synthesized audio signal is the restored audio signal.

[0158] In some embodiments, signal synthesis processing is performed on multiple estimated sub-band signals to obtain synthetic audio signals corresponding to multiple code streams, including: up-sampling processing is performed on the multiple estimated sub-band signals respectively to obtain filtered signals corresponding to the multiple estimated sub-band signals respectively; and filtering synthesis processing is performed on the multiple filtered signals to obtain synthetic audio signals corresponding to the multiple code streams.

[0159] For example, after obtaining a plurality of estimated sub-band signals, sub-band synthesis is performed on the plurality of estimated sub-band signals through a QMF synthesis filter to restore an audio signal.

[0160] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0161] The embodiments of the present application can be applied to various audio scenarios, such as voice calls, instant messaging, etc. The following is an example of a voice call:

[0162] In the related art, speech coding is roughly based on the following coding principles: speech coding can directly encode speech waveform samples sample by sample; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0163] The above coding principles all come from speech signal modeling, that is, a compression method based on signal processing. In order to improve the coding efficiency while ensuring the voice quality relative to the compression method based on signal processing. The embodiment of the present application provides a speech coding method (i.e., an audio processing method) based on multi-channel signal decomposition and neural network. Based on the characteristics of the speech signal, the speech signal of a specific sampling rate is decomposed into multiple sub-band signals. For example, the multiple sub-band signals include sub-band signals with relatively low sampling rates and sub-band signals with relatively high sampling rates. Different sub-band signals can be compressed using different data compression mechanisms. For the important part (sub-band signal with relatively low sampling rate), after processing based on the neural network (NN, Neural Network) technology, a feature vector with lower dimension than the input sub-band signal can be obtained. For the relatively unimportant part (sub-band signal with relatively high sampling rate), fewer bits are used for encoding.

[0164] The embodiments of the present application can be applied to Figure 6 The voice communication link shown takes the Voice over Internet Protocol (VoIP) conference system as an example, and deploys the voice codec technology involved in the embodiment of the present application in the encoding and decoding parts to solve the basic function of voice compression. The encoder is deployed on the uplink client 601, and the decoder is deployed on the downlink client 602. The voice is collected through the uplink client, and pre-processing enhancement, encoding and other processing are performed. The encoded code stream is transmitted to the downlink client 602 through the network, and decoding, enhancement and other processing are performed by the downlink client 602 to play back the decoded voice on the downlink client 602.

[0165] Considering forward compatibility (i.e., the new encoder is compatible with the existing encoder), a transcoder needs to be deployed in the background of the system (i.e., the server) to solve the problem of interoperability between the new encoder and the existing encoder. For example, if the sender (uplink client) is the new NN encoder and the receiver (downlink client) is the public switched telephone network (PSTN, Public Switched Telephone Network) (G.722). In the background, it is necessary to execute the NN decoder to generate the voice signal, and then call the G.722 encoder to generate a specific code stream to realize the transcoding function, so that the receiver can correctly decode based on the specific code stream.

[0166] Combine the following Figure 7 The speech coding method based on multi-channel signal decomposition and neural network provided in the embodiment of the present application is described as follows:

[0167] The following processing is performed on the encoding end:

[0168] For the input speech signal x(n) of the nth frame, a multi-channel analysis filter is used to decompose it into N sub-band signals. For example, after the input signal is decomposed by multi-channel QMF, N sub-band signals x can be obtained. k (n),k=1,2,…,N.

[0169] For the kth subband signal x k (n), call the kth channel analysis to obtain the low-dimensional feature vector F k (n), eigenvector F k (The dimension of n is smaller than the subband signal x k (n) dimension to reduce the amount of data. For example, for each frame x k (n), call the dilated CNN to generate a lower-dimensional feature vector F k (n). The embodiments of the present application do not limit other NN structures, such as autoencoder (Autoencoder), fully connected (FC, Full-Connection) network, long short-term memory (LSTM, Long Short-Term Memory) network, convolutional neural network (CNN, Convolutional Neural Network) + LSTM, etc.

[0170] For sub-band signals with relatively high sampling rates, other schemes can be used to extract feature vectors for sub-band signals with relatively high sampling rates, considering that the importance of sub-band signals with high sampling rates is not as important as that of low frequencies. For example, the frequency band extension technology based on speech signal analysis can generate high-frequency sub-band signals with only 1-2 kbps bit rate.

[0171] The feature vector corresponding to the sub-band is vector quantized or scalar quantized, and the quantized index value is entropy encoded, and the encoded code stream is transmitted to the decoding end.

[0172] The decoding end performs the following processing:

[0173] Decode the code stream received by the decoding end and obtain the estimated value F′ of the feature vector of each channel k (n),k=1,2,…,N.

[0174] For channel k (i.e., F′ k (n)) Perform k-th channel synthesis to generate the estimated value x′ of the subband signal k (n).

[0175] Finally, QMF synthesis filtering is called to generate the reconstructed speech signal x′(n).

[0176] Before specifically introducing the speech coding method based on multi-channel signal decomposition and neural network provided in the embodiment of the present application, the QMF filter group, the hole convolution network and the frequency band extension technology are first introduced.

[0177] The QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into N sub-band signals with a sampling rate of Fs / N. Fig. 8A The spectral response of the low-pass part H_Low(z) (equivalent to H1(z)) and the high-pass part H_High(z) (equivalent to H2(z)) of the 2-channel QMF filter shown. Based on the relevant theoretical knowledge of the QMF analysis filter bank, the correlation between the coefficients of the above low-pass filter and high-pass filter can be easily described, as shown in formula (1):

[0178] h High (k) = -1 k h Low (k) (1)

[0179] Among them, h Low (k) represents the coefficient of low-pass filtering, h High (k) represents the coefficient of high-pass filtering.

[0180] Similarly, according to QMF related theory, the QMF synthesis filter group can be described based on the QMF analysis filter group H_Low(z) and H_High(z), as shown in formula (2).

[0181] G Low (z) = H Low (z)

[0182] G High (z)=(-1)*H High (z) (2)

[0183] Among them, G Low (z) represents the restored low-pass signal, G High (z) represents the recovered high-pass signal.

[0184] The low-pass and high-pass signals recovered by the decoding end are synthesized by the QMF synthesis filter bank, so that the reconstructed signal with the sampling rate Fs corresponding to the input signal can be restored.

[0185] In addition, based on the above 2-channel QMF solution, it can also be expanded to an N-channel QMF solution. For example, a binary number method can be used to iteratively perform 2-channel QMF analysis on the current sub-band signal to obtain a sub-band signal with lower resolution. Figure 8BIt means that after two layers of iterative 2-channel QMF analysis filter, a 4-channel sub-band signal can be obtained. Figure 8C This is another implementation method. Considering that the high-frequency signal has little impact on the quality, it does not require such high-precision analysis. Therefore, only one high-pass filter is required for the original signal. Similarly, more channels can be implemented, such as 8, 16, and 32 channels, which will not be further expanded here.

[0186] See also Fig.9A and Fig. 9B , Fig.9A is a schematic diagram of a common convolutional network provided in an embodiment of the present application, Fig. 9B Schematic diagram of a dilated convolutional network provided in an embodiment of the present application. Compared with ordinary convolutional networks, dilated convolution can increase the receptive field while keeping the size of the feature map unchanged, and can also avoid errors caused by upsampling and downsampling. Fig.9A and Fig. 9B The convolution kernel size shown in is 3×3; however, Fig.9A The receptive field of the ordinary convolution shown is only 3, while Fig. 9B The receptive field 902 of the dilated convolution shown reaches 5. That is, for a convolution kernel of size 3×3, Fig.9A The receptive field of the ordinary convolution shown is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1; Fig. 9B The dilated convolution shown has a receptive field of 5 and a dilation rate of 2.

[0187] The convolution kernel can also be used in similar Fig.9A or Fig. 9B The convolution kernel moves on the plane, which involves the concept of stride rate. For example, each time the convolution kernel shifts 1 grid, the corresponding stride rate is 1.

[0188] In addition, there is the concept of the number of convolution channels, which is the number of convolution kernel parameters used to perform convolution analysis. In theory, the more channels there are, the more comprehensive the signal analysis is and the higher the accuracy is; however, the higher the number of channels, the higher the complexity is. For example, a 1×320 tensor can use a 24-channel convolution operation, and the output is a 24×320 tensor.

[0189] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the size of the convolution kernel can be set to 1×3), the expansion rate, the shift rate and the number of channels can be defined according to actual application needs. The embodiments of the present application do not make specific limitations on this.

[0190] like Fig.10The schematic diagram of frequency band extension (or frequency band replication) shown in the figure first reconstructs the broadband signal, then copies the broadband signal to the ultra-wideband signal, and finally performs shaping based on the ultra-wideband envelope. Fig.10 The frequency domain implementation scheme shown specifically includes:

[0191] 1) Implement a core layer encoding at a low sampling rate; 2) Select the low-frequency spectrum and copy it to the high frequency; 3) Perform gain control on the copied high-frequency spectrum based on the pre-recorded boundary information (describing the energy correlation between high and low frequencies, etc.). With only a bit rate of 1-2kbps, the sampling rate can be doubled.

[0192] The following is a detailed description of the speech encoding and decoding method based on subband decomposition and neural network provided in an embodiment of the present application.

[0193] In some embodiments, a speech signal with a sampling rate of Fs=32000 Hz is taken as an example (it should be noted that the method provided in the embodiment of the present application is also applicable to scenarios with other sampling rates, including but not limited to: 8000 Hz, 32000 Hz, and 48000 Hz). At the same time, assuming that the frame length is set to 20 ms, therefore, for Fs=32000 Hz, it is equivalent to containing 640 sample points per frame.

[0194] Reference below Figure 7 The flowchart shown in the figure describes the encoding end and the decoding end in detail through two implementation modes.

[0195] The first implementation is as follows:

[0196] The process on the encoding side is as follows:

[0197] 1. Generate input signal.

[0198] For a speech signal with a sampling rate of Fs=32000 Hz, the input signal of the nth frame includes 640 sample points, which are recorded as input signal x(n).

[0199] 2. QMF signal decomposition.

[0200] Call the QMF analysis filter (2-channel QMF) to perform downsampling. Figure 8B As shown, the 4-channel decomposition is realized by QMF analysis filter, and finally 4 sub-band signals x can be obtained. k (n), n = 1, 2, 3, 4. The effective bandwidths of the sub-band signals are 0-4kHz, 4-8kHz, 8-12kHz and 12-16kHz, respectively, and the number of sample points of each frame of the sub-band signal is 160.

[0201] 3. Subband signal x k (n) Perform k-th channel analysis.

[0202] For any channel analysis, the subband signal x is analyzed by calling the deep network (i.e. neural network) k (n), generating a lower dimensional feature vector F k (n). In the embodiment of the present application, for a 4-channel QMF filter, x k The dimension of (n) is 160. The eigenvectors of the output sub-band signals can be set according to the channels to which they belong. From the perspective of data volume, channel analysis plays a role of "dimensionality reduction" and realizes the function of data compression.

[0203] See also Fig.11 The network structure diagram of channel analysis is shown in the figure. The following takes channel 1 as an example to explain the process of channel analysis in detail:

[0204] First, a 24-channel causal convolution is called to expand the input tensor (i.e., vector) to a 24×320 tensor.

[0205] Then, the 24×320 tensor is preprocessed. For example, a pooling operation with a factor of 2 is performed on the 24×320 tensor, and the activation function may be ReLU to generate a 24×160 tensor.

[0206] Next, cascade three coding blocks with different downsampling factors (Down_factor). Taking the coding block (Down_factor=4) as an example, one or more dilated convolutions can be performed first, and the size of each convolution kernel is fixed to 1×3 and the shift rate (Stride Rate) is 1. In addition, the dilation rate (Dilation Rate) of one or more dilated convolutions can be set according to demand, such as 3. Of course, the embodiment of the present application does not limit the dilation rate of different dilated convolution settings. Then, the Down_factor of the three coding blocks is set to 4, 5, and 8, respectively, which is equivalent to setting pooling factors of different sizes, which plays a role in downsampling. Finally, the number of channels of the three coding blocks is set to 48, 96, and 192, respectively. Therefore, after three coding blocks, the 24×160 tensor is converted into 48×40, 96×8, and 192×1 tensors respectively.

[0207] Finally, after similar preprocessing of the 192×1 tensor, a 32-dimensional feature vector can be output.

[0208] like Fig.11 As shown in the figure, the network results of the channels corresponding to higher frequencies are relatively simpler. This is reflected in the number of channels in the coding block and the dimension of the output feature vector. This is mainly because the high frequency part does not affect the quality relatively, and there is no need to extract the feature vector of the sub-band signal with high precision and high dimension as in the first channel.

[0209] like Fig.11 As shown in the figure, after 4-channel analysis, the feature vectors {F1(n), F2(n), F3(n), F4(n)} of the subband signals corresponding to the 4 channels are obtained, and the dimensions are 32, 16, 16 and 8 respectively. It can be seen that the original input signal is 640-dimensional, and the dimension of all the output feature vectors is only 72-dimensional, which significantly reduces the amount of data.

[0210] 4. Quantization coding.

[0211] For the feature vectors of the four channels, scalar quantization (each component is quantized separately) and entropy coding methods can be performed. In addition, the embodiment of the present application does not limit the technical combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy coding.

[0212] After quantizing and encoding the feature vector, the corresponding bit stream can be generated. According to experiments, high-quality compression of 32kHz ultra-wideband signals can be achieved at a bit rate of 6-10kbps.

[0213] The process of decoding is as follows:

[0214] 1. Quantization decoding.

[0215] Quantization decoding is the inverse process of quantization encoding. For the received bit stream, entropy decoding is first performed, and the estimated value F′ of the feature vector of the four channels is obtained by looking up the quantization table. k (n),k=1,2,3,4.

[0216] 2. Estimation of the eigenvector F′ k (n) Perform k-th channel synthesis.

[0217] The purpose of the k-th channel synthesis is based on the estimated value F′ of the feature vector k (n), k = 1, 2, 3, 4, calling a deep neural network (such as Fig.12 As shown), generate the estimated value x′ of the subband signal k (n),k=1,2,3,4.

[0218] like Fig.12 As shown in the figure, the flowchart of the network structure of channel synthesis is similar to that of the network structure of the analysis network. For example, the post-processing structure in the network structure of causal convolution and channel synthesis is similar to the pre-processing in the network structure of the analysis network. The structure of the decoding block is symmetrical to the encoding block at the encoding end: the encoding block at the encoding end first performs dilated convolution and then performs pooling to complete downsampling, while the decoding block at the decoding end first performs pooling to complete upsampling and then performs dilated convolution.

[0219] 3. Synthesis Filter

[0220] At the decoding end, the estimated value x′ of the subband signal of the four channels is obtained k (n), k = 1, 2, 3, 4, only need to call 4-channel QMF synthesis filter (such as Figure 8B As shown), a reconstructed signal x′(n) of 640 points can be generated.

[0221] The second implementation is as follows:

[0222] It should be noted that the second implementation is mainly to simplify the compression process of the two high-frequency related sub-band signals. As mentioned above, the third and fourth channels of the first implementation correspond to 8-16kHz (8-12kHz and 12-16kHz), and contain 24-dimensional feature vectors (16-dimensional feature vectors and 8-dimensional feature vectors). According to the basic characteristics of the speech signal, 8-16kHz can completely use more streamlined frequency band extension and other technologies, and the encoding end extracts feature vectors with fewer dimensions. It saves bits and complexity. Below, the encoding end and decoding end of the second implementation are described in detail.

[0223] The process on the encoding side is as follows:

[0224] 1. Generate input signal.

[0225] For a speech signal with a sampling rate of Fs=32000 Hz, the input signal of the nth frame includes 640 sample points, which are recorded as input signal x(n).

[0226] 2. QMF signal decomposition.

[0227] Call the QMF analysis filter (2-channel QMF) to perform downsampling. Figure 8C As shown, the 3-channel decomposition is realized by QMF analysis filter, and finally 3 sub-band signals x can be obtained HB (n), x 2,1 (n) and x 2,2 (n).

[0228] like Figure 8C As shown, x 2,1 (n) and x 2,2 (n) are the 0-4kHz and 4-8kHz spectra, respectively, which are generated by two iterations of 2-channel QMF analysis filtering, equivalent to x1(n) and x2(n) in the first implementation. 2,1 (n) and x 2,2 (n) contain 160 sample points respectively.

[0229] like Figure 8CAs shown, it corresponds to the 8-16kHz frequency band, and no detailed analysis is required. Therefore, only one QMF high-pass filter is required to generate a high-frequency sub-band signal x for the original 32kHz sampled input signal. HB (n), each frame contains 320 sample points.

[0230] 3. Perform k-th channel analysis on the sub-band signal.

[0231] According to the equivalence introduced above, x 2,1 (n) and x 2,2 (n) The analysis of the two corresponding sub-band signals can refer to the first two channels in the first implementation method ( Fig.11 As a result, the feature vectors F1(n) and F2(n) of the corresponding sub-band signals are generated, with dimensions of 32 and 16 respectively.

[0232] For the subband signal related to 8-16kHz (including 320 sample points / frame), a frequency band extension method is used (recovering a wideband speech signal from a band-limited narrowband speech signal). The application of frequency band extension in the embodiment of the present application is specifically described below:

[0233] For the high frequency subband signal x including 320 points HB (n), call the modified discrete cosine transform (MDCT) to generate 320-point MDCT coefficients. Specifically, if there is a 50% overlap, the high-frequency data of the n+1th frame can be merged (joined) with the high-frequency data of the nth frame, and the 640-point MDCT is calculated to obtain 320-point MDCT coefficients.

[0234] The MDCT coefficients of 320 points are divided into N sub-bands, where the sub-bands are groups of adjacent multiple MDCT coefficients, and the MDCT coefficients of 320 points can be divided into 8 sub-bands. For example, the 320 points can be evenly distributed, that is, the number of points included in each sub-band is consistent. Of course, the embodiment of the present application cannot divide the 320 points non-uniformly, such as the MDCT coefficients included in the sub-bands with low frequencies are less (the frequency resolution is higher), and the MDCT coefficients included in the sub-bands with high frequencies are more (the frequency resolution is lower).

[0235] According to Nyquist sampling theorem (to restore the original signal from the sampled signal without distortion, the sampling frequency should be greater than 2 times the highest frequency of the original signal. When the sampling frequency is less than 2 times the highest frequency of the spectrum, the spectrum of the signal is aliased. When the sampling frequency is greater than 2 times the highest frequency of the spectrum, the spectrum of the signal is not aliased), the 320-point MDCT coefficients above represent the spectrum of 8-16kHz. However, ultra-wideband voice communication does not necessarily require the spectrum to be 16kHz. For example, if the spectrum is set to 14kHz, only the MDCT coefficients of the first 240 points need to be considered. Correspondingly, the number of subbands can be controlled to 6.

[0236] For each subband, the average energy of all MDCT coefficients in the current subband is calculated as the subband spectrum envelope (the spectrum envelope is a smooth curve passing through the main peak points of the spectrum). For example, the MDCT coefficients included in the current subband are x(n), n = 1, 2, ..., 40, then the average energy Y = ((x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40). When the 320-point MDCT coefficients are divided into 8 sub-bands, 8 sub-band spectrum envelopes can be obtained. These 8 sub-band spectrum envelopes are the generated high-frequency sub-band signals x HB The eigenvector F of (n) HB (n).

[0237] In summary, using any of the above two methods (NN structure and frequency band expansion), the 320-dimensional subband signal can be output as an 8-dimensional feature vector. Therefore, only a small amount of data is needed to represent high-frequency information, and the coding efficiency is significantly improved.

[0238] 4. Quantization coding.

[0239] For the feature vectors of the above three sub-band signals (32, 16, and 8 dimensions respectively), scalar quantization (each component is quantized separately) and entropy coding methods can be performed. In addition, the embodiment of the present application does not limit the technical combination of vector quantization (combining multiple adjacent components into a vector for joint quantization) and entropy coding.

[0240] After quantizing and encoding the feature vector, the corresponding bit stream can be generated. According to experiments, high-quality compression of 32kHz ultra-wideband signals can be achieved at a bit rate of 6-10kbps.

[0241] The process of decoding is as follows:

[0242] 1. Quantization decoding.

[0243] Quantization decoding is the inverse process of quantization encoding. For the received bit stream, entropy decoding is first performed, and the estimated value F′ of the feature vector of the three channels is obtained by looking up the quantization table.k (n), k=1,2 and F′ HB (n).

[0244] 2. Perform channel synthesis on the estimated value of the feature vector.

[0245] For two channels with 0-8kHz correlation, the estimated value of the feature vector F′ can be obtained based on decoding k (n), k = 1, 2 Refer to the relevant steps in the first implementation method (see Fig.12 The first channel and the second channel in the subband are obtained to obtain the estimated values ​​x′ of the two subband signals. 2,1 (n) and x′ 2,2 (n), x′ 2,1 (n) and x′ 2,2 The dimension of (n) is 160.

[0246] In addition, based on x′ 2,1 (n) and x′ 2,2 (n), calling a 2-channel QMF synthesis filter can generate an estimate x′ corresponding to the 0-8kHz subband signal LB (n), the dimension is 320. x′ LB (n) is for subsequent band expansion of 8-16kHz.

[0247] like Fig.12 As shown in the figure, the flowchart of the network structure of channel synthesis is similar to that of the network structure of the analysis network. For example, the post-processing structure in the network structure of causal convolution and channel synthesis is similar to the pre-processing in the network structure of the analysis network. The structure of the decoding block is symmetrical to the encoding block at the encoding end: the encoding block at the encoding end first performs dilated convolution and then performs pooling to complete downsampling, while the decoding block at the decoding end first performs pooling to complete upsampling and then performs dilated convolution.

[0248] For the 8-16kHz channel synthesis process, it is based on the 8 sub-band spectrum envelopes (i.e. F′) decoded from the bitstream. HB (n)) and the estimated value x′ of the 0-8kHz subband signal generated locally at the decoding end LB (n) is realized. The specific channel synthesis process is as follows:

[0249] First, the estimated value x′ of the low-frequency subband signal generated by the decoding end LB (n), a 640-point MDCT transform similar to that at the encoding end is also performed to generate 320-point MDCT coefficients (i.e., MDCT coefficients of the low-frequency part).

[0250] Then, x′ LB(n) The 320-point MDCT coefficients generated are copied to generate the MDCT coefficients of the high-frequency part. Referring to the basic characteristics of the speech signal, the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid simple copying, which causes the artificially generated MDCT spectrum of the high-frequency part to contain too many harmonics, the last 160 points of the 320-point MDCT coefficients that the low-frequency subband depends on can be used as the master, and the spectrum can be copied twice to generate the reference value of the 320-point MDCT coefficients of the high-frequency subband signal.

[0251] Next, the 8 sub-band spectral envelopes obtained previously (i.e., the 8 sub-band spectral envelopes obtained by querying the quantization table) are called. These 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands, and the reference values ​​of the MDCT coefficients of the generated 320-point high-frequency sub-band signals are divided into 8 reference high-frequency sub-bands. In a band-by-band manner, based on a high-frequency sub-band and the corresponding reference high-frequency sub-band, the reference values ​​of the MDCT coefficients of the generated 320-point high-frequency sub-band signals are gain controlled (multiplication in the frequency domain). For example, the gain factor is calculated based on the average energy of the high-frequency sub-band and the average energy of the corresponding reference high-frequency sub-band, and the MDCT coefficient corresponding to each point in the corresponding reference high-frequency sub-band is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to the original coefficient energy at the encoding end.

[0252] For example, assuming that the average energy of the high-frequency sub-band of the high-frequency sub-band signal generated by copying the past is Y_L, and the average energy of the high-frequency sub-band to be gain-controlled (i.e., the high-frequency sub-band corresponding to the sub-band spectrum envelope decoded based on the bitstream) is Y_H, then a gain factor a=sqrt(Y_H / Y_L) is calculated. After the gain factor a is obtained, each point in the high-frequency sub-band generated by copying is directly multiplied by a.

[0253] Finally, the inverse MDCT transform is called to generate the estimated value x′ of the high-frequency subband signal HB (n) Perform an inverse MDCT transform on the 320-point MDCT coefficients after gain to generate 640-point estimated values. By overlapping, take the first 320 effective estimated values ​​as x′ HB (n).

[0254] 3. Synthesis Filter

[0255] At the decoding end, the estimated value x′ of the low-frequency subband signal is obtained LB (n) and the estimated value x′ of the high frequency subband signal HB After that, it only needs to upsample and call the 2-channel QMF synthesis filter once to generate the 640-point reconstructed signal x′(n).

[0256] The embodiment of the present application can collect data and jointly train the relevant networks of the encoding end and the decoding end to obtain the optimal parameters. The user only needs to prepare the data and set the corresponding network structure. After the training is completed in the background, the trained model can be put into use.

[0257] In summary, the speech coding method based on multi-channel signal decomposition and neural network provided in the embodiment of the present application significantly improves the coding efficiency compared with the signal processing scheme while ensuring audio quality and acceptable complexity through the organic combination of signal decomposition, signal processing technology and deep neural network.

[0258] So far, the audio processing method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the terminal device provided by the embodiment of the present application. The embodiment of the present application also provides an audio processing device. In actual applications, each functional module in the audio processing device can be implemented by hardware resources of an electronic device (such as a terminal device, a server or a server cluster), such as computing resources such as a processor, communication resources (such as for supporting various communication methods such as optical cables and cellular), and memory. Figure 3 An audio processing device 555 stored in a memory 550 is shown, which may be software in the form of programs and plug-ins, for example, software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules in large software systems, application programming interfaces, plug-ins, cloud services, and other implementation methods. Different implementation methods are described below with examples.

[0259] The audio processing device 555 includes a series of modules, including a decomposition module 5551, a compression module 5552, and an encoding module 5553. The following further describes the solution of implementing audio encoding by cooperating among the modules in the audio processing device 555 provided in the embodiment of the present application.

[0260] A decomposition module is used to perform multi-channel signal decomposition processing on the audio signal to obtain N sub-band signals of the audio signal, wherein N is an integer greater than 2, and the frequency bands of the N sub-band signals are increased in sequence; a compression module is used to perform signal compression processing on each of the sub-band signals to obtain sub-band signal characteristics of each of the sub-band signals; and an encoding module is used to perform quantization encoding processing on the sub-band signal characteristics of each of the sub-band signals to obtain a code stream of each of the sub-band signals.

[0261] In some embodiments, the multi-channel signal decomposition processing is implemented by multi-layer two-channel sub-band decomposition; the decomposition module is also used to perform the two-channel sub-band decomposition processing of the first layer on the audio signal to obtain the low-frequency sub-band signal of the first layer and the high-frequency sub-band signal of the first layer; perform the two-channel sub-band decomposition processing of the i+1 layer on the sub-band signal of the i-th layer to obtain the low-frequency sub-band signal of the i+1 layer and the high-frequency sub-band signal of the i+1 layer; wherein the sub-band signal of the i-th layer is the low-frequency sub-band signal of the i-th layer, or the high-frequency sub-band signal of the i-th layer and the low-frequency sub-band signal of the i-th layer, and i is an increasing natural number and has a value range of 1≤i<N; the sub-band signal of the last layer and the high-frequency sub-band signal of each layer that has not been subjected to the two-channel sub-band decomposition processing are used as the sub-band signal of the audio signal.

[0262] In some embodiments, the decomposition module is also used to perform sampling processing on the audio signal to obtain a sampled signal, wherein the sampled signal includes multiple sample points obtained by sampling; perform low-pass filtering processing of the first layer on the sampled signal to obtain the low-pass filtered signal of the first layer; down-sample the low-pass filtered signal of the first layer to obtain the low-frequency sub-band signal of the first layer; perform high-pass filtering processing on the sampled signal of the first layer to obtain the high-pass filtered signal of the first layer; down-sample the high-pass filtered signal of the first layer to obtain the high-frequency sub-band signal of the first layer.

[0263] In some embodiments, the compression module is also used to perform the following processing on any of the sub-band signals: calling the first neural network model corresponding to the sub-band signal; performing feature extraction processing on the sub-band signal through the first neural network model to obtain the sub-band signal characteristics of the sub-band signal; wherein the structural complexity of the first neural network model is positively correlated with the dimension of the sub-band signal characteristics of the sub-band signal.

[0264] In some embodiments, the compression module is also used to perform the following processing on the subband signal through the first neural network model: performing convolution processing on the subband signal to obtain the convolution feature of the subband signal; performing pooling processing on the convolution feature to obtain the pooling feature of the subband signal; performing downsampling processing on the pooling feature to obtain the downsampling feature of the subband signal; performing convolution processing on the downsampling feature to obtain the subband signal feature of the subband signal.

[0265] In some embodiments, the compression module is also used to perform feature extraction processing on the first k sub-band signals respectively to obtain sub-band signal features corresponding to the first k sub-band signals respectively; perform frequency band expansion processing on the last Nk sub-band signals respectively to obtain sub-band signal features corresponding to the last Nk sub-band signals respectively; wherein k is an integer and its value range is 1<k<N.

[0266] In some embodiments, the compression module is also used to perform the following processing on any of the sub-band signals among the last Nk sub-band signals: perform frequency domain transform processing based on multiple sample points included in the sub-band signal to obtain transform coefficients corresponding to the multiple sample points respectively; divide the transform coefficients corresponding to the multiple sample points respectively into multiple sub-bands; perform mean processing on the transform coefficients included in each of the sub-bands to obtain the average energy corresponding to each of the sub-bands, and use the average energy as the sub-band spectral envelope corresponding to each of the sub-bands; determine the sub-band spectral envelopes corresponding to the multiple sub-bands as the sub-band signal characteristics corresponding to the sub-band signal.

[0267] In some embodiments, the compression module is further used to obtain a reference subband signal of a reference audio signal, wherein the reference audio signal is an audio signal adjacent to the audio signal, and the reference subband signal has the same frequency band as the subband signal; based on a plurality of sample points included in the reference subband signal and a plurality of sample points included in the subband signal, discrete cosine transform processing is performed on the plurality of sample points included in the subband signal to obtain transform coefficients corresponding to the plurality of sample points included in the subband signal.

[0268] In some embodiments, the encoding module is further used to perform quantization processing on the subband signal characteristics of each of the subband signals to obtain an index value of the subband signal characteristics; and perform entropy encoding processing on the index value of the subband signal characteristics to obtain a subcode stream of the subband signal.

[0269] The audio processing device 555 includes a series of modules, including a decoding module 5554, a decompression module 5555, and a synthesis module 5556. The following further describes the solution of implementing audio decoding by cooperating among the modules in the audio processing device 555 provided in the embodiment of the present application.

[0270] A decoding module is used to perform quantization decoding processing on N code streams to obtain sub-band signal characteristics corresponding to each of the code streams; wherein N is an integer greater than 2, and the N code streams are obtained by respectively encoding N sub-band signals obtained after the audio signal is subjected to multi-channel signal decomposition processing; a decompression module is used to perform signal decompression processing on each of the sub-band signal characteristics to obtain an estimated sub-band signal corresponding to each of the sub-band signal characteristics; a synthesis module is used to perform signal synthesis processing on multiple estimated sub-band signals to obtain synthesized audio signals corresponding to the multiple code streams.

[0271] In some embodiments, the decompression module is also used to perform the following processing for any of the sub-band signal features: calling a second neural network model corresponding to the sub-band signal feature; reconstructing the sub-band signal feature through the second neural network model to obtain an estimated sub-band signal corresponding to the sub-band signal feature; wherein the structural complexity of the second neural network model is positively correlated with the dimension of the sub-band signal feature.

[0272] In some embodiments, the decompression module is also used to perform the following processing on the subband signal features through the second neural network model: performing convolution processing on the subband signal features to obtain convolution features of the subband signal features; performing upsampling processing on the convolution features to obtain upsampled features of the subband signal features; performing pooling processing on the upsampled features to obtain pooled features of the subband signal features; performing convolution processing on the pooled features to obtain an estimated subband signal corresponding to the subband signal features.

[0273] In some embodiments, the decompression module is also used to perform feature reconstruction processing on the first k sub-band signal features, respectively, to obtain estimated sub-band signals corresponding to the first k sub-band signal features; perform inverse processing of frequency band expansion on the last Nk sub-band signal features, respectively, to obtain estimated sub-band signals corresponding to the last Nk sub-band signal features; wherein k is an integer and its value range is 1<k<N.

[0274] In some embodiments, the decompression module is also used to perform the following processing on any of the subband signal features among the last Nk subband signal features: perform signal synthesis processing on the estimated subband signal associated with the subband signal feature in the first k estimated subband signals to obtain a low-frequency subband signal corresponding to the subband signal feature; perform frequency domain transform processing based on multiple sample points included in the low-frequency subband signal to obtain transform coefficients corresponding to the multiple sample points respectively; perform spectrum replication processing on the transform coefficients of the second half of the transform coefficients corresponding to the multiple sample points respectively to obtain reference transform coefficients of the reference subband signal; perform gain processing on the reference transform coefficients of the reference subband signal based on the subband spectral envelope corresponding to the subband signal feature to obtain the reference transform coefficients after gain; perform inverse frequency domain transform processing on the reference transform coefficients after gain to obtain the estimated subband signal corresponding to the subband signal feature.

[0275] In some embodiments, the decompression module is also used to divide the reference transform coefficient of the reference subband signal into multiple subbands based on the subband spectral envelope corresponding to the subband signal characteristics; perform the following processing for any of the multiple subbands: determine the first average energy corresponding to the subband in the subband spectral envelope, and determine the second average energy corresponding to the subband; determine a gain factor based on the ratio of the first average energy to the second average energy; multiply the gain factor by each of the reference transform coefficients included in the subband to obtain the reference transform coefficient after gain.

[0276] In some embodiments, the decoding module is further used to perform the following processing on any of the N code streams: perform entropy decoding on the code stream to obtain an index value corresponding to the code stream; perform inverse quantization on the index value corresponding to the code stream to obtain a sub-band signal feature corresponding to the code stream.

[0277] In some embodiments, the synthesis module is further used to perform upsampling processing on the multiple estimated sub-band signals respectively to obtain filtered signals corresponding to the multiple estimated sub-band signals respectively; and perform filtering synthesis processing on the multiple filtered signals to obtain synthesized audio signals corresponding to the multiple code streams.

[0278] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the audio processing method described above in the embodiment of the present application.

[0279] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the audio processing method provided by the present application embodiment, for example, Figure 4-5 The audio processing method is shown.

[0280] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0281] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0282] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0283] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0284] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0285] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: Acquire N subband signals of the audio signal, where N is an integer greater than 2; Performing feature extraction processing on the first k sub-band signals respectively through a neural network model to obtain sub-band signal features corresponding to the first k sub-band signals respectively; Performing frequency band extension processing on the last Nk sub-band signals respectively to obtain sub-band signal features corresponding to the last Nk sub-band signals respectively, wherein k is an integer and has a value range of 1≤k<N; The sub-band signal characteristics of each of the sub-band signals are quantized and encoded to obtain a code stream of each of the sub-band signals.

2. The method according to claim 1, characterized in that The performing frequency band extension processing on the last Nk sub-band signals respectively to obtain sub-band signal features corresponding to the last Nk sub-band signals respectively includes: The following processing is performed on each of the sub-band signals in the last Nk sub-band signals: Performing frequency domain transform processing based on multiple sample points included in the subband signal to obtain transform coefficients corresponding to the multiple sample points respectively; Dividing the transform coefficients corresponding to the plurality of sample points into a plurality of sub-bands; Performing mean processing on the transform coefficients included in each of the sub-bands to obtain average energy corresponding to each of the sub-bands, and using the average energy as the sub-band spectrum envelope corresponding to each of the sub-bands; Subband spectrum envelopes respectively corresponding to the multiple subbands are determined as subband signal features corresponding to the subband signals.

3. The method according to claim 2, characterized in that The performing frequency domain transform processing based on the multiple sample points included in the subband signal to obtain transform coefficients respectively corresponding to the multiple sample points includes: Acquire a reference subband signal of a reference audio signal, wherein the reference audio signal is an audio signal adjacent to the audio signal, and the reference subband signal has the same frequency band as the subband signal; Based on the multiple sample points included in the reference subband signal and the multiple sample points included in the subband signal, discrete cosine transform processing is performed on the multiple sample points included in the subband signal to obtain transform coefficients corresponding to the multiple sample points included in the subband signal.

4. The method according to claim 1, characterized in that: The performing feature extraction processing on the first k sub-band signals respectively through the neural network model to obtain sub-band signal features corresponding to the first k sub-band signals respectively includes: The following processing is performed on each of the first k sub-band signals by the neural network model: Performing convolution processing on the sub-band signal to obtain convolution features of the sub-band signal; Performing pooling processing on the convolutional features to obtain pooled features of the sub-band signal; Downsampling the pooled features to obtain downsampled features of the subband signal; The down-sampled features are convolved to obtain sub-band signal features of the sub-band signal.

5. The method according to claim 1, characterized in that The feature dimension of the subband signal feature of each of the subband signals is not positively correlated with the frequency band of each of the subband signals, and the feature dimension of the subband signal feature of the Nth subband signal is smaller than the feature dimension of the subband signal feature of the first subband signal.

6. The method according to claim 1, characterized in that Before obtaining N sub-band signals of the audio signal, the method further includes: Perform multi-channel signal decomposition processing on the audio signal to obtain N sub-band signals of the audio signal.

7. An audio processing method, characterized in that: The method comprises: Acquire N subband signals of the audio signal, where N is an integer greater than 2; Performing feature extraction processing on each of the sub-band signals through a first neural network model corresponding to each of the sub-band signals to obtain a sub-band signal feature of each of the sub-band signals; The sub-band signal characteristics of each of the sub-band signals are quantized and encoded to obtain a code stream of each of the sub-band signals.

8. The method according to claim 7, characterized in that The structural complexity of the first neural network model is positively correlated with the dimension of the sub-band signal feature of the sub-band signal.

9. The method according to claim 7, characterized in that: The step of performing feature extraction processing on each of the sub-band signals by using the first neural network model corresponding to each of the sub-band signals to obtain sub-band signal features of each of the sub-band signals comprises: The following processing is performed on each of the sub-band signals by using the first neural network model corresponding to each of the sub-band signals: Performing convolution processing on the sub-band signal to obtain convolution features of the sub-band signal; Performing pooling processing on the convolutional features to obtain pooled features of the sub-band signal; Downsampling the pooled features to obtain downsampled features of the subband signal; The down-sampled features are convolved to obtain sub-band signal features of the sub-band signal.

10. An audio processing method, characterized in that: The method comprises: Performing quantization decoding processing on N code streams to obtain sub-band signal characteristics corresponding to each of the code streams; Wherein, N is an integer greater than 2, and the N code streams are obtained by respectively encoding N sub-band signals of the audio signal; Perform feature reconstruction processing on the first k sub-band signal features respectively through a neural network model to obtain estimated sub-band signals corresponding to the first k sub-band signal features respectively; Performing inverse processing of frequency band expansion on the last Nk sub-band signal features respectively to obtain estimated sub-band signals corresponding to the last Nk sub-band signal features respectively, wherein k is an integer and has a value range of 1≤k<N; Based on the multiple estimated sub-band signals, synthetic audio signals corresponding to the multiple code streams are determined.

11. The method according to claim 10, characterized in that The performing inverse processing of the frequency band expansion on the features of the last Nk sub-band signals to obtain estimated sub-band signals corresponding to the features of the last Nk sub-band signals respectively includes: The following processing is performed for each of the sub-band signal features of the last Nk sub-band signal features: Performing signal synthesis processing on the estimated subband signals associated with the subband signal characteristics among the first k estimated subband signals to obtain a low-frequency subband signal corresponding to the subband signal characteristics; Performing frequency domain transform processing based on multiple sample points included in the low-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points respectively; Performing spectrum replication processing on the transform coefficients of the second half of the transform coefficients corresponding to the plurality of sample points, respectively, to obtain reference transform coefficients of a reference subband signal; Based on the subband spectrum envelope corresponding to the subband signal feature, performing gain processing on the reference transform coefficient of the reference subband signal to obtain the reference transform coefficient after gain; An inverse frequency domain transform process is performed on the reference transform coefficient after gain to obtain an estimated sub-band signal corresponding to the sub-band signal feature.

12. The method according to claim 11, characterized in that The step of performing gain processing on the reference transform coefficient of the reference subband signal based on the subband spectrum envelope corresponding to the subband signal feature to obtain the reference transform coefficient after gain processing includes: Based on the subband spectral envelope corresponding to the subband signal feature, dividing the reference transform coefficient of the reference subband signal into a plurality of subbands; The following processing is performed for each of the plurality of sub-bands: Determining a first average energy corresponding to the sub-band in the sub-band spectral envelope, and determining a second average energy corresponding to the sub-band; determining a gain factor based on a ratio of the first average energy to the second average energy; The gain factor is multiplied by each of the reference transform coefficients included in the subband to obtain the reference transform coefficient after gain.

13. The method according to claim 10, characterized in that The process of performing feature reconstruction processing on the first k sub-band signal features respectively through the neural network model to obtain estimated sub-band signals respectively corresponding to the first k sub-band signal features comprises: The following processing is performed on each of the first k sub-band signal features by the neural network model: Performing convolution processing on the sub-band signal feature to obtain a convolution feature of the sub-band signal feature; Performing up-sampling processing on the convolution feature to obtain up-sampled features of the sub-band signal feature; Performing a reverse pooling process on the up-sampled features to obtain reverse pooling features of the sub-band signal features; The de-pooled features are convolved to obtain estimated sub-band signals corresponding to the sub-band signal features.

14. The method according to claim 10, characterized in that The step of determining the synthesized audio signals corresponding to the plurality of code streams based on the plurality of estimated sub-band signals comprises: Performing up-sampling processing on the plurality of estimated sub-band signals respectively to obtain filtered signals respectively corresponding to the plurality of estimated sub-band signals; Perform filtering synthesis processing on the multiple filtered signals to obtain synthesized audio signals corresponding to the multiple code streams.

15. An audio processing method, characterized in that: The method comprises: Performing quantization decoding processing on N code streams to obtain sub-band signal characteristics corresponding to each of the code streams; Wherein, N is an integer greater than 2, and the N code streams are obtained by respectively encoding N sub-band signals of the audio signal; Reconstructing each of the sub-band signal features through a second neural network model corresponding to each of the sub-band signal features to obtain an estimated sub-band signal corresponding to each of the sub-band signal features; Based on the multiple estimated sub-band signals, synthetic audio signals corresponding to the multiple code streams are determined.

16. The method according to claim 15, characterized in that The structural complexity of the second neural network model is positively correlated with the dimension of the sub-band signal features.

17. The method according to claim 15, characterized in that The step of reconstructing each of the sub-band signal features by using the second neural network model corresponding to each of the sub-band signal features to obtain an estimated sub-band signal corresponding to each of the sub-band signal features comprises: The following processing is performed on each of the sub-band signal features by using the second neural network model corresponding to each of the sub-band signal features: Performing convolution processing on the sub-band signal feature to obtain a convolution feature of the sub-band signal feature; Performing up-sampling processing on the convolution feature to obtain up-sampled features of the sub-band signal feature; Performing a reverse pooling process on the up-sampled features to obtain reverse pooling features of the sub-band signal features; The de-pooled features are convolved to obtain estimated sub-band signals corresponding to the sub-band signal features.

18. An audio processing device, characterized in that: The device comprises: A decomposition module, used to obtain N sub-band signals of the audio signal, wherein N is an integer greater than 2; A compression module, used to perform feature extraction processing on the first k sub-band signals respectively through a neural network model to obtain sub-band signal features corresponding to the first k sub-band signals respectively; Performing frequency band extension processing on the last Nk sub-band signals respectively to obtain sub-band signal features corresponding to the last Nk sub-band signals respectively, wherein k is an integer and has a value range of 1≤k<N; The encoding module is used to perform quantization encoding processing on the sub-band signal characteristics of each of the sub-band signals to obtain a code stream of each of the sub-band signals.

19. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the audio processing method described in any one of claims 1 to 17 when executed by a processor.

20. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the audio processing method according to any one of claims 1 to 17 is implemented.