Audio encoding method, audio decoding method, apparatus, readable storage medium

Abstract

This application provides an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The audio decoding method includes: in response to a decoding request for an audio stream encapsulation, obtaining a target encoding mode and a target bitrate mode from the frame header included in the audio stream encapsulation; wherein the audio stream included in the audio stream encapsulation is obtained by audio encoding an audio signal using the target encoding mode and the target bitrate mode, the target encoding mode being obtained from multiple encoding modes, and the target bitrate mode being obtained from multiple bitrate modes; performing signal decoding processing on the audio stream using the target encoding mode and the target bitrate mode to obtain an estimated value of the encoding features corresponding to the audio stream; and performing reconstruction processing on the estimated value of the encoding features corresponding to the audio stream using the target encoding mode to obtain a synthesized audio signal corresponding to the audio stream.

Description

[0001] This application is a divisional application of patent application No. 202311614893.4, filed on November 29, 2023, entitled "Audio Encoding Method, Audio Decoding Method, Apparatus, and Readable Storage Medium". Technical Field

[0002] This application relates to artificial intelligence technology, and more particularly to an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0003] Artificial Intelligence (AI) is a comprehensive technology within computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a multidisciplinary field encompassing a wide range of areas, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in more fields and play an increasingly important role.

[0004] Audio encoding and decoding technology is a crucial application in the field of artificial intelligence, and a core technology in communication services, including remote audio and video calls. Simply put, speech coding technology aims to transmit as much speech information as possible with minimal network bandwidth. From the perspective of Shannon's information theory, speech coding is a type of source coding. The goal of source coding is to compress the amount of data we want to transmit as much as possible at the encoding end, removing redundancy, while simultaneously achieving lossless (or near-lossless) reconstruction at the decoding end.

[0005] In related technologies, the audio quality generated by the decoding end is relatively uniform and cannot meet user needs. Summary of the Invention

[0006] This application provides an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can output synthesized audio signals of different quality levels while ensuring the efficiency of audio decoding.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides an audio encoding method, including:

[0009] In response to an encoding request for an audio signal, a target encoding mode for the audio signal is obtained from multiple encoding modes, and a target bitrate mode for the audio signal is obtained from multiple bitrate modes;

[0010] The encoding features of the audio signal are extracted from the audio signal using the target encoding pattern.

[0011] By using the target bitrate mode, the encoding features of the audio signal are processed by signal encoding to obtain the audio bitstream of the audio signal;

[0012] The frame header is determined based on the target encoding mode and the target bit rate mode;

[0013] Based on the audio bitstream and the frame header, an audio bitstream encapsulation of the audio signal is generated.

[0014] This application provides an audio decoding method, including:

[0015] In response to a decoding request for an audio stream encapsulation, a target encoding mode and a target bitrate mode are obtained from the frame header included in the audio stream encapsulation.

[0016] The audio stream encapsulation includes an audio stream obtained by encoding an audio signal using the target encoding mode and the target bitrate mode. The target encoding mode is obtained from multiple encoding modes, and the target bitrate mode is obtained from multiple bitrate modes.

[0017] The audio bitstream is decoded using the target encoding mode and the target bitrate mode to obtain the estimated encoding features corresponding to the audio bitstream.

[0018] By using the target encoding mode, the estimated value of the encoding features corresponding to the audio bitstream is reconstructed to obtain the synthetic audio signal corresponding to the audio bitstream.

[0019] This application provides an audio encoding device, including:

[0020] The second acquisition module is used to, in response to an encoding request for an audio signal, acquire a target encoding mode for the audio signal from a plurality of encoding modes and acquire a target bitrate mode for the audio signal from a plurality of bitrate modes.

[0021] An extraction module is used to extract the encoding features of the audio signal from the audio signal using the target encoding pattern;

[0022] The signal encoding module is used to perform signal encoding processing on the encoding features of the audio signal through the target bitrate mode to obtain the audio bitstream of the audio signal;

[0023] The construction module is used to determine the frame header based on the target encoding mode and the target bitrate mode;

[0024] The generation module is used to generate an audio stream encapsulation of the audio signal based on the audio bitstream and the frame header.

[0025] This application provides an audio decoding device, including:

[0026] The first acquisition module is used to acquire the target encoding mode and the target bitrate mode from the frame header included in the audio stream encapsulation in response to a decoding request for the audio stream encapsulation.

[0027] The audio stream encapsulation includes an audio stream obtained by encoding an audio signal using the target encoding mode and the target bitrate mode. The target encoding mode is obtained from multiple encoding modes, and the target bitrate mode is obtained from multiple bitrate modes.

[0028] The signal decoding module is used to perform signal decoding processing on the audio bitstream using the target encoding mode and the target bitrate mode to obtain the estimated value of the encoding features corresponding to the audio bitstream;

[0029] The reconstruction module is used to reconstruct the estimated values ​​of the coding features corresponding to the audio bitstream using the target coding mode, so as to obtain the synthesized audio signal corresponding to the audio bitstream.

[0030] This application provides an electronic device, the electronic device comprising:

[0031] Memory is used to store executable instructions for a computer;

[0032] The processor, when executing computer-executable instructions stored in the memory, implements the audio encoding method or audio decoding method provided in the embodiments of this application.

[0033] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the audio encoding or audio decoding method provided in this application.

[0034] This application provides a computer program product, including computer-executable instructions, which, when executed by a processor, implement the audio encoding method or audio decoding method provided in this application.

[0035] The embodiments of this application have the following beneficial effects:

[0036] By decoding the audio stream using different encoding modes and bitrate modes, we can obtain encoding feature estimates of varying precision. Then, we can reconstruct the encoded feature estimates of different precision to obtain synthesized audio signals of different quality levels. This improves the diversity of synthesized audio signal quality and meets the actual application needs of users. Attached Figure Description

[0037] Figure 1 This is a schematic diagram comparing the spectrum at different code rates provided in the embodiments of this application;

[0038] Figure 2 This is a schematic diagram of the architecture of the audio encoding and decoding system provided in the embodiments of this application;

[0039] Figure 3A This is a first structural schematic diagram of the electronic device provided in the embodiments of this application;

[0040] Figure 3B This is a schematic diagram of the second structure of the electronic device provided in the embodiments of this application;

[0041] Figure 4A This is a first flowchart illustrating the audio encoding method provided in this application embodiment;

[0042] Figure 4B This is a second flowchart illustrating the audio encoding method provided in the embodiments of this application;

[0043] Figure 4C This is a schematic diagram of the third process of the audio encoding method provided in the embodiments of this application;

[0044] Figure 4D This is a schematic diagram of the fourth process of the audio encoding method provided in the embodiments of this application;

[0045] Figure 4E This is a fifth flowchart illustrating the audio encoding method provided in the embodiments of this application;

[0046] Figure 5A This is a first flowchart illustrating the audio decoding method provided in this application embodiment;

[0047] Figure 5B This is a second flowchart illustrating the audio decoding method provided in the embodiments of this application;

[0048] Figure 5C This is a schematic diagram of the third process of the audio decoding method provided in the embodiments of this application;

[0049] Figure 6A This is a schematic diagram of a channel without grouped convolution provided in an embodiment of this application;

[0050] Figure 6BThis is a schematic diagram of a channel using grouped convolution provided in an embodiment of this application;

[0051] Figure 6C This is a schematic diagram of a voice communication link provided in an embodiment of this application;

[0052] Figure 7 This is a flowchart illustrating the multi-mode, multi-rate speech coding method provided in an embodiment of this application.

[0053] Figure 8 This is a schematic diagram of the filter bank provided in an embodiment of this application;

[0054] Figure 9A This is a schematic diagram of a common convolutional network provided in an embodiment of this application;

[0055] Figure 9B This is a schematic diagram of the dilated convolutional network provided in an embodiment of this application;

[0056] Figure 10 This is a schematic diagram of the bandwidth extension provided in an embodiment of this application;

[0057] Figure 11 This is a schematic diagram of the third neural network provided in an embodiment of this application;

[0058] Figure 12A This is a schematic diagram of the residual block structure used in the coding block provided in the embodiments of this application;

[0059] Figure 12B This is a schematic diagram of the residual unit structure provided in an embodiment of this application;

[0060] Figure 13A This is a schematic diagram of the code stream encapsulation structure under the wideband coding mode provided in the embodiments of this application;

[0061] Figure 13B This is a schematic diagram of the structure of the code stream encapsulation that does not contain residuals or flatness edge information under the ultra-wideband coding mode provided in this application embodiment;

[0062] Figure 13C This is a schematic diagram of the structure of a bitstream encapsulation that includes residuals but does not include flatness edge information under the ultra-wideband coding mode provided in this application embodiment;

[0063] Figure 13D This is a schematic diagram of the structure of a bitstream encapsulation that does not contain residuals but contains flatness edge information under the ultra-wideband coding mode provided in this application embodiment;

[0064] Figure 13E This is a schematic diagram of the structure of a bitstream encapsulation containing residuals and flatness edge information in the ultra-wideband coding mode provided in this application embodiment;

[0065] Figure 14 This is a schematic diagram of the first neural network provided in an embodiment of this application;

[0066] Figure 15A This is a schematic diagram of the code stream encapsulation structure in the broadband stereo coding mode provided in the embodiments of this application;

[0067] Figure 15B This is a schematic diagram of the structure of the code stream encapsulation that does not contain residuals or flatness edge information in the ultra-wideband stereo coding mode provided in this application embodiment;

[0068] Figure 15C This is a schematic diagram of the structure of a bitstream encapsulation that includes residuals but does not include flatness edge information in the ultra-wideband stereo coding mode provided in this application embodiment;

[0069] Figure 15D This is a schematic diagram of the structure of a bitstream encapsulation that does not contain residuals but contains flatness edge information in the ultra-wideband stereo coding mode provided in this application embodiment;

[0070] Figure 15E This is a schematic diagram of the structure of a bitstream encapsulation containing residuals and flatness edge information in the ultra-wideband stereo coding mode provided in this application embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0074] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0075] 1) Neural Network (NN): A mathematical model that mimics the behavioral characteristics of animal neural networks to perform distributed parallel information processing. This type of network relies on the complexity of the system to adjust the relationships between a large number of interconnected nodes, thereby achieving the purpose of information processing.

[0076] 2) Deep Learning (DL): This is a new research direction in the field of Machine Learning (ML). Deep learning learns the inherent patterns and representational layers of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to be able to recognize data such as text, images, and sound.

[0077] 3) Quantization: This refers to the process of approximating a signal's continuous values ​​(or a large number of discrete values) into a finite number (or a small number) of discrete values. Quantization includes vector quantization (VQ) and scalar quantization.

[0078] Vector quantization is an effective lossy compression technique based on Shannon's rate distortion theory. The basic principle of vector quantization is to use the index (also called the quantization value) of the codeword in the code table that best matches the input vector for transmission and storage, while decoding only requires a simple table lookup operation. For example, a vector space can be formed by combining several scalar data points, and this vector space can be divided into several small regions. During quantization, the vector falling into a small region is replaced with its corresponding index as the input vector.

[0079] Scalar quantization is the quantization of a scalar, that is, one-dimensional vector quantization. It divides the dynamic range into several small intervals, each with a representative value (i.e., index). When the input signal falls into a certain interval, the input signal is quantized into that representative value.

[0080] 4) Entropy coding: A lossless coding method that does not lose any information during the encoding process according to the entropy principle. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exponential Golomb coding, and arithmetic coding.

[0081] 5) Quadrature Mirror Filters (QMF): This is a filter pair that includes analysis and synthesis. The QMF analysis filter is used to decompose the subband signal to reduce the signal bandwidth, so that each subband signal can pass smoothly through its respective channel for processing. The QMF synthesis filter is used to synthesize the subband signals recovered from the decoder, such as by zero-value interpolation and bandpass filtering, to reconstruct the original audio signal.

[0082] Speech coding technology aims to transmit as much speech information as possible with minimal network bandwidth. Speech codecs can achieve compression ratios of over 10 times, meaning that 10MB of speech data can be compressed to only 1MB for transmission, significantly reducing the bandwidth required for information transmission. For example, for a wideband speech signal with a sampling rate of 16000Hz, using a 16-bit sampling depth (the fineness of speech intensity recording during sampling), the uncompressed version has a bitrate (data transmission rate per unit time) of 256kbps. However, with speech coding technology, even lossy coding, the reconstructed speech signal quality can approach that of the uncompressed version within a bitrate range of 10-20kbps, and may even be perceived as indistinguishable. For services requiring higher sampling rates, such as 32000Hz ultrawideband speech, the bitrate range must be at least 30kbps.

[0083] In communication systems, to ensure smooth communication, the industry deploys standard voice codec protocols, such as standards from international and domestic standards organizations like ITU-T, 3GPP, IETF, AVS, and CCSA, including standards like G.711, G.722, AMR series, EVS, and OPUS. Figure 1 A comparative spectral analysis diagram at different bitrates is provided to illustrate the relationship between compression bitrate and quality. Curve 101 represents the spectral curve of the original speech, i.e., the uncompressed signal; curve 102 represents the spectral curve of the OPUS encoder at a bitrate of 20kbps; and curve 103 represents the spectral curve of the OPUS encoding at a bitrate of 6kbps. Figure 1 It can be seen that as the encoding bit rate increases, the compressed signal is closer to the original signal.

[0084] The principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples one by one; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0085] The above-mentioned encoding principles are all derived from speech signal modeling, i.e., compression methods based on signal processing, which cannot guarantee the encoding quality of audio. To improve encoding efficiency while ensuring speech quality, embodiments of this application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as a terminal device, a server, or a combination of both. The following description uses the example of an electronic device implemented as a terminal device.

[0086] For example, see Figure 2 , Figure 2 This is a schematic diagram of the architecture of the audio encoding and decoding system 10 provided in the embodiments of this application. The audio decoding system 10 includes: server 200, network 300, terminal device 400 (i.e. encoding end) and terminal device 500 (i.e. decoding end). The network 300 can be a local area network, a wide area network, or a combination of the two.

[0087] In some embodiments, a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as instant messaging clients, web conferencing clients, live streaming clients, browsers, etc. In response to an audio acquisition command triggered by the sender (e.g., the initiator of a web conference, the host, the initiator of a voice call, etc.), the client 410 calls the microphone built into the terminal device 400 to acquire audio signals, and performs audio encoding processing on the acquired audio signals to obtain an audio bitstream.

[0088] For example, client 410 calls the audio encoding method provided in the embodiments of this application to encode the acquired audio signal, that is, to obtain a target encoding mode for the audio signal from multiple encoding modes, and to obtain a target bitrate mode for the audio signal from multiple bitrate modes; to extract the encoding features of the audio signal from the audio signal through the target encoding mode; to perform signal encoding processing on the encoding features of the audio signal through the target bitrate mode to obtain the audio bitstream of the audio signal; to determine the frame header based on the target encoding mode and the target bitrate mode; and to generate an audio bitstream encapsulation of the audio signal based on the audio bitstream and the frame header.

[0089] Client 410 can encapsulate the audio stream and send it to server 200 via network 300, so that server 200 can encapsulate and send the audio stream to the terminal device 500 associated with the recipient (such as participants in a web conference, audience members, or receivers of a voice call).

[0090] After receiving the audio stream encapsulation sent by the server 200, the client 510 (such as an instant messaging client, web conferencing client, live streaming client, browser, etc.) can perform audio decoding processing on the audio stream encapsulation to obtain a synthesized audio signal, thereby realizing audio communication.

[0091] For example, client 510 calls the audio decoding method provided in this application embodiment to decode the received audio bitstream encapsulation, that is, to obtain the target encoding mode and target bitrate mode from the frame header included in the audio bitstream encapsulation; wherein, the audio bitstream included in the audio bitstream encapsulation is obtained by audio encoding the audio signal through the target encoding mode and target bitrate mode, the target encoding mode is obtained from multiple encoding modes, and the target bitrate mode is obtained from multiple bitrate modes; through the target encoding mode and target bitrate mode, the audio bitstream is subjected to signal decoding processing to obtain the encoding feature estimation value corresponding to the audio bitstream; through the target encoding mode, the encoding feature estimation value corresponding to the audio bitstream is subjected to reconstruction processing to obtain the synthesized audio signal corresponding to the audio bitstream.

[0092] In some embodiments, the embodiments of this application can be implemented using cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0093] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. The service interaction functions between the aforementioned 200 servers can be achieved through cloud technology.

[0094] Example, Figure 2 The server 200 shown can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Figure 2 The terminal devices 400 and 500 shown can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, in-vehicle terminals, etc., but are not limited to these. The terminal devices (such as terminal devices 400 and 500) and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0095] In some embodiments, the terminal device or server 200 can also implement the audio encoding or decoding method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a live streaming APP, a web conferencing APP, or an instant messaging APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0096] In some embodiments, multiple servers can form a blockchain, and server 200 is a node on the blockchain. Information connections can exist between each node in the blockchain, and information can be transmitted between nodes through these connections. Data related to the audio encoding or decoding methods provided in this application embodiment (e.g., audio encoding logic, audio decoding logic, high-frequency bitstream, and low-frequency bitstream) can be stored on the blockchain.

[0097] See Figure 3A , Figure 3A This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Taking the electronic device 500 as a terminal device as an example, the following explanation is provided. Figure 3A The illustrated electronic device 500 includes at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The various components in the electronic device 500 are coupled together via a bus system 550. It is understood that the bus system 550 is used to implement communication between these components. In addition to a data bus, the bus system 550 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3A The general labeled all buses as Bus System 550.

[0098] Processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0099] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 520.

[0100] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0101] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0102] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0103] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 including Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0104] In some embodiments, the audio encoding apparatus provided in this application can be implemented in software. Figure 3A An audio encoding device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: a second acquisition module 5551, an extraction module 5552, a signal encoding module 5553, a construction module 5554, and a generation module 5555. The second acquisition module 5551, the extraction module 5552, the signal encoding module 5553, the construction module 5554, and the generation module 5555 are used to implement audio encoding functions. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions implemented.

[0105] See Figure 3B , Figure 3B This is a schematic diagram of the structure of the electronic device 600 provided in the embodiments of this application. Taking the electronic device 600 as a terminal device as an example, the following description is provided. Figure 3B The illustrated electronic device 600 includes at least one processor 620, a memory 650, at least one network interface 630, and a user interface 640. The various components of the electronic device 600 are coupled together via a bus system 650. The memory 650 includes an operating system 651 and a network communication module 652. It should be noted that... Figure 3B The function of the structure in Figure 3A The structure in this application functions similarly. The audio encoding device provided in this embodiment can be implemented in software. Figure 3B An audio decoding device 655 stored in memory 650 is shown. It can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 6551, a signal decoding module 6552, and a reconstruction module 6553. The first acquisition module 6551, the signal decoding module 6552, and the reconstruction module 6553 are used to implement the audio decoding function. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0106] As mentioned above, the audio encoding method provided in this application embodiment can be implemented by various types of electronic devices. See also Figure 4A , Figure 4A This is a flowchart illustrating the audio encoding method provided in this application embodiment. The audio encoding function is achieved through this method. The following is a summary of the process. Figure 4A Steps 101-105 are shown and explained.

[0107] In step 101, in response to an encoding request for an audio signal, a target encoding mode for the audio signal is obtained from a plurality of encoding modes, and a target bitrate mode for the audio signal is obtained from a plurality of bitrate modes.

[0108] The encoding request instructs the audio signal to undergo audio encoding processing. It includes encoding configuration information such as the audio signal's sampling rate and type (e.g., wideband or ultra-wideband). Multiple encoding modes are available, including wideband and ultra-wideband encoding modes. The bitrate mode indicates the use of a specified code table and corresponding bitrate for signal encoding; different bitrate modes correspond to different code tables and bitrates. During the configuration phase, different encoding and bitrate modes can be configured for different audio signals.

[0109] As an example of acquiring audio signals, the encoding end responds to the audio acquisition command triggered by the sender (such as the initiator of a web conference, the host, the initiator of a voice call, etc.) by calling the microphone built into the terminal device of the encoding end to acquire audio signals (also known as input signals).

[0110] In step 102, the encoding features of the audio signal are extracted from the audio signal using the target encoding mode.

[0111] like Figure 4B As shown, Figure 4B This is a flowchart illustrating an audio encoding method provided in an embodiment of this application. When the target encoding mode is a wideband encoding mode, Figure 4B Shown in Figure 4A Step 102 can be implemented through steps 1021A-1022A: In step 1021A, the first neural network is invoked based on the wideband coding mode; in step 1022A, the coding features of the audio signal are extracted from the audio signal through the first neural network.

[0112] Here, when the target coding mode is wideband coding mode, artificial intelligence technology is used to directly extract the coding features of the frequency signal.

[0113] like Figure 4C As shown, Figure 4C This is a flowchart illustrating an audio encoding method provided in an embodiment of this application. Figure 4C Shown in Figure 4B Step 1022A can be achieved through steps 10221A-10222A:

[0114] In step 10221A, feature extraction processing is performed on the audio signal to obtain the audio features of the audio signal.

[0115] Here, in this embodiment, a first neural network (NN) can be invoked based on the audio signal to extract audio features from the audio signal, so that subsequent feature extraction can be performed based on the audio features. It should be noted that this embodiment is not limited to the structure of the first NN; the first NN can be a convolutional neural network, a deep neural network, etc.

[0116] In step 10222A, the audio features are processed using at least one residual unit included in the first neural network to obtain the encoded features of the audio signal.

[0117] Here, the first neural network includes four coding blocks, each of which includes four or five residual units.

[0118] like Figure 4D As shown, Figure 4D This is a flowchart illustrating an audio encoding method provided in an embodiment of this application. When the target encoding mode is ultra-wideband encoding mode, Figure 4D Shown in Figure 4A Step 102 can be achieved through steps 1021B-1024B:

[0119] In step 1021B, the audio signal is subjected to subband decomposition processing to obtain the low-frequency subband signal and the high-frequency subband signal of the audio signal.

[0120] Here, when the target coding mode is ultra-wideband coding mode, it means that the audio signal is an ultra-wideband signal. Subband decomposition processing can be performed on the ultra-wideband signal to obtain the low-frequency subband signal and the high-frequency subband signal of the audio signal.

[0121] It should be noted that the embodiments of this application do not limit the frequency bands of the low-frequency sub-band signal and the high-frequency sub-band signal. That is, the low-frequency sub-band signal and the high-frequency sub-band signal obtained by decomposition can be two sub-band signals obtained by equally dividing the frequency band of the audio signal, or they can be two sub-band signals obtained by non-equally dividing the frequency band of the audio signal. For example, if the effective bandwidth of the audio signal x(n) is 0-16kHz, then the low-frequency sub-band signal X LB (n) and high-frequency subband signal X HB The effective bandwidths of (n) are 0-8kHz and 8-16kHz, respectively, and the low-frequency subband signal X LB (n) and high-frequency subband signal X HB The effective bandwidth of (n) can also be 0-6kHz and 6-16kHz respectively. In addition, the embodiments of this application do not limit the number of segmented frequency bands, that is, two sub-band signals can be obtained by average / non-average segmentation, or more than two sub-band signals can be obtained by average / non-average segmentation of the audio signal frequency band, such as 3, 4 or more sub-band signals.

[0122] As an example of subband decomposition, an audio signal is decomposed into low-frequency subband signals x using a QMF analysis filter. LB (n) and high-frequency subband signal X HB (n), since the low-frequency subband signal has a greater impact on audio coding than the high-frequency subband signal, the low-frequency subband signal and the high-frequency subband signal can be processed differently in the future.

[0123] In some embodiments, step 1021B can be implemented by: sampling the audio signal to obtain a sampled signal, wherein the sampled signal includes multiple sample points obtained by sampling; performing low-pass filtering on the sampled signal to obtain a low-pass filtered signal; performing downsampling on the low-pass filtered signal to obtain a low-frequency subband signal of the audio signal; performing high-pass filtering on the sampled signal to obtain a high-pass filtered signal; and performing downsampling on the high-pass filtered signal to obtain a high-frequency subband signal of the audio signal.

[0124] Here, the audio signal is a continuous analog signal, the sampling signal is a discrete digital signal, and the sampling point is the sampled value obtained from the audio signal.

[0125] As an example, taking an audio signal with a sampling rate of Fs = 32000Hz as the input signal, the audio signal is sampled to obtain a sampled signal x(n) including 640 sample points. The analysis filter (2 channels) in the QMF filter bank is called to perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal. Then, the sampled signal is subjected to high-pass filtering to obtain a high-pass filtered signal. Finally, the low-pass filtered signal is downsampled to obtain the low-frequency subband signal x of the audio signal. LB (n) is used to downsample the high-pass filtered signal to obtain the high-frequency subband signal X of the audio signal. HB (n). Low-frequency subband signal X LB (n) and high-frequency subband signal X HB The effective bandwidths of (n) are 0-8kHz and 8-16kHz, respectively, and the low-frequency subband signal X LB (n) and high-frequency subband signal X HB The number of sample points for (n) is 320.

[0126] It should be noted that a QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, representing the QMF low-pass signal and the QMF high-pass signal, respectively. The low-pass and high-pass signals recovered at the decoding end are then synthesized by the QMF synthesis filter to recover the reconstructed signal with the corresponding sampling rate Fs of the input signal.

[0127] In step 1022B, the low-frequency features of the low-frequency subband signal are extracted from the low-frequency subband signal using the second neural network.

[0128] It should be noted that step 1022B is similar to step 10221A, the difference being that step 10221A processes audio signals, while step 1022B processes low-frequency subband signals. The first neural network and the second neural network have similar structures. For example, the second neural network, like the first neural network, includes four coding blocks, each of which includes four or five residual units.

[0129] In some embodiments, step 1022B can be implemented through steps 10221B-10222B:

[0130] In step 10221B, feature extraction processing is performed on the low-frequency subband signal to obtain the low-frequency features of the low-frequency subband signal.

[0131] Here, in this embodiment, a second neural network (NN) can be invoked based on the low-frequency sub-band signal to extract low-frequency features from the low-frequency sub-band signal, so that subsequent feature extraction can be performed based on important low-frequency features. It should be noted that this embodiment is not limited to the structure of the second NN; the second NN can be a convolutional neural network, a deep neural network, etc.

[0132] In some embodiments, step 1022B can be implemented by: performing causal convolution processing on the low-frequency subband signal of the audio signal through a second neural network to obtain causal convolution features; and performing pooling processing on the causal convolution features to obtain low-frequency coding features of the low-frequency subband signal.

[0133] For example, see Figure 11 The network structure diagram of the second NN shown is as follows. The second NN includes a causal convolutional layer and a preprocessing layer. First, a 16-channel causal convolutional layer is invoked to expand the input tensor (i.e., the low-frequency subband signal) into a 16×320 causal convolutional feature. Then, the 16×320 causal convolutional feature is preprocessed by the preprocessing layer. For example, after performing a convolution operation on the 16×320 causal convolutional feature, a pooling process with a factor of 2 is performed, and the activation function can be a Parametric Rectified Linear Unit (PReLU) to generate a 16×160 tensor (i.e., low-frequency encoded features).

[0134] In step 10222B, at least one residual unit included in the second neural network is used to perform residual processing on the low-frequency coding features to obtain the low-frequency features of the low-frequency subband signal.

[0135] Here, residual processing is performed on the low-frequency coding features of the low-frequency subband signal to ensure that while the low-frequency coding features are fully learned, the shallow feature information of the low-frequency coding features can be better utilized, thus avoiding the omission of the shallow feature information of the low-frequency coding features.

[0136] In some embodiments, step 10222B can be implemented by steps 102221B-102222B.

[0137] In step 102221B, the low-frequency features are processed by feature residual processing through at least one residual unit included in the second neural network to obtain the residual features of the low-frequency subband signal.

[0138] In some embodiments, when at least one residual unit is a residual unit, step 102221B can be implemented in the following way: by performing residual processing on the low-frequency features through a residual unit, the residual features of the low-frequency subband signal are obtained.

[0139] In some embodiments, when at least one residual unit is a plurality of cascaded residual units, step 102221B can be implemented in the following manner: performing residual processing on the low-frequency coding features through the first residual unit of the plurality of cascaded residual units; outputting the residual result output by the first residual unit to the subsequent cascaded residual units, continuing to perform residual processing and outputting residual results through the subsequent cascaded residual units; and using the residual result output by the last residual unit as the residual feature of the low-frequency subband signal.

[0140] For example, the second neural network includes four coding blocks, each of which includes four or five residual units. Figure 12A As shown, when at least one residual unit used for feature residuals has five cascaded residual units, the first residual unit performs residual processing on the low-frequency coded features and outputs the residual result of the first residual unit to the second residual unit; the second residual unit performs residual processing on the residual result of the first residual unit and outputs the residual result of the second residual unit to the third residual unit; the third residual unit performs residual processing on the residual result of the second residual unit and outputs the residual result of the third residual unit to the fourth residual unit; the fourth residual unit performs residual processing on the residual result of the third residual unit and outputs the residual result of the fourth residual unit to the fifth residual unit; the fifth residual unit performs residual processing on the residual result of the fourth residual unit to obtain the residual features of the low-frequency subband signal.

[0141] The embodiments of this application are not limited to the number of coding blocks in the neural network, and can be any positive integer such as 2, 3, 4, 5, etc. The embodiments of this application are also not limited to the number of residual units in the coding block, and can be any positive integer such as 2, 3, 4, 5, 6, etc. The number of residual units in multiple coding blocks can be the same or different. For example, one coding block contains 4 residual units and another coding block contains 5 residual units.

[0142] In some embodiments, the processing of the residual unit is as follows: the k-th residual unit of multiple cascaded residual units performs the following processing: convolving the input of the k-th residual unit to obtain the convolution result of the k-th residual unit; adding the convolution result of the k-th residual unit to the input of the k-th residual unit to obtain the residual result output by the k-th residual unit, where k is a positive integer that increases sequentially, 1≤k≤J, and J is the number of residual units. When k is 1, the input of the k-th residual unit is the low-frequency coding feature; when k is not 1, the input of the k-th residual unit is the residual result output by the (k-1)-th residual unit of the residual feature.

[0143] Following the above embodiments, each residual unit includes a dilated convolution operator. The convolution result of the k-th residual unit is obtained by performing convolution processing on the input of the k-th residual unit. This can be achieved as follows: The k-th residual unit of multiple cascaded residual units performs the following processing: dilated convolution processing is performed on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit. That is, the low-frequency encoded features are dilated convolutionally processed using the dilated convolution operator included in the first residual unit to obtain the dilated convolution result of the first residual unit. The j-th residual unit of multiple cascaded residual units performs the following processing: dilated convolution processing is performed on the residual result output by the (j-1)-th residual unit using the dilated convolution operator included in the j-th residual unit to obtain the dilated convolution result of the j-th residual unit, where j is a sequentially increasing positive integer, 1 < j ≤ J, and J is the number of residual units. It should be noted that each residual unit contains a dilated convolution operator with a specified dilation rate. Using asymptotic dilation rate dilated convolution operators is equivalent to using different receptive fields to extract features from the input at different resolutions, allowing for a more comprehensive analysis of the data. After each residual unit undergoes convolution processing by the dilated convolution operator with the specified dilation rate, it is added to the shallow features from the jump connections (i.e., the input of each residual unit), thus directly utilizing shallow feature information. This allows the network to fully leverage shallow feature information during the learning process.

[0144] Following the above embodiments, each residual unit includes not only a dilated convolution operator but also at least one causal convolution operator. After performing convolution processing on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit, causal convolution processing is performed on the obtained dilated convolution result through the at least one causal convolution operator included in the k-th residual unit, and the resulting causal convolution result is used as the convolution result of the k-th residual unit. That is, causal convolution processing is performed on the dilated convolution result of the first residual unit through the at least one causal convolution operator included in the first residual unit, and the resulting causal convolution result is used as the convolution result output by the first residual unit. The dilated convolution operator included in the j-th residual unit performs dilated convolution processing on the residual result output of the (j-1)-th residual unit to obtain the dilated convolution result of the j-th residual unit. Then, the dilated convolution result of the j-th residual unit is subjected to causal convolution processing using at least one causal convolution operator included in the j-th residual unit. The causal convolution result of the j-th residual unit is then used as the convolution result of the j-th residual unit. It should be noted that each residual unit also includes at least one causal convolution operator, which is used to further extract local information of the features input to the causal convolution operator.

[0145] In some embodiments, grouped convolution can be applied to the convolution operators (including dilated convolution operators and causal convolution operators) of residual units. Grouped convolution divides the input channels into multiple groups for convolution operations, associating only the input and output channels within each group. It should be noted that when the input channels are divided into multiple groups, the corresponding output channels are also divided into multiple groups; that is, the number of input channel groups is the same as the number of output channel groups, thus ensuring that after convolution within a group, the input and output channels are associated only within that group. Here, it is assumed that the input channels and output channels of a feature input to a certain convolution operator are 4. If the number of groups is 1, each input channel is associated with all 4 output channels. If the number of groups is 2, the 4 input channels are first divided into two groups, 0-1 and 2-3. Within each group, the input channel is associated with the output channel within that group. For example, input channel 0-1 in the first group is associated with output channel 0-1, and input channel 2-3 in the second group is associated with output channel 2-3. Figure 6A As shown, when the grouped convolution scheme is not used, each input channel is associated with four output channels; for example... Figure 6B As shown, when the grouped convolution scheme is not used, the 0th output channel is only associated with the 0-1 input channel and not with the 2-3 input channels; similarly, the 2nd output channel is only associated with the 2-3 input channels and not with the 0-1 input channel. This comparison demonstrates that introducing grouped convolution avoids any input channel being associated with all output channels, reducing the number of connections and thus lowering complexity.

[0146] For example, when applying grouped convolution to the dilated convolution operator included in the residual unit, dilated convolution processing of low-frequency encoded features can be achieved as follows: the input channels of the low-frequency encoded features are grouped to obtain multiple groups, wherein each group includes a first element corresponding to at least two channels in the low-frequency encoded features; dilated convolution processing is then performed on the first element within each group. Similarly, when applying grouped convolution to the causal convolution operator included in the residual unit, causal convolution processing of the resulting dilated convolution can be achieved as follows: the input channels of the dilated convolution result are grouped to obtain multiple groups, wherein each group includes a second element corresponding to at least two channels in the dilated convolution result; causal convolution processing is then performed on the second element within each group.

[0147] Following step 102221B, in step 102222B, feature encoding processing is performed on the residual features to obtain the low-frequency features of the low-frequency subband signal.

[0148] Here, the residual features are processed by feature encoding to obtain the low-frequency features of the low-frequency subband signal, so that subsequent signal encoding processing can be performed based on the low-frequency features to obtain the low-frequency bitstream of the audio signal.

[0149] In some embodiments, step 102222B can be implemented by: performing convolution processing on the residual features to obtain convolution features, wherein the number of channels of the convolution features is greater than the number of channels of the residual features; and performing pooling processing on the convolution features to obtain low-frequency features of the low-frequency subband signal.

[0150] For example, a second neural network (NN) is invoked based on the low-frequency sub-band signal. After processing by the residual units in the second NN, residual features are obtained. Then, convolutional layers in the second NN are used to convolve the residual features to increase the number of channels. Finally, pooling layers in the second NN are used to pool the convolutional features to obtain the low-frequency features of the low-frequency sub-band signal. Alternatively, the second NN can include causal convolutional layers. Causal convolutions are performed on the low-frequency features to obtain causal convolutional low-frequency features. These causal convolutional low-frequency features are then used for signal encoding to obtain the low-frequency bitstream of the audio signal.

[0151] For example, such as Figure 11 As shown, when the second neural network is invoked based on the low-frequency subband signal, the low-frequency coding features are obtained through the second neural network. Figure 11The first neural network (NN) consists of a 16×160 tensor obtained after preprocessing. The second NN comprises four concatenated encoding blocks with different downsampling factors. Each encoding block contains a residual block (including at least one residual unit), a convolutional layer, and a pooling layer. Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer doubles the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The pooling layer is a pooling operation with a down factor to perform downsampling and achieve data compression. Here, the down factors of the four encoding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four encoding blocks are set to 32, 64, 128, and 256, respectively. After processing by the four encoding blocks, the input 16×160 tensor is transformed into tensors of 32×80, 64×20, 128×5, and 256×1, respectively. For example, the low-frequency encoded features (i.e., a 16×160 tensor) are processed using the residual block in the first encoding block, and the residual result output from the residual block in the first encoding block is fed into the feature encoding block in the first encoding block. After processing by the feature encoding block in the first encoding block (including a convolutional layer and a pooling layer), the encoding result of the feature encoding block in the first encoding block (i.e., a 32×80 tensor) is obtained, and the encoding result of the feature encoding block in the first encoding block (i.e., a 32×80 tensor) is output into the second encoding block; the low-frequency encoded features (i.e., a 16×160 tensor) are processed using the residual block in the second encoding block, and the residual result is fed into the feature encoding block in the first encoding block. The encoding result of the feature encoding block in the first encoding block (i.e., a 32×80 tensor) is subjected to residual processing. The residual result of the residual block in the second encoding block is then output to the feature encoding block in the second encoding block. After processing by the feature encoding block in the second encoding block (including a convolutional layer and a pooling layer), the encoding result of the feature encoding block in the second encoding block (i.e., a 64×20 tensor) is obtained. The encoding result of the feature encoding block in the first encoding block (i.e., a 64×20 tensor) is then output to the third encoding block. This process is repeated sequentially, and the output of the last encoding block is used as the low-frequency feature. The number of encoding blocks in this embodiment is not limited and can be any positive integer, such as 2, 3, 4, or 5.

[0152] In step 1023B, high-frequency analysis processing is performed on the high-frequency subband signal to obtain the high-frequency characteristics of the high-frequency subband signal.

[0153] In some embodiments, a fifth neural network can be invoked to extract high-frequency features from the high-frequency subband signal. The fifth neural network has a similar structure to the second neural network. The dimension of the high-frequency features is smaller than the dimension of the low-frequency features.

[0154] In some embodiments, high-frequency analysis processing is performed on the high-frequency subband signal to obtain its high-frequency characteristics. This can be achieved by: performing frame segmentation on the high-frequency subband signal to obtain multiple subframes; performing band extension processing on each subframe to obtain the subband spectral envelope of each subframe; and using the subband spectral envelopes corresponding to the multiple subframes as the high-frequency characteristics of the high-frequency subband signal. The number of subframes can be two, four, six, etc., and the embodiments of this application are not limited to the number of subframes.

[0155] Here, since the high-frequency subband signal is of relatively low importance to quality compared to the low-frequency subband signal, the high-frequency subband signal can be compressed by another method, namely bandwidth spreading (recovering broadband speech signals from bandwidth-limited narrowband speech signals), to quickly compress the high-frequency subband signal and extract its high-frequency features.

[0156] In some embodiments, the frequency band extension processing of each subframe to obtain the sub-band spectral envelope of each subframe can be achieved by: performing frequency domain transformation processing based on multiple sample points included in the subframe to obtain the transformation coefficients corresponding to the multiple sample points respectively; dividing the transformation coefficients corresponding to the multiple sample points into multiple sub-bands; performing mean processing on the transformation coefficients included in each sub-band to obtain the average energy corresponding to each sub-band, and using the average energy as the sub-band spectral envelope corresponding to each sub-band.

[0157] It should be noted that the frequency domain transformation methods in this application include Modified Discrete Cosine Transform (MDCT), Discrete Cosine Transform (DCT), Fast Fourier Transform (FFT), etc., and the embodiments of this application are not limited to frequency domain transformation methods. The mean processing in this application includes arithmetic mean and geometric mean, and the embodiments of this application are not limited to mean processing methods.

[0158] In some embodiments, frequency domain transformation processing is performed based on multiple sample points included in a subframe to obtain the transformation coefficients corresponding to the multiple sample points respectively, including: performing the following processing for each subframe: obtaining neighboring subframes adjacent to the subframe; performing discrete cosine transform processing on the multiple sample points included in the subframe based on the multiple sample points included in the neighboring subframes and the multiple sample points included in the subframe to obtain the transformation coefficients corresponding to the multiple sample points included in the subframe respectively.

[0159] As an example, for a high-frequency subband signal X comprising 320 points... HB (n), representing the 320-point high-frequency subband signal XHB (n) is divided into two subframes (the first subframe and the second subframe), each containing 160 points. For any subframe, i.e., the high-frequency sub-band signal containing 160 points, the Modified Discrete Cosine Transform (MDCT) is called to generate 160 MDCT coefficients (i.e., the transform coefficients corresponding to the multiple sample points included in the subframe). Specifically, if there is a 50% overlap, the first subframe of the (n+1)th frame can be merged (stitched) with the second subframe of the nth frame, and the MDCT of 320 points can be calculated to obtain the 160 MDCT coefficients of the first subframe of the (n+1)th frame; the second subframe of the (n+1)th frame can be merged with the first subframe of the (n+1)th frame, and the MDCT of 320 points can be calculated to obtain the 160 MDCT coefficients of the second subframe of the (n+1)th frame.

[0160] In some embodiments, the process of performing geometric mean processing on the transform coefficients included in each subband is as follows: determining the sum of squares of the transform coefficients corresponding to the sample points included in each subband; and determining the ratio of the sum of squares to the number of sample points included in the subband as the average energy corresponding to each subband.

[0161] Following step 1023B, in step 1024B, the low-frequency features and high-frequency features are determined as the encoding features of the audio signal.

[0162] Following step 102, in step 103, the encoding features of the audio signal are processed by signal encoding using the target bitrate mode to obtain the audio bitstream of the audio signal.

[0163] In some embodiments, when the target coding mode is a wideband coding mode, the target bitrate mode is used to indicate the use of a target code table to encode the coding features of the audio signal; step 103 can be implemented in the following way: the coding features of the audio signal are quantized using the target code table to obtain quantized values ​​of the coding features; the quantized values ​​are entropy encoded using the target bitrate corresponding to the target code table to obtain the audio bitstream of the audio signal.

[0164] like Figure 4E As shown, Figure 4E This is a flowchart illustrating an audio encoding method provided in an embodiment of this application. When the target encoding mode is an ultra-wideband encoding mode, the target bit rate mode includes indicating that a first code table is used to encode low-frequency features and a second code table is used to encode high-frequency features. Figure 4E Shown in Figure 4AStep 103 can be implemented through steps 1031-1033: In step 1031, the low-frequency features are quantized using a first code table to obtain quantized values, and then entropy-encoded using the code rate corresponding to the first code table to obtain the low-frequency bitstream of the low-frequency subband signal; In step 1032, the high-frequency features are quantized using a second code table to obtain quantized values, and then entropy-encoded using the code rate corresponding to the second code table to obtain the high-frequency bitstream of the high-frequency subband signal; In step 1033, the audio bitstream of the audio signal is constructed based on the low-frequency and high-frequency bitstreams. The quantization degree (i.e., quantization precision) corresponding to different code tables can be different. For example, the quantization precision of the first code table is greater than that of the second code table. Since the value range of each dimension in the low-frequency feature is [-1, 1], the interval [-1, 1] can be divided into 11 equal parts to form a first code table containing 11 elements. The low-frequency feature is then quantized using the first code table. Since the value range of each dimension in the high-frequency feature is [-1, 1], the interval [-1, 1] can be divided into 5 equal parts to form a second code table containing 5 elements. The low- and high-frequency features are then quantized using the second code table.

[0165] Here, for the low-frequency characteristics F of the low-frequency subband signal LB (n) and the high-frequency coding characteristics F of the high-frequency subband signal HB (n) can be a method for scalar quantization (quantizing each component individually) and entropy coding. Furthermore, the embodiments of this application do not limit the combination of vector quantization (combining multiple adjacent components into a vector for joint quantization) and entropy coding techniques.

[0166] It should be noted that when the target bit rate mode only includes the instruction to use the first code table to encode the low-frequency features and the second code table to encode the high-frequency features, the low-frequency bit stream and the high-frequency bit stream are combined to obtain the audio bit stream of the audio signal.

[0167] In some embodiments, the target bitrate mode further includes instructing the use of a third code table to perform residual coding on the residual features of high-frequency features; before step 1033, the sub-band corresponding to the sub-frame of the high-frequency sub-band signal is determined, and the sub-band is divided into multiple subsets; the transform coefficients included in each subset are averaged to obtain the average energy corresponding to each subset, and the average energy is used as the envelope value corresponding to each subset; in the quantized values ​​of the high-frequency features, the quantized value of the sub-band spectrum envelope corresponding to the subset is determined, and the difference between the envelope value corresponding to the subset and the quantized value of the sub-band spectrum envelope is used as the first residual value; based on the first residual value, the residual features of the high-frequency features are determined; the residual features of the high-frequency features are quantized using the third code table to obtain the quantized value of the residual features of the high-frequency features, and the quantized value of the residual features of the high-frequency features is entropy-coded using the bitrate corresponding to the third code table to obtain the residual bitstream of the residual features; correspondingly, step 1033 can be implemented in the following way: the low-frequency bitstream, the high-frequency bitstream, and the residual bitstream are used as the audio bitstream of the audio signal. The quantization level of the third code table can be different from that of the first and second code tables.

[0168] In some embodiments, when the number of residual values ​​is N, where N is a positive integer greater than 1, the residual features of high-frequency features are determined based on the first residual value. This can be achieved by: determining the quantization value of the nth residual value and the sum of the quantization value of the nth residual value and the quantization value of the subband spectrum envelope of the corresponding subband; taking the difference between the envelope value corresponding to the subset and the sum as the (n+1)th residual value; and determining the N residual values ​​as the residual features of high-frequency features. Here, n is a positive integer that increases sequentially, and 1 ≤ n ≤ N.

[0169] Taking a case with two residual values ​​as an example, the residual values ​​are determined in the following way:

[0170] 1) For every 10ms subframe, the subframe is divided into 4 subbands, and each subband is further divided into 2 subsets evenly.

[0171] For example, the set of envelope values ​​for the four subbands is {e1, e2, e3, e4}.

[0172] 2) For each subset of each subband, calculate the envelope value of that subset.

[0173] For example, the set of envelope values ​​for all subsets is {e 1a e 1b e 2a e 2b e 3a e 3b e 4a e 4b}

[0174] 3) Subtract the corresponding high-frequency feature vector quantization value (i.e., the quantization value of the high-frequency feature) from the envelope value of each subset to obtain the first high-frequency feature vector residual (referred to as the first residual value).

[0175] For example, the quantization value of the high-frequency feature vector is The residual of the first high-frequency eigenvector is

[0176] 4) The first high-frequency feature residual above is subjected to similar quantization and entropy encoding to obtain the quantized value of the first high-frequency feature vector residual (referred to as the quantized value of the first residual value).

[0177] Here, considering that the dynamic range of the residual value is much lower than that of the original value, only 2 bits need to be allocated when scalar quantizing each residual. Thus, a reference code rate of 2*16*50=1.6kbps is needed (similarly, because it is non-uniformly distributed entropy coding, the actual code rate is generally less than 1.6kbps) to obtain the quantized value of the first high-frequency feature vector residual.

[0178] 5) Subtract the sum of the quantized value of the corresponding high-frequency feature vector and the quantized value of the residual of the first high-frequency feature vector from the envelope value of each subset to obtain the second high-frequency feature vector residual (referred to as the second residual value).

[0179] In step 104, the frame header is determined based on the target encoding mode and the target bit rate mode.

[0180] Here, an empty frame header can be constructed first. This empty frame header includes encoding mode bits and bitrate mode bits. The target encoding mode is written into the encoding mode bits included in the empty frame header, and the target bitrate mode is written into the bitrate mode bits included in the empty frame header, thus determining the frame header. The embodiments of this application are not limited to the number of bits in the frame header, nor are they limited to the positions of the target encoding mode and target bitrate mode in the frame header. For example, the target encoding mode can be written into the first bit of the frame header, or it can be written into the last bit of the frame header, and it is not limited to the number of bits occupied by the encoding mode bits and bitrate mode bits in the frame header.

[0181] In step 105, an audio stream encapsulation of the audio signal is generated based on the audio bitstream and the frame header.

[0182] In some embodiments, before step 105, the flatness edge information of the high-frequency subband signal is determined; therefore, step 105 can be implemented in the following way: combining the audio bitstream, frame header and flatness edge information to obtain the audio bitstream encapsulation of the audio signal.

[0183] For example, the frame header can be placed at the beginning of the audio stream encapsulation, and the audio stream and flatness edge information can be placed after the frame header. This application embodiment is not limited to the order of the audio stream and flatness edge information.

[0184] In some embodiments, determining the flatness edge information of the high-frequency subband signal can be achieved by: dividing the transform coefficients of a subframe of the high-frequency subband signal into multiple blocks; determining a first flatness for each block and a second flatness for a specified low-frequency band of the audio signal; when the first flatness is less than the second flatness, or the first flatness is less than a flatness threshold, setting the flatness edge information to a first value, wherein the first value indicates that flattening processing is required during audio decoding; when the first flatness is greater than or equal to the second flatness, and the first flatness is greater than or equal to the flatness threshold, setting the flatness edge information to a second value, wherein the second value indicates that flattening processing is not required during audio decoding.

[0185] For example, the first value can be 1 and the second value can be 0; or the first value can be 0 and the second value can be 1.

[0186] In some embodiments, the frame header further includes at least one channel bit, which indicates whether the audio signal is encoded using a mono coding method or a stereo coding method. When the audio signal is a stereo signal downmixed from the input signals of the left and right channels, the channel bit in the frame header is set to a first value (e.g., 1), indicating that the audio signal is encoded using a stereo coding method. Parametric stereo coding is performed on the input signals of the left and right channels to obtain a stereo feature vector. Based on the stereo feature vector, the audio bitstream, and the frame header, an audio bitstream encapsulation of the audio signal is generated. When the audio signal is a mono input signal, the channel bit in the frame header is set to a second value (e.g., 0), indicating that the audio signal is encoded using a mono coding method. Based on the audio bitstream and the frame header, an audio bitstream encapsulation of the audio signal is generated.

[0187] As mentioned above, the audio decoding method provided in this application can be implemented by various types of electronic devices. See also Figure 5A , Figure 5A This is a flowchart illustrating the audio decoding method provided in this application embodiment. The audio decoding method implements the audio decoding function, and it is the inverse process of the audio encoding method described above. The following describes the process in conjunction with... Figure 5A The steps shown are explained.

[0188] In step 201, in response to a decoding request for the audio stream encapsulation, the target encoding mode and the target bitrate mode are obtained from the frame header included in the audio stream encapsulation.

[0189] The audio stream encapsulation includes an audio stream obtained by encoding the audio signal using a target encoding mode and a target bitrate mode. The target encoding mode is obtained from multiple encoding modes, and the target bitrate mode is obtained from multiple bitrate modes.

[0190] Here, the decoding request is used to instruct the audio signal to undergo audio decoding processing.

[0191] As an example, through such Figure 4A The audio encoding method shown encodes an audio stream encapsulation, which is then transmitted to the decoding end. Upon receiving the audio stream encapsulation, the decoding end performs audio decoding processing. First, it obtains the target encoding mode and target bitrate mode from the frame header included in the audio stream encapsulation. For example, it obtains the target encoding mode from the encoding mode bit included in the frame header and the target bitrate mode from the bitrate mode bit included in the frame header.

[0192] In step 202, the audio bitstream is processed by signal decoding using the target encoding mode and the target bitrate mode to obtain the estimated value of the encoding features corresponding to the audio bitstream.

[0193] It should be noted that signal decoding is the inverse process of signal encoding. Since the decoding process of the received bitstream is the reverse of the encoding process, the value generated during decoding is an estimate relative to the value generated during encoding. For example, the estimated value of the encoded features generated during decoding is an estimate relative to the encoded features in the encoding process.

[0194] In some embodiments, the target coding mode is a wideband coding mode, and the target bitrate mode is used to indicate the use of a target code table to encode the coding features of the audio signal. Step 202 can be implemented in the following way: by performing entropy decoding on the audio bitstream using the target bitrate corresponding to the target code table, the quantization value corresponding to the audio bitstream is obtained; by performing inverse quantization on the quantization value corresponding to the audio bitstream using the target code table, the estimated value of the coding features corresponding to the audio bitstream is obtained.

[0195] See Figure 5B , Figure 5B This is a flowchart illustrating the audio decoding method provided in this application embodiment. The target encoding mode is an ultra-wideband encoding mode. The target bitrate mode includes an instruction to use a first code table to encode the low-frequency features of the audio signal and a second code table to encode the high-frequency features of the audio signal. The audio bitstream includes a low-frequency bitstream and a high-frequency bitstream. Figure 5BStep 202 can be implemented through steps 2021-2024: In step 2021, low-frequency and high-frequency bitstreams are obtained from the audio bitstream using the target coding mode; in step 2022, entropy decoding is performed on the low-frequency bitstream using the bitrate corresponding to the first code table to obtain the quantization value corresponding to the low-frequency bitstream, and inverse quantization is performed on the quantization value corresponding to the low-frequency bitstream using the first code table to obtain the low-frequency feature estimate value corresponding to the low-frequency bitstream; in step 2023, entropy decoding is performed on the high-frequency bitstream using the bitrate corresponding to the second code table to obtain the quantization value corresponding to the high-frequency bitstream, and inverse quantization is performed on the quantization value corresponding to the high-frequency bitstream using the second code table to obtain the high-frequency feature estimate value corresponding to the high-frequency bitstream; in step 2024, the coding feature estimate value corresponding to the audio bitstream is determined based on the low-frequency feature estimate value and the high-frequency feature estimate value.

[0196] For example, for a low-frequency bitstream, entropy decoding is first performed on the low-frequency bitstream using the bitrate corresponding to the first code table to obtain the quantized value corresponding to the low-frequency bitstream. Then, by looking up the first code table, the estimated low-frequency feature value F′ corresponding to the low-frequency bitstream is obtained. LB (n); For high-frequency bitstreams, first, entropy decoding is performed on the high-frequency bitstream using the bitrate corresponding to the second code table to obtain the quantized value corresponding to the high-frequency bitstream, and then the high-frequency feature estimate F′ corresponding to the high-frequency bitstream is obtained by looking up the second code table. HB (n). It should be noted that when the target bit rate mode only includes the instruction to use the first code table to encode the low-frequency features of the audio signal and the second code table to encode the high-frequency features of the audio signal, the low-frequency feature estimates and the high-frequency feature estimates are determined as the encoding feature estimates corresponding to the audio bitstream. Among them, the high-frequency feature estimates are used to participate in the high-frequency reconstruction processing to generate high-frequency subband signal estimates.

[0197] In some embodiments, the audio bitstream further includes a residual bitstream, and the target bitrate mode further includes an instruction to use a third code table to perform residual encoding on the residual features of the high-frequency features corresponding to the audio bitstream. Before step 2024, the residual bitstream is entropy-decoded using the bitrate corresponding to the third code table to obtain the quantization value corresponding to the residual bitstream, and the quantization value corresponding to the residual bitstream is inverse-quantized using the third code table to obtain the residual feature estimate corresponding to the residual bitstream. Correspondingly, step 2024 can be implemented as follows: the high-frequency feature estimate and the residual feature estimate are summed to determine the final high-frequency feature estimate; the low-frequency feature estimate and the final high-frequency feature estimate are determined as the coding feature estimate corresponding to the audio bitstream.

[0198] In this process, the i-th value in the high-frequency feature estimate is added to the (2i-1)-th and 2i-th values ​​in the residual feature estimate, respectively, to obtain the (2i-1)-th and 2i-th values ​​in the final high-frequency feature estimate. For example, if the high-frequency feature estimate is F′... HB (n)=[e′1,e′2,...,e′8],The residual characteristic estimate is F′C(n)=[e′ 1C ,e′ 2C , ..., e′ 16C ], then the final estimate of the high-frequency features is F′ FHB (n)=[e′1+e′ 1C ,e′1+e′ 2C , ..., e′8+e′ 16C ].

[0199] In step 203, the estimated value of the coding features corresponding to the audio bitstream is reconstructed using the target coding mode to obtain the synthesized audio signal corresponding to the audio bitstream.

[0200] It should be noted that the reconstruction process is the inverse of the extraction process. Since the decoding process at the decoding end is the inverse of the encoding process at the encoding end, the value generated during the decoding process is an estimate relative to the value generated during the encoding process. For example, the synthesized audio signal generated during the decoding process is an estimate relative to the audio signal generated during the encoding process.

[0201] In some embodiments, the target coding mode is a wideband coding mode, and step 203 can be implemented in the following way: calling a third neural network based on the wideband coding mode; and reconstructing the estimated value of the coding features corresponding to the audio bitstream through the third neural network to obtain the synthetic audio signal corresponding to the audio bitstream.

[0202] It should be noted that the reconstruction process using the third neural network is the reverse process of the extraction process using the first neural network.

[0203] As an example, reconstructing the estimated encoded features of an audio bitstream to obtain a synthesized audio signal can be achieved as follows: Using at least one residual unit within a third neural network, residual processing is performed on the estimated encoded features of the audio bitstream to obtain estimated audio features; then, feature reconstruction processing is performed on these estimated audio features to obtain the synthesized audio signal. The third neural network comprises four decoding blocks, each containing four or five residual units.

[0204] The embodiments of this application are not limited to the number of decoding blocks, which can be any positive integer such as 2, 3, 4, 5, etc. The embodiments of this application are also not limited to the number of residual units in the decoding block, which can be any positive integer such as 2, 3, 4, 5, 6, etc. See also Figure 5C , Figure 5C This is a flowchart illustrating the audio decoding method provided in this application embodiment, where the target encoding mode is ultra-wideband encoding mode. Figure 5C Step 203 can be achieved through steps 2031-2033:

[0205] In step 2031, the low-frequency feature estimates included in the encoded features are reconstructed using the fourth neural network to obtain the low-frequency sub-band signal estimates corresponding to the audio bitstream. The structure of the fourth neural network is similar to that of the third neural network.

[0206] In some embodiments, step 2031 can be implemented by steps 20311-20312.

[0207] In step 20311, the low-frequency feature estimates are processed using at least one residual unit of the fourth neural network to obtain low-frequency coded feature estimates. The fourth neural network comprises four decoding blocks, each of which includes four or five residual units.

[0208] In some embodiments, step 20311 can be implemented through steps 203111-203112:

[0209] In step 203111, the low-frequency feature estimates are subjected to feature decoding processing to obtain residual features.

[0210] For example, feature decoding is the inverse process of feature encoding. By performing feature decoding on the low-frequency feature estimate, residual features (an estimate) are obtained.

[0211] In some embodiments, step 203111 can be implemented by: performing convolution processing on the low-frequency features to obtain convolutional features, wherein the number of channels in the convolutional features is less than the number of channels in the low-frequency features; and performing upsampling processing on the convolutional features to obtain residual features.

[0212] Of course, before step 203111, causal convolution can be performed on the low-frequency features to obtain the low-frequency features after causal convolution, and step 203111 can be performed based on the low-frequency features after causal convolution, that is, feature decoding processing is performed on the low-frequency features after causal convolution to obtain residual features.

[0213] In step 203112, the residual features are processed by at least one residual unit included in the fourth neural network to obtain the low-frequency coded feature estimate.

[0214] Here, by performing residual processing on the residual features, we can ensure that while learning the residual features comprehensively, we can also make better use of the shallow feature information of the residual features and avoid missing shallow feature information.

[0215] In some embodiments, when at least one residual unit is a plurality of cascaded residual units, step 203112 can be implemented in the following manner: by performing residual processing on the residual features through a residual unit, a low-frequency coding feature estimate is obtained.

[0216] In some embodiments, when at least one residual unit is a plurality of cascaded residual units, step 203112 can be implemented in the following manner: residual processing is performed on the residual features through the first residual unit of the plurality of cascaded residual units; the residual result output by the first residual unit is output to the subsequent cascaded residual units, and residual processing and output of residual results are continued through the subsequent cascaded residual units; the residual result output by the last residual unit is used as the low-frequency coding feature estimate.

[0217] In some embodiments, the processing of the residual unit is as follows: the k-th residual unit of multiple cascaded residual units performs the following processing: convolving the input of the k-th residual unit to obtain the convolution result of the k-th residual unit; adding the convolution result of the k-th residual unit to the input of the k-th residual unit to obtain the residual result output by the k-th residual unit, where k is a positive integer that increases sequentially, 1≤k≤J, and J is the number of residual units. When k is 1, the input of the k-th residual unit is the residual feature; when k is not 1, the input of the k-th residual unit is the residual result output by the (k-1)-th residual unit.

[0218] Following the above embodiments, each residual unit includes a dilated convolution operator. The convolution process on the input of the k-th residual unit to obtain its convolution result can be achieved as follows: The k-th residual unit, through multiple cascaded residual units, performs the following processing: Dilated convolution is applied to the input of the k-th residual unit to obtain its convolution result. That is, the residual features are dilated using the dilated convolution operator included in the first residual unit to obtain its dilated convolution result. Similarly, the j-th residual unit, through multiple cascaded residual units, performs the following processing: Dilated convolution is applied to the residual result output by the (j-1)-th residual unit using the dilated convolution operator included in the j-th residual unit to obtain its dilated convolution result, where j is a sequentially increasing positive integer, 1 < j ≤ J, and J is the number of residual units.

[0219] Following the above embodiments, each residual unit includes not only a dilated convolution operator but also at least one causal convolution operator. After performing convolution processing on the input of the k-th residual unit to obtain the convolution result of the k-th residual unit, causal convolution processing is performed on the obtained dilated convolution result through the at least one causal convolution operator included in the k-th residual unit, and the resulting causal convolution result is used as the convolution result of the k-th residual unit. That is, causal convolution processing is performed on the dilated convolution result of the first residual unit through the at least one causal convolution operator included in the first residual unit, and the resulting causal convolution result is used as the convolution result output by the first residual unit. The dilated convolution operator included in the j-th residual unit is used to perform dilated convolution processing on the residual result output by the (j-1)-th residual unit to obtain the dilated convolution result of the j-th residual unit. Then, the dilated convolution result of the j-th residual unit is performed causally convolution processing on the dilated convolution result of the j-th residual unit using at least one causal convolution operator included in the j-th residual unit. The causal convolution result of the j-th residual unit is then used as the convolution result of the j-th residual unit.

[0220] In some embodiments, when applying grouped convolution to the dilated convolution operator included in the residual unit, dilated convolution processing of low-frequency features can be achieved by: grouping the input channels of the residual features to obtain multiple groups, wherein each group includes a first element corresponding to at least two channels in the residual features; and performing dilated convolution processing on the first element within each group. When applying grouped convolution to the causal convolution operator included in the residual unit, causal convolution processing of the obtained dilated convolution result can be achieved by: grouping the input channels of the dilated convolution result to obtain multiple groups, wherein each group includes a second element corresponding to at least two channels in the dilated convolution result; and performing causal convolution processing on the second element within each group.

[0221] In some embodiments, the fourth neural network for audio decoding includes multiple cascaded decoding blocks, each decoding block including a feature decoding block and at least one residual unit; step 203111 can be implemented by step 203111A, and step 203112 can be implemented by step 203112A: in step 203111A, the low-frequency features are processed by the feature decoding blocks in the multiple cascaded decoding blocks to obtain residual features; correspondingly, in step 203112A, the residual features are processed by at least one residual unit in the multiple cascaded decoding blocks to obtain low-frequency coded feature estimates.

[0222] In some embodiments, step 203111A can be implemented as follows: Low-frequency features are processed by feature decoding of the feature decoding block in the first decoding block of a plurality of cascaded decoding blocks, and the decoding result output by the feature decoding block in the first decoding block is output to at least one residual unit in the first decoding block; the residual result output by at least one residual unit in the (i-1)th decoding block is processed by feature decoding of the feature decoding block in the i-th decoding block of a plurality of cascaded decoding blocks, and the decoding result output by the feature decoding block in the i-th decoding block is output to at least one residual unit in the i-th decoding block; the decoding result output by the feature decoding block in the last decoding block is used as the residual feature; where i is a positive integer that increments sequentially, 1 < i ≤ I, and I is the number of decoding blocks. The decoding result output by the feature decoding block in the first decoding block is obtained by performing the following processing on the feature decoding block in the first decoding block of multiple cascaded decoding blocks: convolving the low-frequency features to obtain convolutional features, wherein the number of channels of the convolutional features is less than the number of channels of the low-frequency features; and upsampling the convolutional features to obtain the decoding result output by the feature decoding block in the first decoding block. Similarly, the decoding result output by the feature decoding block in the i-th decoding block is obtained by performing the following processing on the feature decoding block in the i-th decoding block: convolving the residual results output by at least one residual unit in the (i-1)-th decoding block to obtain convolutional features, wherein the number of channels of the convolutional features is less than the number of channels of the residual results output by at least one residual unit; and upsampling the convolutional features to obtain the decoding result output by the feature decoding block in the i-th decoding block.

[0223] As an example, the fourth neural network includes four decoding blocks, each of which includes four or five residual units.

[0224] In some embodiments, step 203112A can be implemented by performing residual processing on the residual features through at least one residual unit in the last decoding block of a plurality of cascaded decoding blocks to obtain low-frequency coding feature estimates.

[0225] In step 2032, the high-frequency feature estimates included in the coding features are subjected to high-frequency reconstruction processing to obtain the high-frequency subband signal estimates corresponding to the audio bitstream.

[0226] It should be noted that feature reconstruction at the decoding end is the reverse process of extraction at the encoding end, and high-frequency reconstruction at the decoding end is the reverse process of high-frequency analysis at the encoding end.

[0227] In some embodiments, high-frequency reconstruction processing is performed on the high-frequency feature estimates included in the coding features to obtain the high-frequency sub-band signal estimates corresponding to the audio bitstream. This can be achieved by calling a sixth neural network to perform feature reconstruction processing on the high-frequency feature estimates included in the coding features to obtain the high-frequency sub-band signal estimates corresponding to the audio bitstream. The sixth neural network has a similar structure to the third neural network.

[0228] In some embodiments, step 2032 can be implemented through steps 20321-20324:

[0229] In step 20321, the sample points of the first half and the sample points of the second half of the low-frequency subband signal estimation are subjected to frequency domain transformation to obtain the first transformation coefficients corresponding to the sample points of the first half and the second transformation coefficients corresponding to the sample points of the second half.

[0230] It should be noted that the frequency domain transformation methods in the embodiments of this application include Modified Discrete Cosine Transform (MDCT), Discrete Cosine Transform (DCT), Fast Fourier Transform (FFT), etc., and the embodiments of this application are not limited to frequency domain transformation methods.

[0231] In step 20322, based on the first transformation coefficients, the high-frequency feature estimate is subjected to inverse processing of bandwidth expansion to obtain the first high-frequency sub-band signal estimate.

[0232] In some embodiments, step 20322 can be implemented as follows: performing spectral copying processing on the latter half of the first transform coefficients to obtain the first reference transform coefficients of the reference high-frequency subband signal; performing gain processing on the first reference transform coefficients based on the first half of the subband spectrum envelope corresponding to the high-frequency feature estimate to obtain the first reference transform coefficients after gain; and performing inverse frequency domain transform processing on the first reference transform coefficients after gain to obtain the estimate of the first high-frequency subband signal.

[0233] In some embodiments, the first reference transform coefficients are amplified based on the first half of the sub-band spectrum envelope corresponding to the high-frequency feature estimate to obtain the amplified first reference transform coefficients. This includes: dividing the first reference transform coefficients of the reference high-frequency sub-band signal into multiple first sub-bands based on the first half of the sub-band spectrum envelope; performing the following processing on any first sub-band among the multiple first sub-bands: determining the first average energy corresponding to the first sub-band in the first half of the sub-band spectrum envelope, and determining the second average energy corresponding to the first sub-band; determining the gain factor based on the ratio of the first average energy to the second average energy; and multiplying the gain factor by each first reference transform coefficient included in the first sub-band to obtain the amplified first reference transform coefficients.

[0234] As an example, first estimate the low-frequency subband signal x′ generated at the decoding end. LB (n) also performs two MDCT transformations of 320 points each to generate two sets of MDCT coefficients of 160 points each (i.e., the first transformation coefficients corresponding to the sample points in the first half and the second transformation coefficients corresponding to the sample points in the second half).

[0235] Then, for the MDCT coefficients of the first group of 160 points (i.e., the first transform coefficients corresponding to the sample points in the first half), x′ LB (n) The generated 160-point MDCT coefficients are copied to generate the MDCT coefficients for the high-frequency portion. Based on the fundamental characteristics of speech signals, the low-frequency portion has more harmonics, while the high-frequency portion has fewer. Therefore, to avoid simple copying that results in the artificially generated high-frequency MDCT spectrum containing too many harmonics, the last 80 points of the 160-point MDCT coefficients dependent on the low-frequency subband are used as a master, and the spectrum is copied twice to generate reference values ​​for the MDCT coefficients of the 160-point high-frequency subband signal (i.e., the first reference transform coefficients of the high-frequency subband signal).

[0236] Next, the previously obtained 8 sub-band spectral envelopes (i.e., the 8 sub-band spectral envelopes obtained by querying the quantization table, i.e., the sub-band spectral envelopes corresponding to the high-frequency features) are called. These 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands. The reference values ​​of the MDCT coefficients of the generated 160-point reference high-frequency sub-band signal are divided into 8 reference high-frequency sub-bands (i.e., the first reference transform coefficients of the reference high-frequency sub-band signal are divided into multiple first sub-bands). In a segmented manner, based on a high-frequency sub-band and the corresponding reference high-frequency sub-band, gain control (multiplication in the frequency domain) is performed on the reference values ​​of the MDCT coefficients of the generated 320-point reference high-frequency sub-band signal. For example, the gain factor is calculated based on the average energy of the high-frequency sub-band (i.e., the first average energy) and the average energy of the corresponding reference high-frequency sub-band (the second average energy). The MDCT coefficients corresponding to each point in the corresponding reference high-frequency sub-band are multiplied by the gain factor to ensure that the energy of the decoded virtual generated high-frequency MDCT coefficients is close to the energy of the original coefficients at the encoding end.

[0237] For example, assuming the average energy of the reference high-frequency subband (i.e., the first subband divided by the reference values ​​of the 160-point MDCT coefficients of the generated high-frequency signal) is YL, and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectrum envelope decoded from the code stream) is Y_H, then a gain factor a = sqrt(Y_H / YL) is calculated, where sqrt() represents the square root calculation function, used to calculate the square root of (Y_H / YL). With the gain factor a, the MDCT coefficients of each point in the reference high-frequency subband are directly multiplied by a. The average energy of the MDCT coefficients (virtually generated) after gain control is very close to the original energy at the encoding end.

[0238] Finally, the MDCT inverse transform is invoked to generate the estimated value of the first sub-frame of the high-frequency sub-band signal (i.e., the estimated value of the first high-frequency sub-band signal).

[0239] In step 20323, based on the second transformation coefficients, the high-frequency feature estimate is subjected to inverse processing of bandwidth expansion to obtain the second high-frequency sub-band signal estimate.

[0240] Here, step 20323 is similar to step 20322, except that step 20323 processes the second transformation coefficient, while step 20322 processes the first transformation coefficient.

[0241] In some embodiments, step 20323 can be implemented as follows: performing spectral copying processing on the latter half of the transformation coefficients in the second transformation coefficients to obtain the second reference transformation coefficients of the reference high-frequency subband signal; performing gain processing on the second reference transformation coefficients based on the latter half of the subband spectrum envelope corresponding to the high-frequency feature estimate to obtain the gained second reference transformation coefficients; and performing inverse frequency domain transformation processing on the gained second reference transformation coefficients to obtain the second high-frequency subband signal estimate.

[0242] In some embodiments, the second reference transform coefficients are amplified based on the latter half of the sub-band spectral envelope corresponding to the high-frequency feature estimate to obtain amplified second reference transform coefficients. This includes: dividing the second reference transform coefficients of the reference high-frequency sub-band signal into multiple second sub-bands based on the latter half of the sub-band spectral envelope; performing the following processing on any second sub-band among the multiple second sub-bands: determining the first average energy corresponding to the second sub-band in the latter half of the sub-band spectral envelope, and determining the second average energy corresponding to the second sub-band; determining a gain factor based on the ratio of the first average energy to the second average energy; and multiplying the gain factor by each second reference transform coefficient included in the second sub-band to obtain amplified second reference transform coefficients.

[0243] In step 20324, the estimated value of the high-frequency sub-band signal corresponding to the audio bitstream is obtained by combining the estimated value of the first high-frequency sub-band signal and the estimated value of the second high-frequency sub-band signal.

[0244] In step 2033, the estimated values ​​of the low-frequency subband signal and the estimated values ​​of the high-frequency subband signal are combined to obtain the synthesized audio signal corresponding to the audio bitstream.

[0245] For example, subband synthesis is the reverse process of subband decomposition. The decoder performs subband synthesis on the low-frequency subband signal and the high-frequency subband signal to recover the synthesized audio signal, which is the recovered reconstructed signal.

[0246] In some embodiments, subband synthesis processing is performed on the low-frequency subband signal and the high-frequency subband signal to obtain a synthesized audio signal, including: upsampling the low-frequency subband signal to obtain a low-pass filtered signal; upsampling the high-frequency subband signal to obtain a high-frequency filtered signal; and filtering and synthesizing the low-pass filtered signal and the high-frequency filtered signal to obtain a synthesized audio signal.

[0247] For example, after acquiring the low-frequency subband signal and the high-frequency subband signal, the low-frequency subband signal and the high-frequency subband signal are synthesized by a QMF synthesis filter to recover the synthesized audio signal.

[0248] In some embodiments, the frame header further includes at least one channel bit, which is used to indicate whether a mono decoding method or a stereo decoding method is used to decode the audio stream. When the channel bit in the frame header included in the audio stream encapsulation is a first value, it indicates that a stereo decoding method is needed to decode the audio stream, that is, the synthesized stereo signal is reconstructed using the decoded synthesized audio signal and the stereo feature vector in the audio stream encapsulation; when the channel bit in the frame header included in the audio stream encapsulation is a second value, it indicates that a mono decoding method is needed to decode the audio stream, that is, the decoded synthesized audio signal is used as a mono synthesized audio signal.

[0249] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0250] The embodiments of this application can be applied to various audio scenarios, such as voice calls and instant messaging. The following explanation uses voice calls as an example:

[0251] In related technologies, the principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples one by one; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0252] The above encoding principles all originate from speech signal modeling, i.e., signal processing-based compression methods. To improve encoding quality while maintaining speech encoding efficiency compared to signal processing-based compression methods, this application provides a multi-mode, multi-rate speech encoding method (i.e., an audio encoding method and an audio decoding method). Based on the characteristics of the audio signal, for important parts (low-frequency sub-band signals), processing using neural network (NN) technology yields feature vectors with lower dimensions than the input low-frequency sub-band signals. The neural network employs a "block-based" operation, which reduces algorithm complexity and improves encoding performance. For low-frequency feature vectors (i.e., the aforementioned low-frequency features), quantization and encoding can be performed using code tables with different quantization accuracies based on the same low-frequency feature vector, achieving multi-rate encoding and decoding effects. For high-frequency feature vectors (i.e., the aforementioned high-frequency features), multi-level quantization and encoding are used to achieve multi-rate encoding and decoding effects.

[0253] The embodiments of this application can be applied to, for example... Figure 6CThe voice communication link shown takes a Voice over Internet Protocol (VoIP) conferencing system as an example. The voice encoding and decoding technologies involved in this application are deployed in the encoding and decoding parts to solve the basic function of voice compression. The encoder is deployed on the uplink client 601, and the decoder is deployed on the downlink client 602. Voice is acquired through the uplink client and pre-processed, enhanced, and encoded. The encoded bitstream is transmitted to the downlink client 602 via the network. The downlink client 602 performs decoding and enhancement processing to play back the decoded voice.

[0254] To ensure forward compatibility (i.e., compatibility between the new encoder and existing encoders), a transcoder needs to be deployed in the system's backend (i.e., the server) to address the interoperability issue between the new and existing encoders. For example, if the sending end (uplink client) is a new NN encoder, and the receiving end (downlink client) is a Public Switched Telephone Network (PSTN) (G.722), the NN decoder needs to be executed in the backend to generate the voice signal. Then, the G.722 encoder needs to be called to generate a specific bitstream to achieve the transcoding function, allowing the receiving end to correctly decode based on this specific bitstream.

[0255] Before introducing the multi-mode multi-rate speech coding method provided in the embodiments of this application, we will first introduce the QMF filter bank, dilated convolutional network, and bandwidth extension.

[0256] A QMF filter bank is a filter pair that includes analysis and synthesis. For a QMF analysis filter, an input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, representing the QMF low-pass signal and the QMF high-pass signal, respectively. For example... Figure 8 The spectral responses of the low-pass component HLow(z) and high-pass component HHigh(z) of the QMF filter are shown. Based on the relevant theoretical knowledge of QMF filter banks, the correlation between the coefficients of the low-pass and high-pass filters can be easily described, as shown in Equation (1):

[0257] h High (k)=-1 k h Low (k) (1)

[0258] Among them, h Low (k) represents the coefficients of the low-pass filter, h High (k) represents the coefficient of the high-pass filter.

[0259] Similarly, based on the relevant QMF theory, the QMF synthesis filter bank can be described based on the QMF analysis filter bank HLow(z) and HHigh(z), as shown in Equation (2).

[0260] G Low (z)=H Low (z)

[0261] G High (z)=(-1)*H High (z) (2)

[0262] Among them, G Low (z) represents the recovered low-pass signal, G High (z) represents the recovered high-pass signal.

[0263] The low-pass and high-pass signals recovered at the decoding end are synthesized by a QMF synthesis filter bank, which can recover the reconstructed signal (i.e., the synthesized signal) corresponding to the sampling rate Fs of the input signal.

[0264] See Figure 9A and Figure 9B , Figure 9A This is a schematic diagram of a common convolutional (e.g., causal convolutional) network provided in an embodiment of this application. Figure 9B This is a schematic diagram of a dilated convolutional network provided in an embodiment of this application. Compared to ordinary convolutional networks, dilated convolution can increase the receptive field while maintaining the same feature map size, and can also avoid errors caused by upsampling and downsampling. Although Figure 9A and Figure 9B The kernel sizes shown are all 3×3; however, Figure 9A The receptive field of the ordinary convolution shown is only 3 (901), while Figure 9B The dilated convolution shown achieves a receptive field of 902, which is 5. That is, for a 3×3 convolution kernel, Figure 9A The receptive field of the ordinary convolution shown is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1; while Figure 9B The receptive field of the dilated convolution shown is 5, and the dilation rate is 2.

[0265] Convolution kernels can also be used in similar... Figure 9A or Figure 9B The movement occurs on a plane, which involves the concept of stride rate. For example, if the convolution kernel moves 1 unit at a time, the corresponding stride rate is 1.

[0266] In addition, there is the concept of convolution channel count, which refers to the number of parameters corresponding to the convolution kernels used in convolution analysis. Theoretically, the more channels there are, the more comprehensive the signal analysis and the higher the accuracy; however, the more channels there are, the higher the complexity. For example, a 1×320 tensor can be used with a 24-channel convolution operation, and the output will be a 24×320 tensor.

[0267] It should be noted that the size of the dilated convolution kernel (e.g., for speech signals, the kernel size can be set to 1×3), dilation rate, shift rate, and number of channels can be defined according to actual application needs. This application does not impose specific limitations on these aspects.

[0268] like Figure 10 The diagram shown illustrates bandwidth expansion (or bandwidth duplication). First, the broadband signal is reconstructed; then, it is copied onto the ultra-wideband signal; finally, shaping is performed based on the ultra-wideband envelope. Figure 10 The frequency domain implementation scheme shown specifically includes:

[0269] 1) Implement a core layer encoding at a low sampling rate; 2) Copy the spectrum of the low-frequency portion to the high-frequency portion; 3) Perform gain control on the copied high-frequency spectrum based on pre-recorded boundary information (describing the energy correlation between high and low frequencies, etc.). A sampling rate of only 1-2 kbps is sufficient to achieve the effect of doubling the sampling rate.

[0270] The low-complexity, low-bit-rate neural network speech compression method provided in this application embodiment will be described below in conjunction with the high-frequency and low-frequency components of the mono signal.

[0271] The following is combined Figure 7 This application describes the multi-mode, multi-rate speech coding method provided in its embodiments:

[0272] The following processing is performed on the encoding end:

[0273] For the input audio signal x(n) of the nth frame, use an analysis filter to decompose it into a low-frequency subband signal X. LB (n) and high-frequency subband signal X HB (n).

[0274] For low-frequency subband signal X LB (n), call the second NN to obtain the low-dimensional feature vector F. LB (n), i.e., low-frequency features, feature vector F LB The dimension of (n) is smaller than the dimension of the low-frequency subband signal to reduce the amount of data. For example, for each frame X LB (n), calling the neural network (encoding part) to generate a lower-dimensional feature vector F. LB(n). This application does not limit the embodiments to other NN structures, such as autoencoders, fully connected (FC) networks, long short-term memory (LSTM) networks, convolutional neural networks (CNN) + LSTM, etc. Among these, the use of a "block-like" operation within the neural network can reduce algorithm complexity and improve encoding performance. For F... LB (n) uses code tables with different quantization precisions for quantization and encoding to achieve multi-rate encoding and decoding effects.

[0275] For high-frequency subband signal X HB (n), considering that high frequencies are not as important to quality as low frequencies, the high-frequency subband signal X HB (n) Other methods can be used to extract the feature vector F. HB (n). For example, bandwidth extension techniques based on speech signal analysis can generate high-frequency subband signals using only a few bits; they can also use the same NN structure as the low-frequency subband signals or a more streamlined network (e.g., the output feature vector is more powerful than the low-frequency feature vector F). LB (n) is smaller). For high-frequency eigenvectors F HB (n) employs multi-level quantization and encoding to achieve multi-rate encoding and decoding effects.

[0276] The eigenvectors corresponding to the sub-band signals (i.e., F) LB (n) and F HB (n) is subjected to vector quantization or scalar quantization, and the quantized value is entropy encoded. The resulting bitstream (low-frequency bitstream and high-frequency bitstream) is then transmitted to the decoding end.

[0277] The following processing is performed on the decoding end:

[0278] The received bitstream (including low-frequency and high-frequency bitstreams) is decoded at the decoding end to obtain the estimated value F′ of the low-frequency feature vector. LB (n) and the estimated value F′ of the high-frequency eigenvectors HB (n).

[0279] For the low-frequency component, the estimated value F′ is based on the low-frequency eigenvector. LB (n) Call the third NN to obtain the estimated value x′ of the low-frequency subband signal. LB (n). The neural network internally employs a "block-based" operation to reduce algorithm complexity and improve encoding performance. Based on the actual codeword length in the bitstream, code tables with different quantization precisions can be used to generate F′ with varying precisions. LB (n) and x′LB (n) enables multi-rate encoding and decoding.

[0280] For the high-frequency component, the estimated value F′ is based on the high-frequency eigenvectors. HB (n) Call high-frequency reconstruction to generate an estimate x′ of the high-frequency sub-band signal. HB (n). Based on the actual codeword length in the bitstream, code tables with different quantization precisions can be used to generate F′ with different precisions. HB (n) and x′ HB (n) enables multi-rate encoding and decoding.

[0281] Finally, QMF synthesis filtering is invoked to generate the reconstructed synthesized speech signal x′(n).

[0282] The multi-mode, multi-rate speech coding method provided in the embodiments of this application will be described in detail below.

[0283] In some embodiments, a speech signal with a sampling rate of Fs = 32000Hz is taken as an example (it should be noted that the method provided in this application embodiment is also applicable to other sampling rate scenarios, including but not limited to: 8000Hz, 32000Hz, 48000Hz). Meanwhile, assuming the frame length is set to 20ms, for Fs = 32000Hz, it is equivalent to each frame containing 640 sample points.

[0284] The following is for reference. Figure 7 The flowchart shown describes the encoding and decoding ends in detail.

[0285] The encoding process for the low-frequency and high-frequency components of a mono signal is as follows:

[0286] For an audio signal (mono signal) with a sampling rate of Fs = 32000Hz, the input signal of the nth frame includes 640 sample points, denoted as input signal x(n), which is the mono input signal.

[0287] Step 11: Call the QMF analysis filter to decompose the signal.

[0288] By applying a QMF analysis filter (2-channel QMF) and downsampling, two subband signals can be obtained: the low-frequency subband signal X. LB (n) and high-frequency subband signal X HB (n). Low-frequency subband signal X LB (n) and high-frequency subband signal X HB The effective bandwidths of (n) are 0-8kHz and 8-16kHz, respectively, and the low-frequency subband signal X LB (n) and high-frequency subband signal X HB The number of sample points for (n) is 320.

[0289] Step 12: Based on the low-frequency subband signal, call the second NN.

[0290] Based on low-frequency subband signal x LB (n), call the second NN to generate a lower-dimensional feature vector F. LB (n). It should be noted that X LB The dimension of (n) is 320, F LB The dimension of (n) is 56. From the perspective of data volume, the second NN plays a role in "dimensionality reduction," achieving data compression. However, the embodiments in this application are not limited to F. LB The dimension of (n) can also be other values ​​smaller than X. LB (n) dimension.

[0291] See Figure 11 The network structure diagram of the second neural network is shown below. The process of data compression performed by the second neural network is explained in detail below:

[0292] First, calling a 16-channel causal convolution expands the input tensor (i.e., vector) into a 16×320 tensor.

[0293] Then, the 16×320 tensor is preprocessed. For example, after performing a convolution operation on the 16×320 tensor, a pooling operation with a factor of 2 is performed, and the activation function can be PReLU, to generate a 16×160 tensor.

[0294] Next, four encoding blocks with different downsampling factors are concatenated. Each encoding block contains a residual block, a convolutional layer, and a pooling layer. Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer doubles the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The pooling layer is a pooling operation with a down factor to perform downsampling and achieve data compression. Here, the down factors of the four encoding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four encoding blocks are set to 32, 64, 128, and 256, respectively. After processing by the four encoding blocks, the input 16×160 tensor is transformed into tensors of 32×80, 64×20, 128×5, and 256×1, respectively. The number of coding blocks in this application is not limited, and can be any positive integer such as 2, 3, 4, or 5. Furthermore, the number of residual units in a coding block is not limited, and can be any positive integer such as 2, 3, 4, 5, or 6. The number of residual units in multiple coding blocks can be the same or different; for example, one coding block may contain 4 residual units, and another coding block may contain 5 residual units.

[0295] This section further introduces the Residual Unit. A Residual Unit is a module in a deep neural network that introduces cross-layer connections, making the neural network easier to optimize during training and avoiding problems such as vanishing or exploding gradients. Its core idea is to perform residual learning on the input within the module, that is, to bypass a portion of the layers and directly pass the input information to the output through a direct path, allowing the network to better utilize shallow feature information during the learning process. Figure 12A This diagram illustrates the residual block structure used in the coding block of the third neural network. This residual block comprises four dilated convolution-based residual units, each containing a dilated convolution block with a specified dilation rate (e.g., dilation rate = 3). In this embodiment, five dilated convolution blocks with progressively increasing dilation rates are used, which is equivalent to using different receptive fields to extract features from the input at different resolutions, allowing for a more comprehensive analysis of the data. After residual processing through the five dilated convolution blocks with specified dilation rates, the input is added to the input from the jump connection to obtain the output of the residual block, which is then output to the convolutional layer connected to the residual block.

[0296] Here, further... Figure 12A Describing it using any residual cell, such as Figure 12B As shown. For any residual unit, it contains at least one dilated convolution with a specified dilation rate (for expanding the receptive field), which can be activated using PReLU; in addition, one or more causal convolutions (for extracting local information) can be cascaded, which can also be activated using PReLU. The kernel size of the dilated convolution with the specified dilation rate can be 3, 5, 7, 9, etc., and the kernel size of the causal convolution can be 1, 3, etc. This application and this embodiment do not limit the kernel size of the dilated convolution with the specified dilation rate or the causal convolution. In addition, the causal convolution or dilated convolution in the embodiments of this application can also be implemented by other convolution units with similar or equivalent functions.

[0297] Furthermore, for residual units, a grouped convolution algorithm is introduced to reduce algorithm complexity. Grouped convolution divides the input channels into multiple groups for convolution operations, associating input and output channels only within each group. Here, we assume there are 16 input channels and 32 output channels. If the number of groups is 1, each input channel is associated with all 32 output channels. If the number of groups is 2, the 16 input channels are first divided into two groups: 0-7 and 8-15. Within each group, the input channel is associated with the output channel within that group. For example, input channels 0-7 in the first group are associated with output channels 0-15, and input channels 8-15 in the second group are associated with output channels 16-31. For instance, the 0th output channel is associated only with input channels 0-7 and not with input channels 8-15, and the 25th output channel is associated only with input channels 8-15 and not with input channels 0-7. This comparison shows that introducing grouped convolution can avoid any correlation between input channels and all output channels, reducing the number of connections and thus lowering complexity. However, since a larger number of groups reduces the correlation between input and output channels, it can also affect encoding performance; therefore, a larger number of groups is not always better. In this embodiment, the dilated convolutions contained in the five residual blocks corresponding to the four coding blocks can use different group number configurations, as shown in Table 1.

[0298] Table 1. Configuration of the number of blocks used by residual units in different coding blocks

[0299] Encoded block (Down_factor=2) 2 Encoded block (Down_factor=4) 4 Encoded block (Down_factor=4) 4 Encoded block (Down_factor=5) 4

[0300] Finally, by performing a causal convolution similar to preprocessing on the 256×1 tensor, a 56-dimensional feature vector F can be output. LB (n). Based on the second neural network calculation, the 56-dimensional feature vector F... LB Each value in (n) is between [-1, 1].

[0301] Step 13: For the high-frequency subband signal X HB (n) Perform high-frequency analysis.

[0302] The purpose of high-frequency analysis is to extract the high-frequency sub-band signal x. HB Using the key information of (n), a lower-dimensional feature vector F is generated. HB (n). The embodiments of this application are not limited to F. HB The dimension of (n) can also be other values ​​smaller than X. LB The dimension of (n), but F HB The dimension of (n) is smaller than that of F. LB (n) dimension.

[0303] In some embodiments, referring to step 12, another structure similar to the second NN can be introduced to generate low-dimensional feature vectors. Compared to low-frequency subband signals, high-frequency subband signals are less important for quality; therefore, the NN structure for high-frequency subband signals does not need to be as complex as the second NN. The NN structure for high-frequency subband signals is similar to the structure of the second NN, but compared to the second NN, the NN structure for high-frequency subband signals significantly reduces the number of channels.

[0304] This application proposes another method for compressing high-frequency subband signals, namely bandwidth spreading (recovering wideband audio signals from bandwidth-limited narrowband audio signals). The application of bandwidth spreading in this application is described in detail below:

[0305] For a high-frequency subband signal X comprising 320 points (i.e., sample points) HB (n), representing the 320-point high-frequency subband signal X HB (n) is divided into two subframes (the first subframe and the second subframe), each containing 160 points. Compared to a 640-point MDCT transform with 50% overlap, using a 320-point MDCT transform with 50% overlap across frames reduces algorithm latency by 10ms. Therefore, for every 20ms frame, two 320-point MDCT transforms can be performed to extract features.

[0306] For any subframe, which includes a high-frequency subband signal of 160 points, the Modified Discrete Cosine Transform (MDCT) is invoked to generate 160-point MDCT coefficients. Specifically, if there is a 50% overlap, the first subframe of the (n+1)th frame can be merged (stitched) with the second subframe of the nth frame, and the MDCT of 320 points can be calculated to obtain the 160-point MDCT coefficients of the first subframe of the (n+1)th frame; the second subframe of the (n+1)th frame can be merged with the first subframe of the (n+1)th frame, and the MDCT of 320 points can be calculated to obtain the 160-point MDCT coefficients of the second subframe of the (n+1)th frame.

[0307] For the 160-point MDCT coefficients of any one of the two subframes mentioned above, the same processing method is used as follows: divide the 160-point MDCT coefficients into N sub-bands. Here, a sub-band is formed by grouping multiple adjacent MDCT coefficients together. The 160-point MDCT coefficients can be divided into 4 sub-bands. For example, the 160 points can be evenly distributed, that is, each sub-band includes the same number of points. Of course, the embodiments of this application cannot divide the 160 points non-uniformly. For example, a sub-band with lower frequencies may include fewer MDCT coefficients (higher frequency resolution), while a sub-band with higher frequencies may include more MDCT coefficients (lower frequency resolution).

[0308] According to the Nyquist sampling theorem (to recover the original signal without distortion from the sampled signal, the sampling frequency should be greater than twice the highest frequency of the original signal; when the sampling frequency is less than twice the highest frequency of the spectrum, the signal spectrum exhibits aliasing; when the sampling frequency is greater than twice the highest frequency of the spectrum, the signal spectrum does not exhibit aliasing), the MDCT coefficients of the above 320 points represent the spectrum from 8 to 16 kHz. However, ultra-wideband voice communication does not necessarily require a spectrum up to 16 kHz. For example, if the spectrum is set to 14 kHz, only the MDCT coefficients of the first 240 points need to be considered, and correspondingly, the number of sub-bands can be controlled to 6.

[0309] For each subband, calculate the average energy of all MDCT coefficients in the current subband as the subband spectral envelope (the spectral envelope is a smooth curve passing through the main peak points of the spectrum). For example, if the MDCT coefficients included in the current subband are x(n), n = 1, 2, ..., 40, then the average energy Y = ((x(1)) / (x(n ... 2 +x(2) 2 +…+x(40) 2 () / 40). For the case where 160 MDCT coefficients are divided into 4 sub-bands, we can obtain 4 sub-band spectral envelopes. These 4 sub-band spectral envelopes are the sub-band spectral envelopes corresponding to the sub-bands. Here, i = 1 represents the first subframe, and i = 2 represents the second subframe. This is combined with the sub-band spectral envelopes corresponding to the two sub-bands. Obtain the eigenvector F of the high-frequency subband signal HB (n).

[0310] In summary, either of the two methods described above (NN structure and band extension) can output a 4-dimensional feature vector from a 160-dimensional high-frequency subband signal. Therefore, for each 20ms frame, only a minimum of 8 dimensions of data are needed to represent high-frequency information, significantly improving coding efficiency.

[0311] Step 14: Quantization encoding.

[0312] For the eigenvector F of the low-frequency subband signal LB (n) and the eigenvector F of the high-frequency subband signal HB (n) can be quantized using scalar quantization (each component is quantized individually) and entropy encoding methods. Furthermore, the embodiments of this application do not limit the combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy encoding techniques.

[0313] This refers to the low-frequency feature vector F. LB The quantization encoding of (n) will be explained. As described above, after processing the low-frequency subband signal through the second NN, a 56-dimensional feature vector F is obtained. LB (n). This application provides a method based on scalar quantization and entropy coding, including: 1) for F LB For each dimension in (n), the interval [-1, 1] is divided into 11 equal parts to form a codebook containing 11 elements, and the value of each dimension is quantized to one of the 11 elements; 2) According to Shannon's entropy theorem, for a codebook containing 11 uniformly distributed elements, the entropy (average bits) is Therefore, the average number of bits corresponding to the low-frequency subband signal in each frame is 193.76 bits; 3) For the 20ms framing method, there are 50 frames per second, so the average bit rate is 9.69kbps. According to entropy coding theory, probability distribution statistics can be performed on each of the above dimensions to generate 56 code tables. Generally, each dimension is non-uniformly distributed, so the actual bit rate is around 9.69kbps, or less than 9.69kbps.

[0314] In addition, in order to achieve the low-frequency feature vector F LB The goal of multi-rate encoding and decoding (n) can be implemented in the following ways:

[0315] The code table for the above implementation, which contains 11 elements in each dimension, is named Low Frequency Code Table-1, with a reference code rate of 9.69 kbps.

[0316] Similarly, for each dimension, the interval [-1, 1] is divided into 9 equal parts, with an entropy of 3.17 and an average code rate of 3.17 * 56 * 50 / 1000 = 8.88 kbps. This is named Low Frequency Code Table-2, with a reference code rate of 8.88 kbps. Thus, there are at least two code rate options, each corresponding to different qualities; multi-rate coding is achieved.

[0317] Similarly, for each dimension, the interval [-1, 1] is divided into 7 equal parts with an average code rate of 7.86 kbps, and this is named Low Frequency Code Table-3; the interval [-1, 1] is divided into 5 equal parts with an average code rate of 6.50 kbps, and this is named Low Frequency Code Table-4.

[0318] In this way, by configuring the quantization precision, the original 56-dimensional feature vector obtained from each frame of data can achieve an encoding mode for low-frequency features at least four bit rates. It should be noted that the embodiments of this application do not limit other multi-bit rate construction methods or the number of code tables.

[0319] For each frame (containing 2 subframes), the eigenvector F of the 8-dimensional high-frequency subband signal HB (n), a specific implementation of its quantization encoding includes: 1) performing scalar quantization on each dimension separately, allocating 5 bits; thus, 40 bits are allocated per frame; 2) for a 20ms framing method, there are 50 frames per second, therefore, the average bitrate is 2kbps. If entropy encoding (including but not limited to Huffman coding or range coding) is performed on the quantized values ​​of each of the above dimensions, the overall bitrate will be less than 2kbps. The above code table is defined as high-frequency code table-1, and the quantized value is defined as the quantized value of the high-frequency feature vector (i.e., the quantized value of the high-frequency feature).

[0320] In addition, in order to achieve high-frequency feature vector F HB The goal of multi-rate encoding and decoding (n) can be further achieved by introducing residual processing, with the following implementation:

[0321] 1) For every 10ms subframe, the four subbands of the subframe are further divided into two subsets evenly.

[0322] For example, the set of envelope values ​​for the four subbands is {e1, e2, e3, e4}.

[0323] 2) For each subset of each subband, calculate the envelope value of that subset.

[0324] For example, the set of envelope values ​​for all subsets is {e 1a e 1b e 2a e 2b e 3a e 3b e 4a e 4b}

[0325] 3) Subtract the corresponding high-frequency feature vector quantization value from the envelope value of each subset to obtain the first high-frequency feature vector residual (referred to as the first residual value).

[0326] For example, the quantization value of the high-frequency feature vector is The residual of the first high-frequency eigenvector is

[0327] 4) The first high-frequency feature residual above is subjected to similar quantization and entropy encoding to obtain the quantized value of the first high-frequency feature vector residual (referred to as the quantized value of the first residual value).

[0328] Here, considering that the dynamic range of the residual value is much lower than that of the original value, only 2 bits need to be allocated when scalar quantizing each residual. Thus, a reference code rate of 2*16*50=1.6kbps is needed (similarly, because it is non-uniformly distributed entropy coding, the actual code rate is generally less than 1.6kbps) to obtain the quantized value of the first high-frequency feature vector residual.

[0329] 5) Subtract the sum of the quantized value of the corresponding high-frequency feature vector and the quantized value of the residual of the first high-frequency feature vector from the envelope value of each subset to obtain the second high-frequency feature vector residual (referred to as the second residual value).

[0330] Similarly, a reference code rate of 1.6 kbps can be used to complete the quantization and entropy coding of the residual of the second high-frequency feature vector.

[0331] Similarly, as more residuals are calculated, it can be ensured that the high-frequency feature vector reconstructed at the decoding end approximates the original high-frequency feature vector infinitely. Furthermore, multi-rate coding effects are achieved based on the presence or absence of high-frequency feature vector residuals, or the number of high-frequency feature vector residuals. In this application, the number of residual coding types (no residual coding, first high-frequency feature vector residual, second high-frequency feature vector residual, third high-frequency feature vector residual, etc.) can be defined according to the bit rate requirements, and no limitation is imposed here.

[0332] Furthermore, this application provides a scheme for extracting additional side information at the encoding end to guide more refined processing at the decoding end, thereby improving audio quality. In principle, the main function of bandwidth expansion is to copy the low-frequency spectrum to a high-frequency spectrum and then perform gain control. However, the high-frequency spectrum is flatter than the low-frequency spectrum and has fewer harmonic components. If it is directly copied and then gain controlled, the artificially generated high frequencies may contain too many harmonic components. Therefore, this application proposes to estimate specific side information at the encoding end and write it into the bitstream. The decoding end determines whether additional processing is needed based on the aforementioned flatness side information. It should be noted that the aforementioned flatness side information only indicates at the decoding end whether additional processing is needed for the current frame. In a special case, no additional processing is needed for the entire speech.

[0333] The above edge information estimation is implemented at the encoding end as follows:

[0334] 1) For each 10 - ms sub - frame, divide the 160 - point MDCT coefficients into two blocks. The division can be uniform, or the MDCT coefficients of the first two sub - bands can be used as the first block, and the MDCT coefficients of the last two sub - bands can be used as the second block.

[0335] 2) For each block, calculate the square of each MDCT coefficient, that is, p(i)=c(i)^2 2 .

[0336] 3) Calculate the arithmetic mean of all MDCT coefficients in the current block: where I represents the number of MDCT coefficients in the current block.

[0337] 4) Calculate the geometric mean of all MDCT coefficients in the current block: where In represents the logarithmic operation, and I represents the number of MDCT coefficients in the current block.

[0338] 5) Calculate the first flatness of the current block Generally, sfpHi is a value in the range of [0, 1].

[0339] 6) Similarly, calculate the second flatness sfp of the MDCT coefficients in the specified low - frequency frequency band Lo .

[0340] 7) When (sfp Hi [i]<sfpL o [i]) or (sfp Hi [i]<SF_THD), Sfp_FLAG = 0; otherwise, Sfp_FLAG = 1. Here, SF_THD represents the set flatness threshold, and Sfp_FLAG represents the flatness side information.

[0341] Here, when Sfp_FLAG = 1, it means that the decoding end needs additional operations to avoid excessive harmonics artificially generated in the high - frequency part. According to the above operations, 4 additional bits are required for each frame, representing the flatness side information of the two blocks in two 10 - ms sub - frames. The embodiments of this application are not limited to the above extraction method, and other analysis methods can also be used to extract the above flatness side information to guide the decoding process at the decoding end.

[0342] After quantization coding, a bit - stream can be generated. According to experiments, in the range of 5 - 10 kbps, high - quality compression can be achieved for a 16 - kHz wide - band signal; in the range of 8 - 15 kbps, high - quality compression can be achieved for a 32 - kHz ultra - wide - band signal.

[0343] In addition, in order to achieve the above - mentioned goal of multi - mode and multi - rate encoding and decoding, a good bit - stream structure needs to be designed. As Figures 13A-13E shown, the implementation form of the bit - stream structure is as follows:

[0344] Each frame of the bitstream typically contains an 8-bit frame header. This 8-bit header includes at least one encoding mode bit, which indicates whether wideband coding (1 bit value = 0) or ultra-wideband coding (1 bit value = 1) is used; that is, there are two encoding modes (wideband coding and ultra-wideband coding). Additionally, it includes 2 encoding rate bits, which indicate which bitrate mode is used in the corresponding wideband or ultra-wideband coding mode. It should be noted that the embodiments of this application are not limited to the positions of the 1-bit encoding mode bit and the 2-bit encoding rate bit in the frame header. For example, the 1-bit encoding mode bit can be located at the first position of the frame header, and the 2-bit encoding rate bit can be located at the last two positions of the frame header.

[0345] like Figure 13A As shown, for wideband coding mode, the frame header is immediately followed by a 56-dimensional low-frequency feature vector index. Because it is entropy coding, the code length varies. As mentioned above, based on the 2-bit code rate mode bits, the low-frequency bitstream quantized and encoded using different low-frequency code tables is stored in the 56-dimensional low-frequency feature vector index part.

[0346] The 2-bit encoding rate bit in wideband coding mode is explained as follows:

[0347] 1)00 indicates that the low-frequency code table -4 is used, with a reference average code rate of 6.5kbps, which is also the lowest code rate in wideband coding mode;

[0348] 2) 01 indicates that the low-frequency code table -3 is used, with a reference average code rate of 7.86kbps;

[0349] 3) 10 indicates that the low-frequency code table -2 is used, with a reference average code rate of 8.88kbps;

[0350] 4)11 indicates the use of low-frequency code table -1, with a reference average code rate of 9.69kbps, which is also the highest code rate in wideband coding mode.

[0351] For ultra-wideband (UWB) coding mode, a 2-bit code rate mode bit is defined based on whether residual coding is used. The 2-bit code rate bit in UWB coding mode is explained as follows:

[0352] 1) 00 (excluding residual coding) indicates that the low-frequency part uses low-frequency code table-4. The code stream encapsulation structure includes quantization coding using low-frequency code table-4, four spectral envelopes of ultra-wideband coding, and 4-bit flatness. The reference average code rate is 6.5+2+0.2=8.7kbps.

[0353] 2) 01 (excluding residual coding) indicates that the low-frequency part uses low-frequency code table-1. The code stream encapsulation structure includes quantization coding using low-frequency code table-1, four spectral envelopes of ultra-wideband coding, and 4-bit flatness. The reference average code rate is 9.69+2+0.2=11.9kbps.

[0354] 3) 10 (including the first residual code) indicates that the low frequency part uses the low frequency code table-1. The code stream encapsulation structure includes quantization encoding using the low frequency code table-1, four spectral envelopes of ultra-wideband encoding, eight first residuals, and four bits of flatness. The reference average code rate is 9.69+2+1.6+0.2=13.5kbps.

[0355] 4) 11 (including first and second residual codes) indicates that the low-frequency portion uses a low-frequency code table-1. The code stream encapsulation structure includes quantization encoding using a low-frequency code table-1, four spectral envelopes encoded with ultra-wideband code, eight first residuals, eight second residuals, and four bits of flatness. The reference average code rate is 9.69 + 2 + 1.6 + 1.6 + 0.2 = 15.0 kbps.

[0356] like Figures 13A-13C Based on the 2-bit code rate mode bits, the relevant bitstreams of the low-frequency feature vector index and the high-frequency feature vector index are immediately appended to the frame header. For example... Figures 13D-13E If the bitstream encapsulation structure contains 4 bits of flatness side information, then the 4 bits of side information can be placed immediately after the 8-dimensional high-dimensional feature vector index.

[0357] In summary, when using an encoder for audio encoding, the user inputs the audio data to be encoded and the correct configuration parameters. For example, when the input audio data is wideband, the sampling rate is 16000Hz; when the input audio data is ultra-wideband, the sampling rate is 32000Hz. For ease of description, we use wideband signals sampled at 16000Hz and ultra-wideband signals sampled at 32000Hz as examples, but the embodiments of this application do not limit other combinations.

[0358] In addition, users need to configure the parameters of the bitrate mode to be used. The encoder can select an appropriate code table according to the configured bitrate mode. Generally, the higher the bitrate mode, the more bits the encoder will use for encoding, resulting in higher quality reconstructed speech. As shown above, this application embodiment provides four bitrate modes for both wideband and ultra-wideband coding modes, but this application embodiment is not limited to four bitrate modes.

[0359] For example, when a user inputs a wideband signal sampled at 16000Hz, the encoder can use a wideband coding mode for encoding. The second neural network (NN) is directly applied to the input wideband signal to extract low-frequency features. These extracted low-frequency features are then quantized and encoded to obtain the wideband signal's bitstream. Conversely, when a user inputs an ultra-wideband signal sampled at 32000Hz, the encoder can use an ultra-wideband coding mode. The input ultra-wideband signal first passes through an analysis filter to separate low-frequency and high-frequency subband signals. The low-frequency subband signal is then directly processed using the second NN to extract low-frequency features F. LB (n), for F LB (n) Quantization encoding is performed to obtain the low-frequency code stream. The high-frequency subband signal undergoes high-frequency analysis to extract high-frequency features F. HB (n), for F HB (n) Quantization encoding is performed to obtain a high-frequency bitstream. Furthermore, based on the bitrate mode input at the encoding end, the encoder can select the corresponding code table for quantization encoding.

[0360] The decoding process for both low-frequency and high-frequency components is as follows:

[0361] Step 21: Quantization and decoding.

[0362] Quantization decoding is the inverse process of quantization encoding. For the received bitstream (including high-frequency and low-frequency bitstreams), entropy decoding is first performed, and then the estimated value F′ of the feature vector of the low-frequency bitstream is obtained by looking up the quantization table. LB The estimated values ​​of (n) and the eigenvectors of the high-frequency code stream, F′ HB (n).

[0363] refer to Figure 13E Here, we take the configuration of ultra-wideband coding mode with a bitrate mode bit of 10 as an example for explanation:

[0364] First, parse the frame header of the bitstream encapsulation. When the encoding mode bit is 1, it indicates that ultra-wideband encoding is used during encoding. When the bit rate mode bit is 10, it indicates that the wideband portion uses low-frequency code table -1 during encoding. After encoding the ultra-wideband portion, the bitstream encapsulation contains an 8-dimensional high-frequency feature vector index and a 16-dimensional first high-frequency feature vector residual index.

[0365] For the broadband portion, the low-frequency bitstream is parsed using entropy encoding and decoding techniques. By using the low-frequency code table-1, the estimated value F′ of the 56-dimensional low-frequency feature vector can be obtained. LB (n).

[0366] For the ultra-wideband portion, firstly, based on a 5-bit code table, 8-dimensional high-frequency feature vector estimates are obtained per frame; then, based on a 2-bit code table, 16-dimensional first high-frequency feature vector residual estimates are obtained per frame. Using the corresponding 8-dimensional high-frequency feature vector estimates (i.e., the high-frequency feature estimates mentioned above) and the 16-dimensional first high-frequency feature vector residual estimates (i.e., the first residual feature estimates mentioned above), the final 16-dimensional high-frequency feature vector estimates (i.e., the final high-frequency feature estimates mentioned above) can be obtained. The i-th value in the high-frequency feature vector estimates is added to the (2i-1)-th and the 2i-th values ​​in the first high-frequency feature vector residual estimates, respectively, to obtain the (2i-1)-th and the 2i-th values ​​in the final high-frequency feature vector estimates. For example, if the high-frequency feature estimate is F′... HB (n) = [e′1, e′2, ..., e′8], the residual estimate of the first high-frequency eigenvector is F′ C (n)=[e′ 1C ,e′ 2C , ..., e′ 16C ], then the final estimate of the high-frequency features is F′ FHB (n)=[e′1+e′ 1C ,e′1+e′ 2C , ..., e′8+e′ 16C ].

[0367] refer to Figure 13D Here, we take the configuration of ultra-wideband coding mode with the code rate mode bit set to 0 or 1 as an example for explanation:

[0368] First, parse the frame header of the bitstream encapsulation. When the encoding mode bit is 1, it indicates that ultra-wideband encoding is used. When the bit rate mode bit is 0 or 1, it indicates that the wideband portion uses a low-frequency code table of -1. The bitstream encapsulation after encoding the ultra-wideband portion contains an 8-dimensional high-frequency feature vector index.

[0369] For the broadband portion, the low-frequency bitstream is parsed using entropy encoding and decoding techniques. Using the low-frequency code table-4, the estimated value F′L of the 56-dimensional low-frequency feature vector can be obtained. B (n).

[0370] For the ultra-wideband portion, based on a 5-bit code table, an 8-dimensional high-frequency feature vector estimate is obtained for each frame. This 8-dimensional estimate is then copied to obtain a final 16-dimensional high-frequency feature vector estimate (i.e., the aforementioned final high-frequency feature estimate). For example, the high-frequency feature estimate is F′. HB (n)=[e′1,e′2,...,e′8],then the final estimate of the high-frequency features is F′ FHB (n)=[e′1, e′1, ..., e′8, e′8].

[0371] Here, we take the configuration of ultra-wideband coding mode with the bitrate mode bit set to 11 as an example for the following explanation:

[0372] First, parse the frame header of the bitstream encapsulation. When the encoding mode bit is 1, it indicates that ultra-wideband encoding is used during encoding. When the bit rate mode bit is 11, it indicates that the wideband part uses low-frequency code table -1 during encoding. After encoding the ultra-wideband part, the bitstream encapsulation includes an 8-dimensional high-frequency feature vector index, a 16-dimensional first high-frequency feature vector residual index, and a 16-dimensional second high-frequency feature vector residual index.

[0373] For the broadband portion, the low-frequency bitstream is parsed using entropy encoding and decoding techniques. By using the low-frequency code table-1, the estimated value F′ of the 56-dimensional low-frequency feature vector can be obtained. LB (n).

[0374] For the ultra-wideband portion, firstly, based on a 5-bit code table, 8-dimensional high-frequency feature vector estimates are obtained for each frame; then, based on a 2-bit code table, 16-dimensional first high-frequency feature vector residual estimates and 16-dimensional second high-frequency feature vector residual estimates are obtained for each frame. Using the corresponding 8-dimensional high-frequency feature vector estimates (i.e., the aforementioned high-frequency feature estimates), the 16-dimensional first high-frequency feature vector residual estimates (i.e., the aforementioned first residual feature estimates), and the 16-dimensional second high-frequency feature vector residual estimates (i.e., the aforementioned second residual feature estimates), the final 16-dimensional high-frequency feature vector estimate (i.e., the aforementioned final high-frequency feature estimate) can be obtained. The i-th value in the high-frequency eigenvector estimate is added to the (2i-1)-th value in the first high-frequency eigenvector residual estimate and the (2i-1)-th value in the second high-frequency eigenvector residual estimate to obtain the (2i-1)-th value in the final high-frequency eigenvector estimate. The i-th value in the high-frequency eigenvector estimate is then added to the (2i-1)-th value in the first high-frequency eigenvector residual estimate and the (2i-1)-th value in the second high-frequency eigenvector residual estimate to obtain the (2i-th)-th value in the final high-frequency eigenvector estimate. For example, if the high-frequency eigenvector estimate is F′... HB (n) = [e′1, e′2, ..., e′8], the residual estimate of the first high-frequency eigenvector is F′ C (n)=[e′ 1C ,e′ 2C , ..., e′ 16C The residual estimate of the second high-frequency eigenvector is F′. B (n)=[e′ 1B ,e′ 2B , ..., e′ 16B ], then the final estimate of the high-frequency features is F′ FHB (n)=[e′1+e′ 1C +e′ 1B ,e′1+e′ 2C +e′ 2B, ..., e′8+e′ 16C +e′ 16B ].

[0375] Step 22: Estimation of F′ based on the feature vector of the low-frequency code stream LB (n), calling the third NN.

[0376] First, the estimated value F′ based on the feature vector of the low-frequency code stream. LB (n), called as follows Figure 14 The third neural network shown generates an estimate x′ for the low-frequency subband signal. LB (n). The third neural network (NN) is similar to the second NN, for example, using causal convolution. Its post-processing structure is similar to the pre-processing structure in the second NN. The decoder block structure is symmetrical to the encoder block. The encoder block first undergoes dilated convolution followed by pooling to achieve downsampling, while the decoder block first undergoes pooling to achieve upsampling, followed by dilated convolution. The specific process of the third NN is as follows:

[0377] First, a causal convolution is invoked to convert the input tensor F′ into a causal convolution. LB (n), a tensor that is expanded from 56×1 to 256×1.

[0378] Next, four decoding blocks with different upsampling factors (Up_factor) are concatenated. Each decoding block contains a convolutional layer, an upsampling module, and a residual block. The convolutional layer halves the number of input channels; the upsampling module contains a specific Up_factor to perform upsampling; and the residual block includes five dilated convolution-based residual units. The Up_factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels for the four decoding blocks are set to 128, 64, 32, and 16, respectively. After processing by the four decoding blocks, the 256×1 tensor is converted into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. Note that the number of decoding blocks in this embodiment is not limited and can be any positive integer such as 2, 3, 4, or 5.

[0379] Here, for upsampling modules that contain a specific Up_factor, a repeat operation can be used to complete the upsampling operation by repeatedly padding. This saves complexity.

[0380] Here, the configuration of the five dilated convolution-based residual units at the decoding end is similar to that at the encoding end, including but not limited to the internal structure of the residual units, convolution kernel size, and dilation rate. The number of groups used in the dilated convolution in the decoding block is configured as shown in Table 2. Here, a higher group size of 2 is used in the decoding block to associate more input and output channels, thereby improving the quality of speech reconstruction.

[0381] Table 2. Configuration of the number of groups used by residual units in different decoding blocks

[0382] Decoding block (Up_factor=5) 4 Decoding block (Up_factor=4) 4 Decoding block (Up_factor=4) 2 Decoding block (Up_factor=2) 2

[0383] Then, the 16×160 tensor output by the concatenated decoder block is post-processed. For example, a factor of 2 Repeat operation is performed on the 16×160 tensor output by the concatenated decoder block to complete upsampling, and then convolution operation is performed and an activation function of PReLU is used to generate a 16×320 tensor.

[0384] Finally, a causal convolution is invoked to convert the input 16×320 tensor into a 1×320 tensor to reconstruct the low-frequency subband signal.

[0385] Step 23: Estimate the eigenvector F′ of the high-frequency subband signal. HB (n) Perform high-frequency reconstruction.

[0386] Similar to high-frequency analysis at the encoding end, the high-frequency reconstruction in this application includes two schemes.

[0387] The first implementation of high-frequency reconstruction corresponds to the first implementation of high-frequency analysis at the encoding end. It is based on the estimation value F′ of the eigenvectors of the high-frequency sub-band signal. HB (n), calling a neural network to generate an estimate x′ of the high-frequency subband signal. HB (n).

[0388] The second implementation of high-frequency reconstruction corresponds to the band extension technique in high-frequency analysis at the encoding end. Based on the 16 MDCT sub-band spectral envelopes decoded from the high-frequency bitstream (each 10ms subframe contains 8 sub-band spectral envelopes), i.e., the final estimated value of the high-frequency feature vector, the following operations are performed:

[0389] First, estimate the low-frequency subband signal x′ generated at the decoding end. LB (n) also performs two MDCT transformations of 320 points each to generate two sets of MDCT coefficients of 160 points each (i.e., the MDCT coefficients of the low-frequency part, including the first set of 160-point MDCT coefficients and the second set of 160-point MDCT coefficients).

[0390] Then, by x′ LB(n) The generated 160-point MDCT coefficients are copied to generate the MDCT coefficients for the high-frequency portion. Based on the fundamental characteristics of speech signals, the low-frequency portion has more harmonics, while the high-frequency portion has fewer. Therefore, to avoid simple copying that results in the artificially generated high-frequency MDCT spectrum containing too many harmonics, the last 80 points of the 160-point MDCT coefficients dependent on the low-frequency subband can be used as a master, and the spectrum is copied twice to generate reference values ​​for the MDCT coefficients of the 160-point high-frequency subband signal.

[0391] It should be noted that if the bitstream encapsulation includes flatness edge information, additional operations are required. When SfpFLAG == 0, no operation is performed; when SfpFLAG == 1, an additional flattening operation is needed. The implementation of the flattening operation is as follows:

[0392] Taking the MDCT coefficient c(i) at position i as an example, calculate the average power spectrum e of the neighborhood of c(i). ave (i), as shown in formula (3).

[0393]

[0394] Where c(i-1) represents the MDCT coefficient at position (i-1) and c(i+1) represents the MDCT coefficient at position (i+1).

[0395] when e ave When (i) = 0, the MDCT coefficient c(i) at position i is not smoothed, i.e., c(i) = c(i); otherwise, the MDCT coefficient c(i) at position i is smoothed, i.e., c(i) is updated to c(i) and e. ave The ratio of (i), i.e., c(i) = c(i) / e ave (i).

[0396] After flattening, the previously obtained 8 sub-band spectral envelopes (i.e., the 8 sub-band spectral envelopes obtained by querying the quantization table) are called. These 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands. The reference values ​​of the MDCT coefficients of the generated 160-point high-frequency sub-band signal are divided into 8 reference high-frequency sub-bands. Based on a high-frequency sub-band and the corresponding reference high-frequency sub-band, gain control (multiplication in the frequency domain) is performed on the reference values ​​of the MDCT coefficients of the generated 160-point high-frequency sub-band signal. For example, the gain factor is calculated based on the average energy of the high-frequency sub-band and the average energy of the corresponding reference high-frequency sub-band. The MDCT coefficients corresponding to each point in the corresponding reference high-frequency sub-band are multiplied by the gain factor to ensure that the energy of the decoded virtual high-frequency MDCT coefficients is close to the energy of the original coefficients at the encoding end.

[0397] For example, assuming the average energy of the reference high-frequency subband (i.e., the subband divided by the reference values ​​of the 160-point MDCT coefficients of the generated high-frequency signal) is Y_L, and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectrum envelope decoded from the code stream) is Y_H, then a gain factor a = sqrt(Y_H / Y_L) is calculated, where sqrt() represents the square root calculation function, used to calculate the square root of (Y_H / Y_L). With the gain factor a, the MDCT coefficients of each point in the reference high-frequency subband are directly multiplied by a. The average energy of the MDCT coefficients (virtually generated) after gain control is very close to the original energy at the encoding end.

[0398] Finally, the inverse MDCT transform is invoked to generate the estimated value of the first subframe of the high-frequency subband signal (the subframe estimate calculated from the first set of 160 MDCT coefficients) and the estimated value of the second subframe of the high-frequency subband signal (the subframe estimate calculated from the second set of 160 MDCT coefficients). Combining the estimated values ​​of the first and second subframes, the estimated value x′ of the high-frequency subband signal is obtained. HB (n). An inverse MDCT transform is performed on the MDCT coefficients of the 320 points after gain, generating 640 estimated values. Through overlap, the first 320 valid estimated values ​​are taken as x′. HB (n).

[0399] Step 24, Synthesis Filter

[0400] An estimate of the low-frequency subband signal x′ was obtained at the decoding end. LB (n) and the estimated value x′ of the high-frequency subband signal HB After (n), simply upsampling and calling the QMF synthesis filter will generate a reconstructed signal x′(n) with 640 points.

[0401] This application embodiment can obtain optimal parameters by jointly training the relevant networks of the encoder and decoder through data collection. Users only need to prepare the data and set the corresponding network structure. After training is completed in the background, the trained model can be put into use.

[0402] In voice communication systems, communication methods include mono voice communication and stereo communication. Therefore, the embodiments of this application are not limited to the above-mentioned mono voice communication codec technology, but can also be stereo codec technology. In this case, the stereo codec technology can enable the encoder to receive signals from both the left and right channels (i.e., the input signal of the left channel and the input signal of the right channel), so that when the user wears headphones to make a call, there can be a certain sense of left and right space.

[0403] The low-complexity, low-bit-rate neural network speech compression method provided in this application embodiment will be described below in conjunction with stereo signals.

[0404] It should be noted that the stereo encoding method of this application embodiment includes, but is not limited to, the following two methods: 1) The left and right channels are respectively encoded and decoded using the above-mentioned mono signal encoding and decoding technology, and the stereo signal is recovered by combining the decoded single-channel reconstructed signal at the decoding end; 2) Parametric stereo technology is used for encoding and decoding.

[0405] Parametric stereo coding is a technique that treats stereo signals as mono signals with a small amount of parametric overhead describing stereo image information for efficient encoding. At the encoding end, the input signals of the left and right channels (i.e., stereo signals) are downmixed into a single signal, which is treated as the aforementioned input signal x(n). Then, the mono signal encoding method described in this application is used for encoding. Furthermore, parametric stereo coding extracts the correlation feature information of the left and right channels (i.e., stereo feature vectors, typically 1-2 kbps), which are used to mix the side information of the mono channels. At the decoding end, a downmixed signal (i.e., the reconstructed signal) is obtained through decoding. The stereo signals of the left and right channels are reconstructed using the decoded downmixed signal and the stereo feature vectors describing the correlation of the left and right channels.

[0406] Following the above embodiments, for the 8-bit frame header in the bitstream encapsulation, a 1-bit audio channel bit can be allocated. The 1-bit stereo encoding bit indicates whether mono or stereo encoding is used. The specific implementation method is as follows:

[0407] At the encoding end, if the number of input channels is 1, the channel flag is set to 0, and the corresponding channel bit in the frame header is written to 0, indicating that mono encoding is performed on the input signal, that is, the encoding method described above for the input signal x(n) is reused for encoding; if the number of input channels is 2, i.e. stereo, the channel flag is set to 1, and the corresponding channel bit in the frame header is written to 1, indicating that stereo encoding is performed on the input signal, that is, the parameter stereo encoding method described above is reused for encoding.

[0408] At the decoding end, the channel bit corresponding to the frame header is parsed. If the channel bit is 0, it indicates mono decoding, and the above decoding method is reused to decode the reconstructed signal; if the channel bit is 1, it indicates stereo decoding, and the stereo decoding method described above is reused for decoding.

[0409] like Figures 15A-15E The bitstream encapsulation shown above Figures 13A-13EThe code stream encapsulation shown carries stereo feature vectors. At the decoding end, the stereo signals of the left and right channels are reconstructed by decoding one downmixed signal and the stereo feature vectors describing the correlation between the left and right channels.

[0410] Additionally, the 8-bit frame header in the bitstream encapsulation may not require the allocation of audio channel bits. For example, if no audio channel bits are specified, mono encoding is used by default.

[0411] Please note that the frame header in each embodiment of this application is not limited to 8 bits, and can be more or less bits. In another embodiment, more bits can be allocated to the channel bits, such as 2 channel bits, to represent more channels of encoding and decoding, such as 3.1 channels or 4.1 channels, etc.

[0412] In summary, the multi-mode multi-rate encoding and decoding method provided in this application, through the organic combination of signal decomposition, signal processing technology and deep neural networks, significantly improves audio quality compared to signal processing schemes while ensuring acceptable complexity.

[0413] The audio encoding or decoding method provided in this application has been described above in conjunction with exemplary applications and implementations of the terminal devices provided in the embodiments of this application. This application also provides an audio encoding device and an audio decoding device. In practical applications, the functional modules in the audio encoding device and the audio decoding device can be collaboratively implemented using the hardware resources of electronic devices (such as terminal devices, servers, or server clusters), such as computing resources like processors, communication resources (such as those used to support various communication methods like optical fiber and cellular), and memory. Figure 3A An audio encoding device 555 stored in a memory 550 is shown, and Figure 3B An audio decoding device 655 stored in memory 650 is shown. It can be software in the form of programs and plug-ins, such as software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules, application programming interfaces, plug-ins, cloud services, etc. in large software systems. Examples of different implementation methods are given below.

[0414] The audio encoding device 555 includes a series of modules, including a feature extraction module 5551, an encoding module 5552, and a signal encoding module 5553. The following description further illustrates the scheme by which the various modules in the audio encoding device 555 provided in this embodiment cooperate to achieve audio encoding.

[0415] The second acquisition module 5551 is used to, in response to an encoding request for an audio signal, acquire a target encoding mode for the audio signal from multiple encoding modes and a target bitrate mode for the audio signal from multiple bitrate modes; the extraction module 5552 is used to extract the encoding features of the audio signal from the audio signal using the target encoding mode; the signal encoding module 5553 is used to perform signal encoding processing on the encoding features of the audio signal using the target bitrate mode to obtain the audio bitstream of the audio signal; the construction module 5554 is used to determine the frame header based on the target encoding mode and the target bitrate mode; and the generation module 5555 is used to generate an audio bitstream encapsulation of the audio signal based on the audio bitstream and the frame header.

[0416] In some embodiments, when the target encoding mode is a wideband encoding mode, the extraction module 5552 is further configured to invoke a first neural network based on the wideband encoding mode; and extract the encoding features of the audio signal from the audio signal through the first neural network.

[0417] In some embodiments, the extraction module 5552 is further configured to perform feature extraction processing on the audio signal to obtain audio features of the audio signal; and to perform residual processing on the audio features using at least one residual unit included in the first neural network to obtain encoded features of the audio signal.

[0418] In some embodiments, the first neural network includes four coding blocks, each coding block including four or five residual units.

[0419] In some embodiments, the target bitrate mode is used to indicate the use of a target code table to encode the encoding features of the audio signal; the signal encoding module 5553 is further used to quantize the encoding features of the audio signal using the target code table to obtain quantized values ​​of the encoding features; and to entropy encode the quantized values ​​using the target bitrate corresponding to the target code table to obtain the audio bitstream of the audio signal.

[0420] In some embodiments, when the target coding mode is an ultra-wideband coding mode, the extraction module 5552 is further configured to perform subband decomposition processing on the audio signal to obtain low-frequency subband signals and high-frequency subband signals of the audio signal; extract low-frequency features of the low-frequency subband signals from the low-frequency subband signals through a second neural network; perform high-frequency analysis processing on the high-frequency subband signals to obtain high-frequency features of the high-frequency subband signals; and determine the low-frequency features and the high-frequency features as the coding features of the audio signal.

[0421] In some embodiments, the extraction module 5552 is further configured to perform frame segmentation processing on the high-frequency subband signal to obtain multiple subframes of the high-frequency subband signal; perform frequency band expansion processing on each subframe to obtain the subband spectrum envelope of each subframe; and use the subband spectrum envelopes corresponding to the multiple subframes as high-frequency features of the high-frequency subband signal.

[0422] In some embodiments, the extraction module 5552 is further configured to perform frequency domain transformation processing based on multiple sample points included in the subframe to obtain the transformation coefficients corresponding to the multiple sample points respectively; divide the transformation coefficients corresponding to the multiple sample points into multiple sub-bands; perform mean processing on the transformation coefficients included in each sub-band to obtain the average energy corresponding to each sub-band, and use the average energy as the sub-band spectral envelope corresponding to each sub-band.

[0423] In some embodiments, the target bitrate mode includes instructing the use of a first code table to encode the low-frequency features and a second code table to encode the high-frequency features. The signal encoding module 5553 is further configured to quantize the low-frequency features using the first code table to obtain quantized values ​​of the low-frequency features, and to perform entropy encoding on the quantized values ​​of the low-frequency features using the bitrate corresponding to the first code table to obtain a low-frequency bitstream of the low-frequency subband signal; to quantize the high-frequency features using the second code table to obtain quantized values ​​of the high-frequency features, and to perform entropy encoding on the quantized values ​​of the high-frequency features using the bitrate corresponding to the second code table to obtain a high-frequency bitstream of the high-frequency subband signal; and to construct an audio bitstream of the audio signal based on the low-frequency bitstream and the high-frequency bitstream.

[0424] In some embodiments, the target bit rate mode further includes instructing the use of a third code table to perform residual encoding on the residual features of the high-frequency features; the signal encoding module 5553 is further configured to determine the sub-band corresponding to the subframe of the high-frequency sub-band signal, and divide the sub-band into multiple subsets; perform mean processing on the transform coefficients included in each subset to obtain the average energy corresponding to each subset, and use the average energy as the envelope value corresponding to each subset; determine the quantization value of the sub-band spectrum envelope of the sub-band corresponding to the subset from the quantization values ​​of the high-frequency features, and use the quantization value of the envelope of the sub-band corresponding to the subset as the envelope value of the sub-band signal; The difference between the network value and the quantized value of the subband spectrum envelope is used as the first residual value; based on the first residual value, the residual feature of the high-frequency feature is determined; the residual feature of the high-frequency feature is quantized using the third code table to obtain the quantized value of the residual feature of the high-frequency feature, and the quantized value of the residual feature of the high-frequency feature is entropy encoded using the code rate corresponding to the third code table to obtain the residual code stream of the residual feature; the construction module 5554 is further used to use the low-frequency code stream, the high-frequency code stream and the residual code stream as the audio code stream of the audio signal.

[0425] In some embodiments, when the number of residual values ​​is N, N is a positive integer greater than 1. The signal encoding module 5553 is further configured to determine the quantization value of the nth residual value and the sum of the quantization value of the quantization value of the nth residual value and the quantization value of the subband spectrum envelope of the corresponding subband; take the difference between the envelope value corresponding to the subset and the sum as the (n+1)th residual value; and determine the N residual values ​​as the residual features of the high-frequency features; wherein n is a positive integer that increases sequentially, 1≤n≤N.

[0426] In some embodiments, before generating the audio stream encapsulation of the audio signal based on the audio stream and the frame header, the signal encoding module 5553 is further configured to determine the flatness edge information of the high-frequency subband signal; the generation module 5555 is further configured to combine the audio stream, the frame header and the flatness edge information to obtain the audio stream encapsulation of the audio signal.

[0427] In some embodiments, the signal encoding module 5553 is further configured to divide the transform coefficients of the subframe of the high-frequency subband signal into multiple blocks; determine a first flatness of each block, and determine a second flatness of a specified low-frequency band of the audio signal; when the first flatness is less than the second flatness, or the first flatness is less than a flatness threshold, set the flatness edge information to a first value, wherein the first value indicates that flattening processing is required during audio decoding; when the first flatness is greater than or equal to the second flatness, and the first flatness is greater than or equal to the flatness threshold, set the flatness edge information to a second value, wherein the second value indicates that flattening processing is not required during audio decoding.

[0428] In some embodiments, the frame header further includes at least one channel bit, which is used to indicate whether the audio signal is encoded using mono or stereo encoding.

[0429] The audio decoding device 655 includes a series of modules, including a first acquisition module 6551, a signal decoding module 6552, and a reconstruction module 6553. The following description further illustrates the scheme by which the various modules in the audio encoding device 555 provided in this application cooperate to achieve audio encoding.

[0430] The first acquisition module 6551 is configured to, in response to a decoding request for an audio stream encapsulation, acquire a target encoding mode and a target bitrate mode from the frame header included in the audio stream encapsulation; wherein the audio stream included in the audio stream encapsulation is obtained by audio encoding an audio signal using the target encoding mode and the target bitrate mode, the target encoding mode being acquired from multiple encoding modes, and the target bitrate mode being acquired from multiple bitrate modes; the signal decoding module 6552 is configured to perform signal decoding processing on the audio stream using the target encoding mode and the target bitrate mode to obtain an estimated value of the encoding features corresponding to the audio stream; the reconstruction module 6553 is configured to perform reconstruction processing on the estimated value of the encoding features corresponding to the audio stream using the target encoding mode to obtain a synthesized audio signal corresponding to the audio stream.

[0431] In some embodiments, when the target encoding mode is a wideband encoding mode, the target bitrate mode is used to indicate the use of a target code table to encode the encoding features of the audio signal; the signal decoding module 6552 is further used to perform entropy decoding processing on the audio bitstream using the target bitrate corresponding to the target code table to obtain the quantization value corresponding to the audio bitstream; and to perform inverse quantization processing on the quantization value corresponding to the audio bitstream using the target code table to obtain the estimated value of the encoding features corresponding to the audio bitstream.

[0432] In some embodiments, when the target encoding mode is a wideband encoding mode, the reconstruction module 6553 is further configured to invoke a third neural network based on the wideband encoding mode; and to reconstruct the estimated encoding features corresponding to the audio bitstream through the third neural network to obtain the synthesized audio signal corresponding to the audio bitstream.

[0433] In some embodiments, the reconstruction module 6553 is further configured to use at least one residual unit included in the third neural network to perform residual processing on the coding feature estimate corresponding to the audio bitstream to obtain the audio feature estimate corresponding to the audio bitstream; and to perform feature reconstruction processing on the audio feature estimate corresponding to the audio bitstream to obtain the synthesized audio signal corresponding to the audio bitstream.

[0434] In some embodiments, the third neural network includes four decoding blocks, each of which includes four or five residual units.

[0435] In some embodiments, when the target encoding mode is an ultra-wideband encoding mode, the target bitrate mode includes instructing the use of a first code table to encode the low-frequency features of the audio signal and a second code table to encode the high-frequency features of the audio signal. The audio bitstream includes a low-frequency bitstream and a high-frequency bitstream. The signal decoding module 6552 is further configured to obtain the low-frequency bitstream and the high-frequency bitstream from the audio bitstream using the target encoding mode; perform entropy decoding on the low-frequency bitstream using the bitrate corresponding to the first code table to obtain the quantization value corresponding to the low-frequency bitstream, and perform inverse quantization on the quantization value corresponding to the low-frequency bitstream using the first code table to obtain the low-frequency feature estimate value corresponding to the low-frequency bitstream; perform entropy decoding on the high-frequency bitstream using the bitrate corresponding to the second code table to obtain the quantization value corresponding to the high-frequency bitstream, and perform inverse quantization on the quantization value corresponding to the high-frequency bitstream using the second code table to obtain the high-frequency feature estimate value corresponding to the high-frequency bitstream; and determine the encoding feature estimate value corresponding to the audio bitstream based on the low-frequency feature estimate value and the high-frequency feature estimate value.

[0436] In some embodiments, the audio bitstream further includes a residual bitstream, and the target bitrate mode further includes an instruction to use a third code table to perform residual encoding on the residual features of the high-frequency features corresponding to the audio bitstream; the signal decoding module 6552 is further configured to perform entropy decoding processing on the residual bitstream using the bitrate corresponding to the third code table to obtain the quantization value corresponding to the residual bitstream, and perform inverse quantization processing on the quantization value corresponding to the residual bitstream using the third code table to obtain the residual feature estimate value corresponding to the residual bitstream; the sum of the high-frequency feature estimate value and the residual feature estimate value is determined as the final estimate value of the high-frequency feature; the low-frequency feature estimate value and the final estimate value of the high-frequency feature are determined as the coding feature estimate value corresponding to the audio bitstream.

[0437] In some embodiments, the reconstruction module 6553 is further configured to, when the target coding mode is ultra-wideband coding, perform feature reconstruction processing on the low-frequency feature estimates included in the coding features through a fourth neural network to obtain low-frequency subband signal estimates corresponding to the audio bitstream; perform high-frequency reconstruction processing on the high-frequency feature estimates included in the coding features to obtain high-frequency subband signal estimates corresponding to the audio bitstream; and perform subband synthesis on the low-frequency subband signal estimates and the high-frequency subband signal estimates to obtain a synthesized audio signal corresponding to the audio bitstream.

[0438] In some embodiments, the reconstruction module 6553 is further configured to perform frequency domain transformation processing on the sample points of the first half and the sample points of the second half of the low-frequency subband signal estimation value to obtain a first transformation coefficient corresponding to the sample points of the first half and a second transformation coefficient corresponding to the sample points of the second half; based on the first transformation coefficient, perform inverse band expansion processing on the high-frequency feature estimation value to obtain a first high-frequency subband signal estimation value; based on the second transformation coefficient, perform inverse band expansion processing on the high-frequency feature estimation value to obtain a second high-frequency subband signal estimation value; and combine the first high-frequency subband signal estimation value and the second high-frequency subband signal estimation value to obtain the high-frequency subband signal estimation value corresponding to the audio code stream.

[0439] In some embodiments, the reconstruction module 6553 is further configured to perform spectral copying processing on the latter half of the first transform coefficients to obtain a first reference transform coefficient of the reference high-frequency subband signal; perform gain processing on the first reference transform coefficient based on the first half of the subband spectrum envelope corresponding to the high-frequency feature estimate to obtain a first reference transform coefficient after gain; and perform inverse frequency domain transform processing on the first reference transform coefficient after gain to obtain an estimate of the first high-frequency subband signal.

[0440] In some embodiments, the audio stream encapsulation includes flatness side information; after performing spectral replication processing on the transformation coefficients in the second half of the first transformation coefficients to obtain the first reference transformation coefficients of the reference high-frequency subband signal, the reconstruction module 6553 is further configured to, when the flatness side information indicates that flattening processing needs to be performed during audio decoding, determine the (i - 1)-th first reference transformation coefficient, the i-th first reference transformation coefficient, and the (i + 1)-th first reference transformation coefficient; determine an average power spectrum based on the (i - 1)-th first reference transformation coefficient, the i-th first reference transformation coefficient, and the (i + 1)-th first reference transformation coefficient; use the ratio of the i-th first reference transformation coefficient to the average power spectrum as the new i-th first reference transformation coefficient; where 1 < i < I, i is a positive integer, and I is the number of the first reference transformation coefficients.

[0441] In some embodiments, the frame header further includes at least 1 channel bit, and the channel bit is used to indicate whether to use a mono decoding method or a stereo decoding method to perform audio decoding on the audio stream.

[0442] An embodiment of the present application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions, so that the electronic device executes the audio encoding method or the audio decoding method described above in the embodiments of the present application.

[0443] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, where the computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the processor will be caused to execute the audio encoding method or the audio decoding method provided in the embodiments of the present application. For example, Figure 4A the audio encoding method shown.

[0444] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various electronic devices including one or any combination of the above memories.

[0445] In some embodiments, the computer-executable instructions (referred to as executable instructions for short) may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0446] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0447] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0448] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0449] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An audio encoding method, characterized in that, The method includes: In response to an encoding request for an audio signal, a target bitrate mode for the audio signal is obtained from multiple bitrate modes, and a target encoding mode for the audio signal is obtained from multiple encoding modes. When the target coding mode is a wideband coding mode, the first neural network is invoked based on the wideband coding mode; The audio features are extracted from the audio signal by the first neural network, and the audio features are subjected to residual processing using at least one residual unit included in the first neural network to obtain the encoded features of the audio signal. By using the target bitrate mode, the encoding features of the audio signal are processed by signal encoding to obtain the audio bitstream of the audio signal; Based on the target bitrate pattern, determine the frame header; Based on the audio bitstream and the frame header, an audio bitstream encapsulation of the audio signal is generated.

2. The method according to claim 1, characterized in that, The target bit rate mode is used to indicate the use of a target code table to encode the encoding features of the audio signal. The step of performing signal encoding processing on the encoding features of the audio signal using the target bitrate mode to obtain the audio bitstream of the audio signal includes: The encoding features of the audio signal are quantized using the target code table to obtain quantized values ​​of the encoding features. The quantization value is entropy encoded using the target bitrate corresponding to the target code table to obtain the audio bitstream of the audio signal.

3. An audio encoding method, characterized in that, The method includes: In response to an encoding request for an audio signal, a target bitrate mode for the audio signal is obtained from multiple bitrate modes, and a target encoding mode for the audio signal is obtained from multiple encoding modes. When the target coding mode is an ultra-wideband coding mode, low-frequency features are extracted from the low-frequency subband signal of the audio signal, and high-frequency features are extracted from the high-frequency subband signal of the audio signal. The target code rate mode is used to perform signal encoding processing on the low-frequency features and the high-frequency features respectively to obtain the low-frequency code stream of the low-frequency subband signal and the high-frequency code stream of the high-frequency subband signal. The target code rate mode also includes instructing the use of a third code table to perform residual encoding on the residual features of the high-frequency features to obtain the residual code stream of the residual features. Based on the target bitrate pattern, determine the frame header; An audio stream encapsulation of the audio signal is generated based on the low-frequency bitstream, the high-frequency bitstream, the residual bitstream, and the frame header.

4. The method according to claim 3, characterized in that, Extracting low-frequency features from the low-frequency sub-band signal of the audio signal includes: The low-frequency features of the low-frequency sub-band signal are extracted from the low-frequency sub-band signal using a second neural network. Extracting high-frequency features from the high-frequency sub-band signal of the audio signal includes: The high-frequency subband signal is subjected to high-frequency analysis processing to obtain the high-frequency characteristics of the high-frequency subband signal.

5. The method according to claim 3, characterized in that, The target code rate mode includes an instruction to use a first code table to encode the low-frequency features and a second code table to encode the high-frequency features. The step of performing signal encoding processing on the low-frequency features and the high-frequency features respectively through the target code rate mode to obtain the low-frequency code stream of the low-frequency sub-band signal and the high-frequency code stream of the high-frequency sub-band signal includes: The low-frequency feature is quantized using the first code table to obtain the quantized value of the low-frequency feature, and the quantized value of the low-frequency feature is entropy encoded using the code rate corresponding to the first code table to obtain the low-frequency code stream of the low-frequency subband signal. The high-frequency feature is quantized using the second code table to obtain the quantized value of the high-frequency feature. Then, the quantized value of the high-frequency feature is entropy encoded using the code rate corresponding to the second code table to obtain the high-frequency code stream of the high-frequency subband signal.

6. The method according to claim 5, characterized in that, The method further includes: Determine the sub-band corresponding to the sub-frame of the high-frequency sub-band signal, and divide the sub-band into multiple subsets; The transformation coefficients included in each subset are averaged to obtain the average energy corresponding to each subset, and the average energy is used as the envelope value corresponding to each subset. Among the quantized values ​​of the high-frequency features, the quantized value of the subband spectrum envelope corresponding to the subset is determined, and the difference between the envelope value corresponding to the subset and the quantized value of the subband spectrum envelope is used as the first residual value. Based on the first residual value, the residual characteristics of the high-frequency features are determined; The residual features of the high-frequency features are quantized using the third code table to obtain quantized values ​​of the residual features. Then, the quantized values ​​of the residual features are entropy encoded using the code rate corresponding to the third code table to obtain the residual code stream of the residual features.

7. The method according to claim 6, characterized in that, When the number of residual values ​​is N, where N is a positive integer greater than 1, determining the residual features of the high-frequency features based on the first residual value includes: Determine the quantization value of the nth residual value, and determine the sum of the quantization value of the nth residual value and the quantization value of the subband spectral envelope of the corresponding subband; The difference between the envelope value corresponding to the subset and the sum is taken as the (n+1)th residual value; The N residual values ​​are determined as the residual features of the high-frequency features; Where n is a positive integer that increases sequentially, 1≤n≤N.

8. The method according to any one of claims 3 to 7, characterized in that, Before generating the audio stream encapsulation of the audio signal based on the low-frequency bitstream, the high-frequency bitstream, the residual bitstream, and the frame header, the method further includes: Determine the flatness edge information of the high-frequency sub-band signal; The audio bitstream encapsulation for generating the audio signal based on the low-frequency bitstream, the high-frequency bitstream, the residual bitstream, and the frame header includes: By combining the low-frequency bitstream, the high-frequency bitstream, the residual bitstream, the frame header, and the flatness edge information, the audio bitstream encapsulation of the audio signal is obtained.

9. The method according to claim 8, characterized in that, Determining the flatness edge information of the high-frequency sub-band signal includes: The transform coefficients of the sub-frames of the high-frequency subband signal are divided into multiple blocks; Determine a first flatness for each of the blocks, and determine a second flatness for a specified low-frequency band of the audio signal; When the first flatness is less than the second flatness, or the first flatness is less than the flatness threshold, the flatness edge information is set to a first value, wherein the first value indicates that flattening processing is required during audio decoding; When the first flatness is greater than or equal to the second flatness, and the first flatness is greater than or equal to the flatness threshold, the flatness edge information is set to a second value, wherein the second value indicates that flattening processing is not required during audio decoding.

10. An audio decoding method, characterized in that, The method includes: In response to a decoding request for an audio stream encapsulation, a target bitrate mode is obtained from the frame header included in the audio stream encapsulation, and a target encoding mode for the audio signal is obtained from multiple encoding modes. The low-frequency and high-frequency bitstreams included in the audio bitstream encapsulation are obtained by audio encoding the low-frequency sub-band signal and the high-frequency sub-band signal of the audio signal respectively through the target bitrate mode and the target encoding mode. The target bitrate mode is obtained from multiple bitrate modes. Using the target bitrate mode, signal decoding processing is performed on the low-frequency bitstream and the high-frequency bitstream respectively to obtain low-frequency feature estimates and high-frequency feature estimates; When the target coding mode is ultra-wideband coding, the low-frequency feature estimates and the high-frequency feature estimates are reconstructed by the fourth neural network and the sixth neural network respectively to obtain the low-frequency subband signal estimates and the high-frequency subband signal estimates. The reconstructed audio signal is determined based on the estimated values ​​of the low-frequency subband signal and the estimated values ​​of the high-frequency subband signal.

11. The method according to claim 10, characterized in that, The target bit rate mode includes an instruction to use a first code table to encode the low-frequency features of the audio signal and a second code table to encode the high-frequency features of the audio signal. The step of performing signal decoding processing on the low-frequency bitstream and the high-frequency bitstream respectively through the target bitrate mode to obtain low-frequency feature estimates and high-frequency feature estimates includes: Using the code rate corresponding to the first code table, the low-frequency code stream is entropy decoded to obtain the quantization value corresponding to the low-frequency code stream. Then, the quantization value corresponding to the low-frequency code stream is inversely quantized using the first code table to obtain the low-frequency feature estimation value corresponding to the low-frequency code stream. Using the code rate corresponding to the second code table, entropy decoding is performed on the high-frequency code stream to obtain the quantization value corresponding to the high-frequency code stream. Then, the quantization value corresponding to the high-frequency code stream is inversely quantized using the second code table to obtain the high-frequency feature estimation value corresponding to the high-frequency code stream.

12. The method according to claim 11, characterized in that, The audio stream encapsulation also includes a residual stream, and the target bitrate mode also includes an instruction to use a third code table to perform residual encoding on the residual features of the high-frequency features; The method further includes: Using the code rate corresponding to the third code table, entropy decoding is performed on the residual code stream to obtain the quantization value corresponding to the residual code stream. Then, the quantization value corresponding to the residual code stream is inversely quantized using the third code table to obtain the residual feature estimate value corresponding to the residual code stream.

13. The method according to claim 10, characterized in that, The high-frequency feature estimation value is reconstructed to obtain the high-frequency sub-band signal estimation value, including: The sample points of the first half and the sample points of the second half of the low-frequency subband signal estimation are subjected to frequency domain transformation to obtain the first transformation coefficients corresponding to the sample points of the first half and the second transformation coefficients corresponding to the sample points of the second half. Based on the first transformation coefficients, the high-frequency feature estimate is subjected to inverse band expansion processing to obtain the first high-frequency sub-band signal estimate. Based on the second transformation coefficients, the high-frequency feature estimate is subjected to inverse band expansion processing to obtain the second high-frequency sub-band signal estimate. By combining the first high-frequency subband signal estimate and the second high-frequency subband signal estimate, the high-frequency subband signal estimate is obtained.

14. The method according to claim 13, characterized in that, The step of performing inverse band-expansion processing on the high-frequency feature estimate based on the first transform coefficients to obtain the first high-frequency sub-band signal estimate includes: The latter half of the first transformation coefficients is subjected to spectral copying processing to obtain the first reference transformation coefficients of the reference high-frequency subband signal; Based on the first half of the subband spectrum envelope corresponding to the high-frequency feature estimate, the first reference transformation coefficient is subjected to gain processing to obtain the first reference transformation coefficient after gain. The first reference transformation coefficients after gain are subjected to inverse frequency domain transformation to obtain the estimated value of the first high-frequency subband signal.

15. The method according to claim 14, characterized in that, The audio stream encapsulation includes flatness edge information; After performing spectral copying processing on the latter half of the first transform coefficients to obtain the first reference transform coefficients of the reference high-frequency subband signal, the method further includes: When the flatness edge information represents the need for flattening processing during audio decoding, the (i-1)th first reference transform coefficient, the ith first reference transform coefficient, and the (i+1)th first reference transform coefficient are determined. The average power spectrum is determined based on the (i-1)th first reference transformation coefficient, the ith first reference transformation coefficient, and the (i+1)th first reference transformation coefficient. The ratio of the i-th first reference transformation coefficient to the average power spectrum is taken as the new i-th first reference transformation coefficient; Where 1 < i < I, i is a positive integer, and I is the number of the first reference transformation coefficients.

16. A method for processing audio bitstreams, characterized in that, An audio stream is stored on a non-volatile computer-readable storage medium, the audio stream being generated based on the audio encoding method according to any one of claims 1 to 9, or decoded based on the audio decoding method according to any one of claims 10 to 15.

17. A computer-readable storage medium, characterized in that, It stores computer-executable instructions, which, when executed by a processor, implement the audio encoding method according to any one of claims 1 to 9, or the audio decoding method according to any one of claims 10 to 15.

Images