Audio coding method, audio decoding method, audio coding device, audio decoding device and readable storage medium

Abstract

The invention provides an audio coding method and device, an audio decoding method and device, electronic equipment, a computer readable storage medium and a computer program product. The audio decoding method comprises the following steps: in response to a decoding request for audio code stream packaging, obtaining a target coding mode and a target code rate mode from a frame header included in the audio code stream packaging; wherein an audio code stream included in the audio code stream package is obtained by performing audio coding on the audio signal through a target coding mode and a target code rate mode, the target coding mode is obtained from a plurality of coding modes, and the target code rate mode is obtained from a plurality of code rate modes; performing signal decoding processing on the audio code stream through the target coding mode and the target code rate mode to obtain a coding feature estimation value corresponding to the audio code stream; and through the target coding mode, performing reconstruction processing on the coding feature estimation value corresponding to the audio code stream to obtain a synthetic audio signal corresponding to the audio code stream.

Description

[0001] This application is a divisional application of the patent application with application number 202311614893.4, application date November 29, 2023, and invention name “audio encoding method, audio decoding method, device, readable storage medium”. Technical Field

[0002] The present application relates to artificial intelligence technology, and in particular to an audio encoding method, an audio decoding method, a device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0003] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, it enables machines to have the functions of perception, reasoning and decision-making. Artificial Intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0004] Audio coding and decoding technology is one of the important applications in the field of artificial intelligence. Audio coding and decoding technology is a core technology in communication services including remote audio and video calls. Speech coding technology, in simple terms, is to use less network bandwidth resources to transmit as much voice information as possible. From the perspective of Shannon's information theory, speech coding is a kind of source coding. The purpose of source coding is to compress the amount of data we want to transmit as much as possible at the encoding end, remove the redundancy in the information, and at the same time be able to restore it losslessly (or nearly losslessly) at the decoding end.

[0005] In the related art, the quality of the audio generated by decoding at the decoding end is relatively simple and cannot meet user needs. Summary of the invention

[0006] The embodiments of the present application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can output synthetic audio signals of different quality levels while ensuring the efficiency of audio decoding.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] The present application provides an audio encoding method, including:

[0009] In response to an encoding request for an audio signal, acquiring a target encoding mode for the audio signal from a plurality of encoding modes, and acquiring a target bit rate mode for the audio signal from a plurality of bit rate modes;

[0010] Extracting the coding feature of the audio signal from the audio signal by using the target coding mode;

[0011] Performing signal encoding processing on the encoding feature of the audio signal through the target bit rate mode to obtain an audio bit stream of the audio signal;

[0012] Determining a frame header based on the target coding mode and the target bit rate mode;

[0013] An audio code stream encapsulation of the audio signal is generated based on the audio code stream and the frame header.

[0014] The present application provides an audio decoding method, including:

[0015] In response to a decoding request for an audio code stream encapsulation, obtaining a target coding mode and a target bit rate mode from a frame header included in the audio code stream encapsulation;

[0016] The audio code stream included in the audio code stream encapsulation is obtained by audio encoding the audio signal through the target coding mode and the target bit rate mode, the target coding mode is obtained from multiple coding modes, and the target bit rate mode is obtained from multiple bit rate modes;

[0017] By using the target coding mode and the target bit rate mode, the audio code stream is subjected to signal decoding processing to obtain a coding feature estimation value corresponding to the audio code stream;

[0018] Through the target coding mode, the coding feature estimation value corresponding to the audio code stream is reconstructed to obtain a synthetic audio signal corresponding to the audio code stream.

[0019] The present application provides an audio encoding device, including:

[0020] a second acquisition module, configured to, in response to an encoding request for an audio signal, acquire a target encoding mode for the audio signal from a plurality of encoding modes, and acquire a target bit rate mode for the audio signal from a plurality of bit rate modes;

[0021] An extraction module, used for extracting the coding feature of the audio signal from the audio signal by using the target coding mode;

[0022] A signal encoding module, used to perform signal encoding processing on the encoding feature of the audio signal through the target bit rate mode to obtain an audio bit stream of the audio signal;

[0023] A construction module, configured to determine a frame header based on the target coding mode and the target bit rate mode;

[0024] A generating module is used to generate an audio code stream encapsulation of the audio signal based on the audio code stream and the frame header.

[0025] The present application provides an audio decoding device, including:

[0026] A first acquisition module, configured to, in response to a decoding request for an audio code stream encapsulation, acquire a target coding mode and a target bit rate mode from a frame header included in the audio code stream encapsulation;

[0027] The audio code stream included in the audio code stream encapsulation is obtained by audio encoding the audio signal through the target coding mode and the target bit rate mode, the target coding mode is obtained from multiple coding modes, and the target bit rate mode is obtained from multiple bit rate modes;

[0028] A signal decoding module, used to perform signal decoding processing on the audio code stream according to the target coding mode and the target bit rate mode, so as to obtain a coding feature estimation value corresponding to the audio code stream;

[0029] The reconstruction module is used to reconstruct the coding feature estimation value corresponding to the audio code stream through the target coding mode to obtain a synthetic audio signal corresponding to the audio code stream.

[0030] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0031] A memory for storing computer executable instructions;

[0032] The processor is used to implement the audio encoding method or audio decoding method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0033] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implements the audio encoding method or audio decoding method provided in the embodiment of the present application.

[0034] An embodiment of the present application provides a computer program product, including computer executable instructions, which, when executed by a processor, implement the audio encoding method or audio decoding method provided in the embodiment of the present application.

[0035] The embodiments of the present application have the following beneficial effects:

[0036] Through different coding modes and bit rate modes, the audio code stream is decoded to obtain coding feature estimation values ​​of different precisions, and then the coding feature estimation values ​​of different precisions are reconstructed to obtain synthetic audio signals of different quality levels, thereby improving the diversity of the quality of the synthetic audio signal to meet the actual application needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of spectrum comparison at different bit rates provided in an embodiment of the present application;

[0038] Figure 2 It is a schematic diagram of the architecture of the audio codec system provided in an embodiment of the present application;

[0039] Figure 3A is a first structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0040] Figure 3B is a second structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0041] Figure 4A It is a schematic diagram of a first process of the audio encoding method provided in an embodiment of the present application;

[0042] Figure 4B is a second flow chart of the audio encoding method provided in an embodiment of the present application;

[0043] Figure 4C is a third flow chart of the audio encoding method provided in an embodiment of the present application;

[0044] Figure 4D is a fourth flow chart of the audio encoding method provided in an embodiment of the present application;

[0045] Figure 4E is a fifth flow chart of the audio encoding method provided in an embodiment of the present application;

[0046] Figure 5A It is a schematic diagram of a first flow chart of an audio decoding method provided in an embodiment of the present application;

[0047] Figure 5B is a second flow chart of the audio decoding method provided in an embodiment of the present application;

[0048] Figure 5C is a third flow chart of the audio decoding method provided in an embodiment of the present application;

[0049] Fig. 6A It is a schematic diagram of a channel provided in an embodiment of the present application without adopting grouped convolution;

[0050] Figure 6BIt is a schematic diagram of a channel using grouped convolution provided in an embodiment of the present application;

[0051] Figure 6C is a schematic diagram of a voice communication link provided in an embodiment of the present application;

[0052] Figure 7 It is a flowchart of a multi-mode multi-rate speech encoding method provided in an embodiment of the present application;

[0053] Figure 8 is a schematic diagram of a filter bank provided in an embodiment of the present application;

[0054] Fig. 9A is a schematic diagram of a common convolutional network provided in an embodiment of the present application;

[0055] Fig. 9B is a schematic diagram of a dilated convolutional network provided in an embodiment of the present application;

[0056] Fig.10 is a schematic diagram of frequency band expansion provided by an embodiment of the present application;

[0057] Fig.11 is a schematic diagram of a third neural network provided in an embodiment of the present application;

[0058] Fig. 12A is a schematic diagram of the residual block structure used in the coding block provided in an embodiment of the present application;

[0059] Fig. 12B is a schematic diagram of the residual unit structure provided in an embodiment of the present application;

[0060] Fig.13A It is a schematic diagram of the structure of code stream encapsulation in broadband coding mode provided by an embodiment of the present application;

[0061] Fig. 13B It is a structural schematic diagram of a code stream encapsulation that does not contain residual and does not contain flatness edge information in an ultra-wideband coding mode provided in an embodiment of the present application;

[0062] Fig. 13C It is a schematic diagram of the structure of a code stream encapsulation including a residual and excluding flatness edge information in an ultra-wideband coding mode provided in an embodiment of the present application;

[0063] Fig.13D It is a structural schematic diagram of a code stream encapsulation that does not contain residuals and contains flatness edge information in an ultra-wideband coding mode provided by an embodiment of the present application;

[0064] Fig.13E It is a schematic diagram of the structure of a code stream encapsulation including residual and flatness edge information in an ultra-wideband coding mode provided in an embodiment of the present application;

[0065] Fig.14 is a schematic diagram of a first neural network provided in an embodiment of the present application;

[0066] Fig.15A It is a structural diagram of code stream encapsulation in a wideband stereo coding mode provided by an embodiment of the present application;

[0067] Fig. 15B It is a structural schematic diagram of a bitstream encapsulation that does not contain residuals and does not contain flatness edge information in an ultra-wideband stereo coding mode provided in an embodiment of the present application;

[0068] Fig. 15C It is a schematic diagram of the structure of a code stream encapsulation including residual and excluding flatness edge information in an ultra-wideband stereo coding mode provided in an embodiment of the present application;

[0069] Fig.15D It is a structural schematic diagram of a code stream encapsulation that does not contain residuals and contains flatness edge information in an ultra-wideband stereo coding mode provided in an embodiment of the present application;

[0070] Fig.15E It is a schematic diagram of the structure of the code stream encapsulation containing residual and flatness edge information in the ultra-wideband stereo coding mode provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0072] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0074] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0075] 1) Neural Network (NN): It is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.

[0076] 2) Deep Learning (DL): It is a new research direction in the field of Machine Learning (ML). Deep learning is to learn the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0077] 3) Quantization: It refers to the process of approximating the continuous value (or a large number of discrete values) of a signal to a finite number (or a small number) of discrete values. Quantization includes vector quantization (VQ) and scalar quantization.

[0078] Among them, vector quantization is an effective lossy compression technology, and its theoretical basis is Shannon's rate-distortion theory. The basic principle of vector quantization is to use the index of the codeword that best matches the input vector in the code table (also known as the quantization value) to replace the input vector for transmission and storage, while decoding only requires a simple table lookup operation. For example, a number of scalar data are combined into a vector space, and the vector space is divided into several small areas. When quantizing, the vectors that fall into the small areas are replaced by the corresponding indexes instead of the input vectors.

[0079] Scalar quantization is the quantization of scalars, that is, one-dimensional vector quantization, which divides the dynamic range into several small intervals, each of which has a representative value (i.e., index). When the input signal falls into a certain interval, the input signal is quantized into the representative value.

[0080] 4) Entropy coding: A lossless coding method that does not lose any information according to the entropy principle during the encoding process. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exp-Golomb coding, and arithmetic coding.

[0081] 5) Quadrature Mirror Filters (QMF): It is a filter pair that includes analysis and synthesis. The QMF analysis filter is used to decompose the sub-band signal to reduce the signal bandwidth so that each sub-band signal can be processed smoothly through its own channel; the QMF synthesis filter is used to synthesize the sub-band signals restored by the decoding end, for example, to reconstruct the original audio signal through zero-value interpolation and bandpass filtering.

[0082] Voice coding technology is to use less network bandwidth resources to transmit as much voice information as possible. The compression rate of voice codec can reach more than 10 times, that is, after the original 10MB of voice data is compressed by the encoder, only 1MB is needed to transmit, which greatly reduces the bandwidth resources consumed by transmitting information. For example, for a wideband voice signal with a sampling rate of 16000Hz, if a 16-bit sampling depth (the degree of precision of recording voice intensity during sampling) is used, the bit rate (the amount of data transmitted per unit time) of the uncompressed version is 256kbps; if voice coding technology is used, even if it is lossy coding, within the bit rate range of 10-20kbps, the quality of the reconstructed voice signal can be close to the uncompressed version, and even the auditory perception is no different. If a higher sampling rate service is required, such as 32000Hz ultra-wideband voice, the bit rate range must be at least 30kbps.

[0083] In the communication system, in order to ensure smooth communication, the industry deploys standard voice codec protocols, such as standards from international and domestic standard organizations such as ITU-T, 3GPP, IETF, AVS, CCSA, G.711, G.722, AMR series, EVS, OPUS and other standards. Figure 1 A spectrum comparison diagram at different bit rates is given to demonstrate the relationship between compression bit rate and quality. Curve 101 is the spectrum curve of the original speech, that is, the signal without compression; Curve 102 is the spectrum curve of the OPUS encoder at a bit rate of 20kbps; Curve 103 is the spectrum curve of the OPUS encoder at a bit rate of 6kbps. Figure 1 It can be seen that as the encoding bit rate increases, the compressed signal is closer to the original signal.

[0084] The principle of speech coding is as follows: speech coding can directly encode speech waveform samples, sample by sample; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0085] The above coding principles are all derived from voice signal modeling, that is, compression methods based on signal processing, which cannot guarantee the coding quality of audio. In order to improve coding efficiency while ensuring voice quality, embodiments of the present application provide an audio coding method, an audio decoding method, a device, an electronic device, a computer-readable storage medium, and a computer program product. The following describes an exemplary application of an electronic device provided by an embodiment of the present application. The electronic device provided by an embodiment of the present application can be implemented as a terminal device, or can be implemented as a server, or can be implemented by a terminal device and a server in collaboration. The following is an example of an electronic device being implemented as a terminal device.

[0086] For example, see Figure 2 , Figure 2 It is a schematic diagram of the architecture of the audio encoding and decoding system 10 provided in an embodiment of the present application. The audio decoding system 10 includes: a server 200, a network 300, a terminal device 400 (i.e., an encoding end) and a terminal device 500 (i.e., a decoding end), wherein the network 300 can be a local area network, a wide area network, or a combination of the two.

[0087] In some embodiments, a client 410 is running on the terminal device 400, and the client 410 can be various types of clients, such as an instant messaging client, a web conference client, a live broadcast client, a browser, etc. In response to an audio collection instruction triggered by a sender (such as the initiator of a web conference, a host, the initiator of a voice call, etc.), the client 410 calls the microphone of the terminal device 400 to collect audio signals, and performs audio encoding processing on the collected audio signals to obtain an audio code stream.

[0088] For example, the client 410 calls the audio encoding method provided in the embodiment of the present application to encode the collected audio signal, that is, obtains a target encoding mode for the audio signal from multiple encoding modes, and obtains a target bit rate mode for the audio signal from multiple bit rate modes; extracts the encoding features of the audio signal from the audio signal through the target encoding mode; performs signal encoding processing on the encoding features of the audio signal through the target bit rate mode to obtain an audio code stream of the audio signal; determines a frame header based on the target encoding mode and the target bit rate mode; and generates an audio code stream encapsulation of the audio signal based on the audio code stream and the frame header.

[0089] The client 410 can send the audio code stream package to the server 200 through the network 300, so that the server 200 sends the audio code stream package to the terminal device 500 associated with the recipient (such as the participant, audience of the network conference, the recipient of the voice call, etc.).

[0090] After receiving the audio code stream package sent by the server 200, the client 510 (such as an instant messaging client, a web conference client, a live broadcast client, a browser, etc.) can perform audio decoding processing on the audio code stream package to obtain a synthesized audio signal, thereby realizing audio communication.

[0091] For example, the client 510 calls the audio decoding method provided in the embodiment of the present application to decode the received audio code stream encapsulation, that is, obtains the target coding mode and the target bit rate mode from the frame header included in the audio code stream encapsulation; wherein the audio code stream included in the audio code stream encapsulation is obtained by audio encoding the audio signal through the target coding mode and the target bit rate mode, the target coding mode is obtained from multiple coding modes, and the target bit rate mode is obtained from multiple bit rate modes; through the target coding mode and the target bit rate mode, the audio code stream is subjected to signal decoding processing to obtain the coding feature estimation value corresponding to the audio code stream; through the target coding mode, the coding feature estimation value corresponding to the audio code stream is reconstructed to obtain a synthetic audio signal corresponding to the audio code stream.

[0092] In some embodiments, the embodiments of the present application can be implemented with the help of cloud technology (Cloud Technology). Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0093] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The service interaction function between the above servers 200 can be realized through cloud technology.

[0094] For example, Figure 2 The server 200 shown in the figure can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. Figure 2 The terminal devices 400 and 500 shown in the figure can be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, vehicle-mounted terminals, etc., but are not limited thereto. The terminal devices (such as the terminal devices 400 and 500) and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0095] In some embodiments, the terminal device or server 200 can also implement the audio encoding method or audio decoding method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP, a web conference APP, or an instant messaging APP, etc.; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0096] In some embodiments, multiple servers may form a blockchain, and server 200 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the audio encoding method or audio decoding method provided in the embodiment of the present application (such as audio encoding logic, audio decoding logic, high-frequency code stream, and low-frequency code stream) may be stored on the blockchain.

[0097] See also Figure 3A , Figure 3A is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, and takes the electronic device 500 as a terminal device as an example. Figure 3A The electronic device 500 shown includes: at least one processor 520, a memory 550, at least one network interface 530 and a user interface 540. The various components in the electronic device 500 are coupled together via a bus system 550. It is understood that the bus system 550 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 550 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 550 is not described in detail. Figure 3A Various buses are labeled as bus system 550 .

[0098] The processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0099] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 520.

[0100] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0101] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0102] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0103] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 530. Exemplary network interfaces 530 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0104] In some embodiments, the audio encoding device provided in the embodiments of the present application can be implemented in software. Figure 3A An audio encoding device 555 stored in a memory 550 is shown, which may be software in the form of a program and a plug-in, etc., including the following software modules: a second acquisition module 5551, an extraction module 5552, a signal encoding module 5553, a construction module 5554 and a generation module 5555, wherein the second acquisition module 5551, the extraction module 5552, the signal encoding module 5553, the construction module 5554 and the generation module 5555 are used to implement the audio encoding function, and these modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented.

[0105] See also Figure 3B , Figure 3B 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application, and takes the electronic device 600 as a terminal device as an example. Figure 3B The electronic device 600 shown includes: at least one processor 620, a memory 650, at least one network interface 630 and a user interface 640. The various components in the electronic device 600 are coupled together via a bus system 650. The memory 650 includes an operating system 651 and a network communication module 652. It should be noted that Figure 3B The function of the structure in Figure 3A The audio encoding device provided in the embodiment of the present application can be implemented in software. Figure 3B An audio decoding device 655 stored in a memory 650 is shown, which may be software in the form of a program or plug-in, including the following software modules: a first acquisition module 6551, a signal decoding module 6552, and a reconstruction module 6553, wherein the first acquisition module 6551, the signal decoding module 6552, and the reconstruction module 6553 are used to implement the audio decoding function, and these modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented.

[0106] As mentioned above, the audio encoding method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 4A , Figure 4A The following is a flow chart of the audio encoding method provided in the embodiment of the present application. The audio encoding function is realized by the audio encoding method. Figure 4A Steps 101 to 105 are shown for explanation.

[0107] In step 101, in response to an encoding request for an audio signal, a target encoding mode for the audio signal is acquired from a plurality of encoding modes, and a target bit rate mode for the audio signal is acquired from a plurality of bit rate modes.

[0108] The encoding request is used to indicate that the audio signal is to be encoded, and the encoding request includes the configuration information of the encoding, such as the sampling rate of the audio signal, the type of the audio signal (such as a wideband signal, an ultra-wideband signal), etc. The multiple encoding modes include a wideband encoding mode and an ultra-wideband encoding mode, and the bit rate mode is used to indicate that the signal is encoded using a specified code table and a corresponding bit rate, and different bit rate modes correspond to different code tables and bit rates. In the configuration stage, different encoding modes and bit rate modes can be configured for different audio signals.

[0109] As an example of obtaining an audio signal, the encoding end responds to an audio capture instruction triggered by a sender (such as the initiator of a web conference, a host, the initiator of a voice call, etc.), and calls the microphone of the terminal device of the encoding end to collect the audio signal to obtain the audio signal (also called the input signal).

[0110] In step 102, a coding feature of the audio signal is extracted from the audio signal by using a target coding mode.

[0111] like Figure 4B As shown, Figure 4B is a flow chart of an audio encoding method provided in an embodiment of the present application. When the target encoding mode is a broadband encoding mode, Figure 4B Shown in Figure 4A Step 102 in the method can be implemented through steps 1021A to 1022A: in step 1021A, the first neural network is called based on the broadband coding mode; in step 1022A, the coding features of the audio signal are extracted from the audio signal through the first neural network.

[0112] Here, when the target coding mode is a broadband coding mode, artificial intelligence technology is used to directly extract the coding features of the frequency signal.

[0113] like Figure 4C As shown, Figure 4C is a flowchart of an audio encoding method provided in an embodiment of the present application, Figure 4C Shown in Figure 4B Step 1022A in can be implemented by steps 10221A-10222A:

[0114] In step 10221A, feature extraction processing is performed on the audio signal to obtain audio features of the audio signal.

[0115] Here, the embodiment of the present application can call a first neural network (NN) based on the audio signal, and extract audio features from the audio signal through the first neural network, so as to continue to perform feature refinement based on the audio features. It should be noted that the embodiment of the present application is not limited to the structure of the first NN, and the first NN can be a convolutional neural network, a deep neural network, etc.

[0116] In step 10222A, residual processing is performed on the audio features using at least one residual unit included in the first neural network to obtain coding features of the audio signal.

[0117] Here, the first neural network includes 4 encoding blocks, and each encoding block includes 4 or 5 residual units.

[0118] like Figure 4D As shown, Figure 4D is a flow chart of an audio coding method provided in an embodiment of the present application. When the target coding mode is an ultra-wideband coding mode, Figure 4D Shown in Figure 4A Step 102 in the embodiment can be implemented by steps 1021B to 1024B:

[0119] In step 1021B, sub-band decomposition processing is performed on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal.

[0120] Here, when the target coding mode is the ultra-wideband coding mode, it means that the audio signal is an ultra-wideband signal, and the ultra-wideband signal can be subjected to sub-band decomposition processing to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal.

[0121] It should be noted that the embodiment of the present application does not limit the frequency bands of the low-frequency sub-band signal and the high-frequency sub-band signal, that is, the low-frequency sub-band signal and the high-frequency sub-band signal obtained by decomposition can be two sub-band signals obtained by evenly dividing the frequency band of the audio signal, or can be two sub-band signals obtained by unequally dividing the frequency band of the audio signal. For example, if the effective bandwidth of the audio signal x(n) is 0-16kHz, then the low-frequency sub-band signal x(n) is LB (n) and the high frequency subband signal X HB The effective bandwidth of (n) is 0-8kHz and 8-16kHz respectively, and the low-frequency subband signal X LB (n) and the high frequency subband signal X HB The effective bandwidth of (n) may also be 0-6kHz and 6-16kHz respectively. In addition, the embodiment of the present application does not limit the number of divided frequency bands, that is, two sub-band signals may be obtained by even / non-even division, or more than two sub-band signals may be obtained by even / non-even division of the frequency band of the audio signal, such as 3, 4 or more sub-band signals.

[0122] As an example of subband decomposition, the audio signal is decomposed into low-frequency subband signals x by a QMF analysis filter. LB (n) and the high frequency subband signal X HB (n) Since the low-frequency sub-band signal has a greater impact on audio coding than the high-frequency sub-band signal, differential signal processing can be performed on the low-frequency sub-band signal and the high-frequency sub-band signal subsequently.

[0123] In some embodiments, step 1021B can be implemented by: sampling the audio signal to obtain a sampled signal, wherein the sampled signal includes multiple sample points obtained by sampling; low-pass filtering the sampled signal to obtain a low-pass filtered signal; down-sampling the low-pass filtered signal to obtain a low-frequency sub-band signal of the audio signal; high-pass filtering the sampled signal to obtain a high-pass filtered signal; down-sampling the high-pass filtered signal to obtain a high-frequency sub-band signal of the audio signal.

[0124] Here, the audio signal is a continuous analog signal, the sampling signal is a discrete digital signal, and the sampling point is a sampling value obtained by sampling the audio signal.

[0125] As an example, take an audio signal as an input signal with a sampling rate of Fs=32000 Hz as an example, sample the audio signal to obtain a sampled signal x(n) including 640 sample points. Call the analysis filter (2 channels) in the QMF filter bank, perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, perform high-pass filtering on the sampled signal to obtain a high-pass filtered signal, perform down-sampling on the low-pass filtered signal to obtain a low-frequency subband signal x of the audio signal. LB (n), downsample the high-pass filtered signal to obtain the high-frequency sub-band signal X of the audio signal HB (n). Low frequency subband signal X LB (n) and the high frequency subband signal X HB The effective bandwidth of (n) is 0-8kHz and 8-16kHz respectively, and the low-frequency subband signal X LB (n) and the high frequency subband signal X HB The number of sample points of (n) is 320.

[0126] It should be noted that the QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, which represent the QMF low-pass signal and the QMF high-pass signal respectively. After the low-pass signal and high-pass signal are restored by the decoding end, they are synthesized by the QMF synthesis filter, and the reconstructed signal with a sampling rate of Fs corresponding to the input signal can be restored.

[0127] In step 1022B, the low-frequency features of the low-frequency sub-band signal are extracted from the low-frequency sub-band signal by a second neural network.

[0128] It should be noted that step 1022B is similar to step 10221A, except that the processing object of step 10221A is an audio signal, and the processing object of step 1022B is a low-frequency subband signal. The structures of the first neural network and the second neural network are similar. For example, the second neural network is the same as the first neural network, and the second neural network includes 4 coding blocks, each coding block includes 4 or 5 residual units.

[0129] In some embodiments, step 1022B may be implemented by steps 10221B-10222B:

[0130] In step 10221B, feature extraction processing is performed on the low-frequency sub-band signal to obtain low-frequency features of the low-frequency sub-band signal.

[0131] Here, the embodiment of the present application can call a second neural network (NN) based on the low-frequency sub-band signal, and extract low-frequency features from the low-frequency sub-band signal through the second neural network, so as to continue to perform feature extraction based on important low-frequency features. It should be noted that the embodiment of the present application is not limited to the structure of the second NN, and the second NN can be a convolutional neural network, a deep neural network, etc.

[0132] In some embodiments, step 1022B may be implemented in the following manner: performing causal convolution processing on the low-frequency subband signal of the audio signal through a second neural network to obtain causal convolution features; and performing pooling processing on the causal convolution features to obtain low-frequency coding features of the low-frequency subband signal.

[0133] For example, see Fig.11 The network structure diagram of the second NN shown in FIG. 1 includes a causal convolution layer and a preprocessing layer. First, a 16-channel causal convolution layer is called to expand the input tensor (i.e., the low-frequency subband signal) to a 16×320 causal convolution feature. Then, the 16×320 causal convolution feature is preprocessed by the preprocessing layer. For example, after a convolution operation is performed on the 16×320 causal convolution feature, a pooling process with a factor of 2 is performed, and the activation function can be a parametric rectified linear unit (PReLU, Parametric Rectified Linear Unit) to generate a 16×160 tensor (i.e., low-frequency coding feature).

[0134] In step 10222B, residual processing is performed on the low-frequency coding features using at least one residual unit included in the second neural network to obtain low-frequency features of the low-frequency sub-band signal.

[0135] Here, residual processing is performed on the low-frequency coding features of the low-frequency sub-band signal to ensure that the low-frequency coding features are fully learned while better utilizing the shallow feature information of the low-frequency coding features to avoid missing the shallow feature information of the low-frequency coding features.

[0136] In some embodiments, step 10222B can be implemented through steps 102221B-102222B.

[0137] In step 102221B, feature residual processing is performed on the low-frequency features through at least one residual unit included in the second neural network to obtain residual features of the low-frequency sub-band signal.

[0138] In some embodiments, when at least one residual unit is one residual unit, step 102221B may be implemented in the following manner: performing residual processing on the low-frequency features through one residual unit to obtain the residual features of the low-frequency sub-band signal.

[0139] In some embodiments, when at least one residual unit is a plurality of cascaded residual units, step 102221B can be implemented in the following manner: performing residual processing on the low-frequency coding feature through the first residual unit of the plurality of cascaded residual units; outputting the residual result output by the first residual unit to the subsequent cascaded residual units, and continuing to perform residual processing and output residual results through the subsequent cascaded residual units; and using the residual result output by the last residual unit as the residual feature of the low-frequency sub-band signal.

[0140] For example, the second neural network includes 4 encoding blocks, each encoding block includes 4 or 5 residual units. Fig. 12A As shown, when at least one residual unit used for feature residual has 5 cascaded residual units, the first residual unit performs residual processing on the low-frequency coding feature, and outputs the residual result output by the first residual unit to the second residual unit; the second residual unit performs residual processing on the residual result output by the first residual unit, and outputs the residual result output by the second residual unit to the third residual unit; the third residual unit performs residual processing on the residual result output by the second residual unit, and outputs the residual result output by the third residual unit to the fourth residual unit; the fourth residual unit performs residual processing on the residual result output by the third residual unit, and outputs the residual result output by the fourth residual unit to the fifth residual unit; the fifth residual unit performs residual processing on the residual result output by the fourth residual unit to obtain the residual feature of the low-frequency sub-band signal.

[0141] The embodiments of the present application are not limited to the number of coding blocks in the neural network, and can be any positive integer such as 2, 3, 4, 5, etc. The embodiments of the present application are also not limited to the number of residual units in the coding block, and can be any positive integer such as 2, 3, 4, 5, 6, etc. The number of residual units in multiple coding blocks can be the same or different, for example, one coding block contains 4 residual units and another coding block contains 5 residual units.

[0142] In some embodiments, the processing process of the residual unit is as follows: the kth residual unit of multiple cascaded residual units performs the following processing: convolution processing is performed on the input of the kth residual unit to obtain the convolution result of the kth residual unit; the convolution result of the kth residual unit is added to the input of the kth residual unit to obtain the residual result output by the kth residual unit, wherein k is a successively increasing positive integer, 1≤k≤J, J is the number of residual units, when k is 1, the input of the kth residual unit is a low-frequency coding feature, and when k is not 1, the input of the kth residual unit is the residual result output by the residual feature k-1th residual unit.

[0143] Continuing with the above embodiment, each residual unit includes a dilated convolution operator; performing convolution processing on the input of the kth residual unit to obtain the convolution result of the kth residual unit can be achieved in the following manner: performing the following processing on the kth residual unit of multiple cascaded residual units: performing dilated convolution processing on the input of the kth residual unit to obtain the convolution result of the kth residual unit. That is, performing dilated convolution processing on the low-frequency coding features through the dilated convolution operator included in the first residual unit to obtain the dilated convolution result of the first residual unit. Performing the following processing on the jth residual unit of multiple cascaded residual units: performing dilated convolution processing on the residual result output by the j-1th residual unit through the dilated convolution operator included in the jth residual unit to obtain the dilated convolution result of the jth residual unit, wherein j is a positive integer that increases successively, 1<j≤J, and J is the number of residual units. It should be noted that each residual unit contains a dilation rate dilation operator. Using a dilation rate dilation operator is equivalent to using different receptive fields to extract input features at different resolutions, which can better analyze the data relatively comprehensively. After each residual unit is convolved by a dilation rate dilation operator, it is added to the shallow features from the jump connection (i.e., the input of each residual unit), thereby directly utilizing the shallow feature information, so that the network can make full use of the shallow feature information during the learning process.

[0144] Following the above embodiment, each residual unit includes not only a dilated convolution operator but also at least one causal convolution operator; after performing convolution processing on the input of the kth residual unit to obtain the convolution result of the kth residual unit, the obtained dilated convolution result is subjected to causal convolution processing by at least one causal convolution operator included in the kth residual unit, and the obtained causal convolution result is used as the convolution result of the kth residual unit. That is, the dilated convolution result of the first residual unit is subjected to causal convolution processing by at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result output by the first residual unit. The dilated convolution operator included in the jth residual unit performs dilated convolution processing on the residual result output by the j-1th residual unit to obtain the dilated convolution result of the jth residual unit. Then, the dilated convolution result of the jth residual unit is causally convolved by at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit. It should be noted that each residual unit also includes at least one causal convolution operator, and the local information of the features input to the causal convolution operator is continuously extracted through the causal convolution operator.

[0145] In some embodiments, grouped convolution can be applied to the convolution operator of the residual unit (including the hole convolution operator and the causal convolution operator). Grouped convolution is to divide the input channels into multiple groups for convolution operation, and only the input channels and output channels in each group are associated. It should be noted that when the input channels are divided into multiple groups, the corresponding output channels will also be divided into multiple groups, that is, the number of groups of input channels is the same as the number of groups of output channels, so that after the convolution is performed within the group, only the input channels and output channels in each group are associated. Here, it is assumed that the input channels of the features input to a certain convolution operator are 4 and the output channels are 4. If the number of groups is 1, each input channel is associated with 4 output channels. If the number of groups is 2, first divide the 4 input channels into two groups of 0-1 and 2-3. In the two groups, the input channels are associated with the output channels in the group. For example, the input channels 0-1 in the first group are associated with the output channels 0-1, and the input channels 2-3 in the second group are associated with the output channels 2-3. Fig. 6A As shown in , when the grouped convolution scheme is not adopted, each input channel is associated with 4 output channels; Figure 6B As shown in the figure, when the grouped convolution scheme is not adopted, the 0th output channel is only associated with the 0th-1st input channels, not with the 2nd-3rd input channels, and the 2nd output channel is only associated with the 2nd-3rd input channels, not with the 0th-1st input channels. From this comparison, it can be seen that the introduction of grouped convolution can avoid the association of any input channel with all output channels, reduce the number of connections, and reduce complexity.

[0146] For example, when grouped convolution is applied to the dilated convolution operator included in the residual unit, dilated convolution processing is performed on the low-frequency coding features, which can be achieved in the following manner: grouping the input channels of the low-frequency coding features to obtain multiple groups, wherein each group includes the first elements corresponding to at least two channels in the low-frequency coding features; and performing dilated convolution processing on the first elements in each group. When grouped convolution is applied to the causal convolution operator included in the residual unit, causal convolution processing is performed on the obtained dilated convolution results, which can be achieved in the following manner: grouping the input channels of the dilated convolution results to obtain multiple groups, wherein each group includes the second elements corresponding to at least two channels in the dilated convolution results; and performing causal convolution processing on the second elements in each group.

[0147] Following step 102221B, in step 102222B, feature encoding processing is performed on the residual features to obtain low-frequency features of the low-frequency sub-band signal.

[0148] Here, feature encoding processing is performed on the residual features to obtain low-frequency features of the low-frequency subband signal, so that signal encoding processing can be performed based on the low-frequency features to obtain a low-frequency code stream of the audio signal.

[0149] In some embodiments, step 102222B can be implemented by: performing convolution processing on the residual features to obtain convolution features, wherein the number of channels of the convolution features is greater than the number of channels of the residual features; and performing pooling processing on the convolution features to obtain low-frequency features of the low-frequency sub-band signal.

[0150] For example, the second NN is called based on the low-frequency sub-band signal, and after being processed by the residual unit in the second NN, the residual feature is obtained, and then the residual feature is convolved through the convolution layer in the second NN to increase the number of channels of the residual feature; finally, the convolution feature is pooled through the pooling layer in the second NN to obtain the low-frequency feature of the low-frequency sub-band signal. Of course, the second NN can also include a causal convolution layer, which performs causal convolution on the low-frequency feature to obtain the low-frequency feature after causal convolution, and performs signal coding processing on the low-frequency feature of the low-frequency sub-band signal after causal convolution to obtain the low-frequency code stream of the audio signal.

[0151] For example, Fig.11 As shown, after the second NN is called based on the low-frequency subband signal, the low-frequency coding feature ( Fig.11The second NN includes 4 cascaded coding blocks with different downsampling factors (Down factor). Each coding block includes a residual block (including at least one residual unit), a convolution layer and a pooling layer, wherein each residual block includes 5 residual units (ResidualUnit) based on dilated convolution (the input and output feature dimensions of the residual unit will not change); a convolution layer is used to double the number of input channels, and the activation function can be PReLU, so as to ensure the data volume and avoid data loss; the pooling layer is a pooling operation including Down factor to complete downsampling and realize data compression. Here, the Down factors of the 4 coding blocks are set to 2, 4, 4, and 5 respectively. Therefore, the number of output channels of the 4 coding blocks is set to 32, 64, 128, and 256 respectively. After being processed by the 4 coding blocks, the input 16×160 tensor is converted into 32×80, 64×20, 128×5, and 256×1 tensors respectively. For example, the residual block in the first coding block is used to perform residual processing on the low-frequency coding features (i.e., a 16×160 tensor), and the residual result output by the residual block in the first coding block is output to the feature coding block in the first coding block. After being processed by the feature coding block in the first coding block (including a convolution layer and a pooling layer), the encoding result of the feature coding block in the first coding block (i.e., a 32×80 tensor) is obtained, and the encoding result of the feature coding block in the first coding block (i.e., a 32×80 tensor) is output to the second coding block; the residual block in the second coding block is used to perform residual processing on the first The encoding result of the feature encoding block in the encoding block (i.e., a 32×80 tensor) is subjected to residual processing, and the residual result output by the residual block in the second encoding block is output to the feature encoding block in the second encoding block. After being processed by the feature encoding block in the second encoding block (including a convolution layer and a pooling layer), the encoding result of the feature encoding block in the second encoding block (i.e., a 64×20 tensor) is obtained, and the encoding result of the feature encoding block in the first encoding block (i.e., a 64×20 tensor) is output to the third encoding block; the above processing is performed in sequence, and the output of the last encoding block is used as a low-frequency feature. Among them, the embodiment of the present application is not limited to the number of encoding blocks, which can be any positive integer such as 2, 3, 4, 5, etc.

[0152] In step 1023B, high frequency analysis processing is performed on the high frequency sub-band signal to obtain high frequency characteristics of the high frequency sub-band signal.

[0153] In some embodiments, a fifth neural network may be called to extract high-frequency features of the high-frequency sub-band signal from the high-frequency sub-band signal, and the structure of the fifth neural network is similar to that of the second neural network, wherein the dimension of the high-frequency features is smaller than the dimension of the low-frequency features.

[0154] In some embodiments, high-frequency analysis processing is performed on the high-frequency sub-band signal to obtain the high-frequency characteristics of the high-frequency sub-band signal, which can be achieved by: performing frame processing on the high-frequency sub-band signal to obtain multiple sub-frames of the high-frequency sub-band signal; performing frequency band expansion processing on each sub-frame to obtain the sub-band spectrum envelope of each sub-frame; and using the sub-band spectrum envelopes corresponding to the multiple sub-frames as the high-frequency characteristics of the high-frequency sub-band signal. The number of the multiple sub-frames can be 2, 4, 6, etc., and the embodiments of the present application are not limited to the number of sub-frames.

[0155] Here, since the high-frequency subband signal is relatively less important to quality than the low-frequency subband signal, the high-frequency subband signal can be compressed by another method, namely, band expansion (recovering the broadband speech signal from the band-limited narrowband speech signal) to quickly compress the high-frequency subband signal and extract the high-frequency features of the high-frequency subband signal.

[0156] In some embodiments, frequency band expansion processing is performed on each subframe to obtain a subband spectrum envelope of each subframe, which can be achieved in the following way: frequency domain transform processing is performed based on multiple sample points included in the subframe to obtain transform coefficients corresponding to the multiple sample points; the transform coefficients corresponding to the multiple sample points are divided into multiple subbands; mean processing is performed on the transform coefficients included in each subband to obtain the average energy corresponding to each subband, and the average energy is used as the subband spectrum envelope corresponding to each subband.

[0157] It should be noted that the frequency domain transformation method of the embodiment of the present application includes modified discrete cosine transform (MDCT, Modified Discrete Cosine Transform), discrete cosine transform (DCT, Discrete Cosine Transform), fast Fourier transform (FFT, Fast Fourier Transform), etc., and the embodiment of the present application is not limited to the method of frequency domain transformation. The mean processing of the embodiment of the present application includes arithmetic mean and geometric mean, and the embodiment of the present application is not limited to the method of mean processing.

[0158] In some embodiments, frequency domain transform processing is performed based on multiple sample points included in a subframe to obtain transform coefficients corresponding to the multiple sample points, including: performing the following processing for each subframe: obtaining an adjacent subframe adjacent to the subframe; based on the multiple sample points included in the adjacent subframe and the multiple sample points included in the subframe, performing discrete cosine transform processing on the multiple sample points included in the subframe to obtain transform coefficients corresponding to the multiple sample points included in the subframe.

[0159] As an example, for a high frequency subband signal X including 320 points HB (n), the 320-point high-frequency subband signal XHB (n), divided into two subframes (the first subframe and the second subframe), each subframe includes 160 points. For any subframe, that is, the high-frequency subband signal including 160 points, call the modified discrete cosine transform (MDCT) to generate 160-point MDCT coefficients (that is, the transform coefficients corresponding to the multiple sample points included in the subframe). Specifically, if it is 50% overlap, the first subframe of the n+1th frame can be merged (spliced) with the second subframe of the nth frame, and the 320-point MDCT is calculated to obtain the 160-point MDCT coefficients of the first subframe of the n+1th frame; the second subframe of the n+1th frame is merged with the first subframe of the n+1th frame, and the 320-point MDCT is calculated to obtain the 160-point MDCT coefficients of the second subframe of the n+1th frame.

[0160] In some embodiments, the process of performing geometric mean processing on the transform coefficients included in each sub-band is as follows: determining the sum of squares of the transform coefficients corresponding to the sample points included in each sub-band; and determining the ratio of the sum of squares to the number of sample points included in the sub-band to obtain the average energy corresponding to each sub-band.

[0161] Following step 1023B, in step 1024B, the low-frequency features and the high-frequency features are determined as coding features of the audio signal.

[0162] Following step 102, in step 103, signal encoding processing is performed on the encoding feature of the audio signal through the target bit rate mode to obtain an audio bit stream of the audio signal.

[0163] In some embodiments, when the target coding mode is a broadband coding mode, the target bit rate mode is used to indicate the use of a target code table to perform signal encoding on the coding features of the audio signal; step 103 can be implemented in the following manner: using the target code table, quantizing the coding features of the audio signal to obtain a quantized value of the coding features; performing entropy coding on the quantized value using the target bit rate corresponding to the target code table to obtain an audio bit stream of the audio signal.

[0164] like Figure 4E As shown, Figure 4E is a flow chart of an audio encoding method provided by an embodiment of the present application. When the target encoding mode is an ultra-wideband encoding mode, the target bit rate mode includes an indication of using a first code table to encode a low-frequency feature signal and using a second code table to encode a high-frequency feature signal; Figure 4E Shown in Figure 4AStep 103 in the above method can be implemented through steps 1031-1033: in step 1031, the low-frequency features are quantized by the first code table to obtain the quantized value of the low-frequency features, and the quantized value of the low-frequency features is entropy coded by the code rate corresponding to the first code table to obtain the low-frequency code stream of the low-frequency subband signal; in step 1032, the high-frequency features are quantized by the second code table to obtain the quantized value of the high-frequency features, and the quantized value of the high-frequency features is entropy coded by the code rate corresponding to the second code table to obtain the high-frequency code stream of the high-frequency subband signal; in step 1033, the audio code stream of the audio signal is constructed based on the low-frequency code stream and the high-frequency code stream. The quantization degree (i.e., quantization accuracy) corresponding to different code tables may be different. For example, the quantization accuracy of the first code table is greater than that of the second code table. Since the value interval of each dimension in the low-frequency feature is [-1, 1], the interval of [-1, 1] can be divided into 11 equal parts to form a first code table containing 11 elements, and the low-frequency feature is quantized by the first code table; since the value interval of each dimension in the high-frequency feature is [-1, 1], the interval of [-1, 1] can be divided into 5 equal parts to form a second code table containing 5 elements, and the low and high frequency features are quantized by the second code table.

[0165] Here, for the low-frequency feature F of the low-frequency subband signal LB (n) and the high-frequency coding feature F of the high-frequency sub-band signal HB (n) A method of scalar quantization (individual quantization of each component) and entropy coding can be performed. In addition, the embodiment of the present application does not limit the technical combination of vector quantization (combining multiple adjacent components into a vector for joint quantization) and entropy coding.

[0166] It should be noted that when the target bit rate mode only includes an instruction to use the first code table to encode the low-frequency features and the second code table to encode the high-frequency features, the low-frequency code stream and the high-frequency code stream are combined to obtain the audio code stream of the audio signal.

[0167] In some embodiments, the target bit rate mode also includes an instruction to use a third code table to perform residual encoding on the residual features of the high-frequency features; before step 1033, determine the subband corresponding to the subframe of the high-frequency subband signal, and divide the subband into multiple subsets; perform mean processing on the transform coefficients included in each subset to obtain the average energy corresponding to each subset, and use the average energy as the envelope value corresponding to each subset; in the quantization value of the high-frequency feature, determine the quantization value of the subband spectrum envelope of the subband corresponding to the subset, and use the difference between the envelope value corresponding to the subset and the quantization value of the subband spectrum envelope as the first residual value; based on the first residual value, determine the residual features of the high-frequency feature; through the third code table, quantize the residual features of the high-frequency feature to obtain the quantization value of the residual features of the high-frequency feature, and perform entropy coding on the quantization value of the residual features of the high-frequency feature through the code rate corresponding to the third code table to obtain the residual code stream of the residual feature; correspondingly, step 1033 can be implemented in the following way: use the low-frequency code stream, the high-frequency code stream and the residual code stream as the audio code stream of the audio signal. The quantization degree of the third code table may be different from the quantization degrees of the first code table and the second code table.

[0168] In some embodiments, when the number of residual values ​​is N, N is a positive integer greater than 1, and based on the first residual value, determining the residual features of the high-frequency features can be achieved in the following manner: determining the quantization value of the nth residual value, and determining the sum of the quantization value of the nth residual value and the quantization value of the subband spectrum envelope of the corresponding subband; taking the difference between the envelope value corresponding to the subset and the sum as the n+1th residual value; determining the N residual values ​​as the residual features of the high-frequency features; wherein n is a successively increasing positive integer, 1≤n≤N.

[0169] Taking the number of residual values ​​as 2 as an example, the residual value is determined in the following way:

[0170] 1) For each 10ms subframe, the subframe is divided into 4 subbands, and each subband is further evenly divided into 2 subsets.

[0171] For example, the set of envelope values ​​of four sub-bands is {e1, e2, e3, e4}.

[0172] 2) For each subset of each subband, calculate the envelope value of the subset.

[0173] For example, the set of envelope values ​​of all subsets is {e 1a , e 1b , e 2a , e 2b , e 3a , e 3b , e 4a , e 4b}.

[0174] 3) Subtract the corresponding high-frequency feature vector quantization value (ie, the quantization value of the high-frequency feature) from the envelope value of each subset to obtain the first high-frequency feature vector residual (referred to as the first residual value).

[0175] For example, the high-frequency feature vector quantization value is Then the first high-frequency eigenvector residual is

[0176] 4) The first high-frequency feature residual is subjected to similar quantization and entropy coding to obtain a first high-frequency feature vector residual quantization value (hereinafter referred to as the quantization value of the first residual value).

[0177] Here, considering that the dynamic range of the residual value is much lower than the original value, only 2 bits need to be allocated for scalar quantization of each residual. In this way, a reference bit rate of 2*16*50=1.6kbps is required (similarly, because it is non-uniformly distributed entropy coding, the actual bit rate is generally less than 1.6kbps) to obtain the residual quantization value of the first high-frequency eigenvector.

[0178] 5) Subtract the sum of the corresponding high-frequency feature vector quantization value and the first high-frequency feature vector residual quantization value from the envelope value of each subset to obtain the second high-frequency feature vector residual (referred to as the second residual value).

[0179] In step 104, a frame header is determined based on a target coding mode and a target bit rate mode.

[0180] Here, an empty frame header can be constructed first, and the empty frame header includes a coding mode bit and a rate mode bit, and the target coding mode is written into the coding mode bit included in the empty frame header, and the target rate mode is written into the rate mode bit included in the empty frame header to determine the frame header. The embodiment of the present application is not limited to the number of bits of the frame header, nor is it limited to the position of the target coding mode and the target rate mode in the frame header. For example, the target coding mode can be written into the first bit of the frame header, or the last bit of the frame header, and is not limited to the number of bits occupied by the coding mode bit and the rate mode bit in the frame header.

[0181] In step 105, an audio code stream package of the audio signal is generated based on the audio code stream and the frame header.

[0182] In some embodiments, before step 105, the flatness side information of the high frequency sub-band signal is determined; therefore, step 105 can be implemented by combining the audio code stream, the frame header and the flatness side information to obtain the audio code stream encapsulation of the audio signal.

[0183] For example, the frame header is set at the first position of the audio code stream encapsulation, and the audio code stream and flatness edge information are set after the frame header. The embodiment of the present application is not limited to the order of the audio code stream and the flatness edge information.

[0184] In some embodiments, determining the flatness side information of a high-frequency subband signal may be achieved in the following manner: dividing the transform coefficients included in a subframe of the high-frequency subband signal into a plurality of blocks; determining a first flatness of each block, and determining a second flatness of a low-frequency designated frequency band of the audio signal; when the first flatness is less than the second flatness, or the first flatness is less than a flatness threshold, setting the flatness side information to a first value, wherein the first value indicates that flattening processing is required during audio decoding; when the first flatness is greater than or equal to the second flatness, and the first flatness is greater than or equal to the flatness threshold, setting the flatness side information to a second value, wherein the second value indicates that flattening processing is not required during audio decoding.

[0185] For example, here the first value may be 1 and the second value may be 0; the first value may also be 0 and the second value may be 1.

[0186] In some embodiments, the frame header further includes at least one channel bit, and the channel bit is used to indicate that the audio signal is encoded by a mono encoding method or a stereo encoding method. When the audio signal is a stereo signal after downmixing the input signal of the left channel and the input signal of the right channel, the channel bit in the frame header is set to a first value (e.g., 1), and the first value indicates that the audio signal is encoded by a stereo encoding method. The input signal of the left channel and the input signal of the right channel are subjected to parameter stereo encoding to obtain a stereo feature vector, and then an audio code stream encapsulation of the audio signal is generated based on the stereo feature vector, the audio code stream and the frame header; when the audio signal is a mono input signal, the channel bit in the frame header is set to a second value (e.g., 0), and the second value indicates that the audio signal is encoded by a mono encoding method, and an audio code stream encapsulation of the audio signal is generated based on the audio code stream and the frame header.

[0187] As mentioned above, the audio decoding method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 5A , Figure 5A The flowchart of the audio decoding method provided by the embodiment of the present application is as follows. The audio decoding function is realized by the audio decoding method. The audio decoding method and the above-mentioned audio encoding method are inverse processes of each other. Figure 5A The steps shown are explained.

[0188] In step 201, in response to a decoding request for an audio code stream encapsulation, a target coding mode and a target bit rate mode are obtained from a frame header included in the audio code stream encapsulation.

[0189] The audio code stream included in the audio code stream encapsulation is obtained by audio encoding the audio signal through a target coding mode and a target bit rate mode, the target coding mode is obtained from multiple coding modes, and the target bit rate mode is obtained from multiple bit rate modes.

[0190] Here, the decoding request is used to instruct to perform audio decoding processing on the audio signal.

[0191] As an example, by Figure 4A After the audio encoding method shown encodes the audio code stream package, the audio code stream package is transmitted to the decoding end. After the decoding end receives the audio code stream package, the audio decoding process is performed on the audio code stream package. First, the target encoding mode and the target bit rate mode are obtained from the frame header included in the audio code stream package, for example, the target encoding mode is obtained from the encoding mode bit included in the frame header, and the target bit rate mode is obtained from the bit rate mode bit included in the frame header.

[0192] In step 202, the audio bitstream is subjected to signal decoding processing by using the target coding mode and the target bit rate mode to obtain a coding feature estimation value corresponding to the audio bitstream.

[0193] It should be noted that signal decoding is the inverse process of signal encoding. Since the process of decoding the received code stream by the decoding end is the inverse process of the encoding process by the encoding end, the value generated in the decoding process is an estimate relative to the value in the encoding process, for example, the estimated value of the coding feature generated in the decoding process is an estimate relative to the coding feature in the encoding process.

[0194] In some embodiments, the target coding mode is a broadband coding mode, and the target bit rate mode is used to indicate the use of a target code table to perform signal encoding on the coding features of the audio signal; step 202 can be implemented in the following manner: performing entropy decoding processing on the audio bit stream according to the target bit rate corresponding to the target code table to obtain a quantization value corresponding to the audio bit stream; performing inverse quantization processing on the quantization value corresponding to the audio bit stream according to the target code table to obtain an estimated value of the coding features corresponding to the audio bit stream.

[0195] See also Figure 5B , Figure 5B is a flow chart of an audio decoding method provided in an embodiment of the present application, wherein the target coding mode is an ultra-wideband coding mode, the target bit rate mode includes an indication of using a first code table to encode a low-frequency feature of an audio signal, and using a second code table to encode a high-frequency feature of an audio signal, and the audio bit stream includes a low-frequency bit stream and a high-frequency bit stream; Figure 5BIt is shown that step 202 can be implemented through steps 2021-2024: in step 2021, a low-frequency code stream and a high-frequency code stream are obtained from the audio code stream through the target coding mode; in step 2022, the low-frequency code stream is entropy decoded through the code rate corresponding to the first code table to obtain a quantization value corresponding to the low-frequency code stream, and the quantization value corresponding to the low-frequency code stream is inversely quantized through the first code table to obtain a low-frequency feature estimation value corresponding to the low-frequency code stream; in step 2023, the high-frequency code stream is entropy decoded through the code rate corresponding to the second code table to obtain a quantization value corresponding to the high-frequency code stream, and the quantization value corresponding to the high-frequency code stream is inversely quantized through the second code table to obtain a high-frequency feature estimation value corresponding to the high-frequency code stream; in step 2024, the coding feature estimation value corresponding to the audio code stream is determined based on the low-frequency feature estimation value and the high-frequency feature estimation value.

[0196] For example, for a low-frequency code stream, firstly, entropy decoding is performed on the low-frequency code stream using the code rate corresponding to the first code table to obtain the quantization value corresponding to the low-frequency code stream, and then the low-frequency feature estimation value F′ corresponding to the low-frequency code stream is obtained by looking up the first code table. LB (n); For the high-frequency code stream, firstly perform entropy decoding on the high-frequency code stream according to the code rate corresponding to the second code table to obtain the quantization value corresponding to the high-frequency code stream, and obtain the high-frequency feature estimation value F′ corresponding to the high-frequency code stream by looking up the second code table HB (n). It should be noted that, when the target bit rate mode only includes instructions for using the first code table to encode the low-frequency features of the audio signal and using the second code table to encode the high-frequency features of the audio signal, the low-frequency feature estimation value and the high-frequency feature estimation value are determined as the coding feature estimation value corresponding to the audio bit stream, wherein the high-frequency feature estimation value is used to participate in the high-frequency reconstruction process to generate a high-frequency subband signal estimation value.

[0197] In some embodiments, the audio code stream also includes a residual code stream, and the target rate mode also includes an instruction to use a third code table to perform residual coding on the residual features of the high-frequency features corresponding to the audio code stream; before step 2024, the residual code stream is entropy decoded using the code rate corresponding to the third code table to obtain a quantization value corresponding to the residual code stream, and the quantization value corresponding to the residual code stream is inversely quantized using the third code table to obtain a residual feature estimation value corresponding to the residual code stream. Correspondingly, step 2024 can be implemented in the following manner: the sum of the high-frequency feature estimation value and the residual feature estimation value is determined as the final high-frequency feature estimation value; the low-frequency feature estimation value and the final high-frequency feature estimation value are determined as the coding feature estimation value corresponding to the audio code stream.

[0198] The ith value in the high-frequency feature estimate is added to the 2i-1th value and the 2ith value in the residual feature estimate to obtain the 2i-1th value and the 2ith value in the high-frequency feature final estimate. For example, the high-frequency feature estimate is F′ HB (n) = [e′1, e′2, ..., e′8], the residual feature estimate is F′C(n) = [e′ 1C , e′ 2C , ..., e′ 16C ], then the final estimated value of the high-frequency feature is F′ FHB (n) = [e′1+e′ 1C , e′1+e′ 2C , ..., e′8+e′ 16C ].

[0199] In step 203, the coding feature estimation value corresponding to the audio bitstream is reconstructed through the target coding mode to obtain a synthetic audio signal corresponding to the audio bitstream.

[0200] It should be noted that the reconstruction process is the inverse process of the extraction process. Since the decoding process of the received bitstream by the decoding end is the inverse process of the encoding process by the encoding end, the value generated in the decoding process is an estimated value relative to the value in the encoding process. For example, the synthesized audio signal generated in the decoding process is an estimated value relative to the audio signal in the encoding process.

[0201] In some embodiments, the target coding mode is a broadband coding mode, and step 203 can be implemented in the following manner: calling a third neural network based on the broadband coding mode; reconstructing the coding feature estimation value corresponding to the audio bit stream through the third neural network to obtain a synthetic audio signal corresponding to the audio bit stream.

[0202] It should be noted that the reconstruction process through the third neural network is the inverse process of the extraction process through the first neural network.

[0203] As an example, reconstructing the estimated coding feature value corresponding to the audio code stream to obtain the synthetic audio signal corresponding to the audio code stream can be achieved in the following manner: using at least one residual unit included in the third neural network to perform residual processing on the estimated coding feature value corresponding to the audio code stream to obtain the audio feature estimated value corresponding to the audio code stream; performing feature reconstruction processing on the audio feature estimated value corresponding to the audio code stream to obtain the synthetic audio signal corresponding to the audio code stream. The third neural network includes 4 decoding blocks, and each decoding block includes 4 or 5 residual units.

[0204] The embodiments of the present application are not limited to the number of decoding blocks, which can be any positive integer such as 2, 3, 4, 5, etc. The embodiments of the present application are not limited to the number of residual units in the decoding block, which can be any positive integer such as 2, 3, 4, 5, 6, etc. Figure 5C , Figure 5C is a flow chart of an audio decoding method provided in an embodiment of the present application, wherein the target coding mode is an ultra-wideband coding mode, Figure 5C It is shown that step 203 can be implemented through steps 2031-2033:

[0205] In step 2031, the low-frequency feature estimation value included in the coding feature is subjected to feature reconstruction processing by a fourth neural network, so as to obtain a low-frequency subband signal estimation value corresponding to the audio bitstream. The structure of the fourth neural network is similar to that of the third neural network.

[0206] In some embodiments, step 2031 can be implemented through steps 20311-20312.

[0207] In step 20311, residual processing is performed on the low-frequency feature estimation value using at least one residual unit included in the fourth neural network to obtain a low-frequency coding feature estimation value. The fourth neural network includes 4 decoding blocks, and each decoding block includes 4 or 5 residual units.

[0208] In some embodiments, step 20311 can be implemented by steps 203111-203112:

[0209] In step 203111, feature decoding processing is performed on the low-frequency feature estimation value to obtain residual features.

[0210] For example, feature decoding is the inverse process of feature encoding, and feature decoding is performed on the low-frequency feature estimation value to obtain the residual feature (an estimation value).

[0211] In some embodiments, step 203111 can be implemented in the following manner: performing convolution processing on the low-frequency features to obtain convolution features, wherein the number of channels of the convolution features is smaller than the number of channels of the low-frequency features; and performing upsampling processing on the convolution features to obtain residual features.

[0212] Of course, before step 203111, causal convolution may be performed on the low-frequency features to obtain the low-frequency features after causal convolution, and step 203111 may be executed based on the low-frequency features after causal convolution, that is, feature decoding processing may be performed on the low-frequency features after causal convolution to obtain residual features.

[0213] In step 203112, feature residual processing is performed on the residual features through at least one residual unit included in the fourth neural network to obtain a low-frequency coding feature estimation value.

[0214] Here, residual processing is performed on the residual features to ensure that the residual features are fully learned while better utilizing the shallow feature information of the residual features to avoid missing the shallow feature information.

[0215] In some embodiments, when at least one residual unit is a plurality of cascaded residual units, step 203112 may be implemented in the following manner: residual processing is performed on the residual feature through one residual unit to obtain a low-frequency coding feature estimation value.

[0216] In some embodiments, when at least one residual unit is a plurality of cascaded residual units, step 203112 can be implemented in the following manner: performing residual processing on the residual feature through the first residual unit of the plurality of cascaded residual units; outputting the residual result output by the first residual unit to the subsequent cascaded residual units, and continuing to perform residual processing and output residual results through the subsequent cascaded residual units; and using the residual result output by the last residual unit as a low-frequency coding feature estimate.

[0217] In some embodiments, the processing process of the residual unit is as follows: the kth residual unit of multiple cascaded residual units performs the following processing: convolution processing is performed on the input of the kth residual unit to obtain the convolution result of the kth residual unit; the convolution result of the kth residual unit is added to the input of the kth residual unit to obtain the residual result output by the kth residual unit, wherein k is a successively increasing positive integer, 1≤k≤J, J is the number of residual units, when k is 1, the input of the kth residual unit is the residual feature, when k is not 1, the input of the kth residual unit is the residual result output by the k-1th residual unit.

[0218] Continuing with the above embodiment, each residual unit includes a dilated convolution operator; performing convolution processing on the input of the kth residual unit to obtain the convolution result of the kth residual unit can be achieved in the following manner: performing the following processing on the kth residual unit of multiple cascaded residual units: performing dilated convolution processing on the input of the kth residual unit to obtain the convolution result of the kth residual unit. That is, performing dilated convolution processing on the residual features through the dilated convolution operator included in the first residual unit to obtain the dilated convolution result of the first residual unit. Performing the following processing on the jth residual unit of multiple cascaded residual units: performing dilated convolution processing on the residual result output by the j-1th residual unit through the dilated convolution operator included in the jth residual unit to obtain the dilated convolution result of the jth residual unit, wherein j is a positive integer that increases successively, 1<j≤J, and J is the number of residual units.

[0219] Following the above embodiment, each residual unit includes not only a dilated convolution operator but also at least one causal convolution operator; after performing convolution processing on the input of the kth residual unit to obtain the convolution result of the kth residual unit, the obtained dilated convolution result is subjected to causal convolution processing by at least one causal convolution operator included in the kth residual unit, and the obtained causal convolution result is used as the convolution result of the kth residual unit. That is, the dilated convolution result of the first residual unit is subjected to causal convolution processing by at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result output by the first residual unit. The residual result output by the j-1th residual unit is subjected to dilated convolution processing through the dilated convolution operator included in the jth residual unit. After obtaining the dilated convolution result of the jth residual unit, the dilated convolution result of the jth residual unit is subjected to causal convolution processing through at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit.

[0220] In some embodiments, when grouped convolution is applied to the dilated convolution operator included in the residual unit, dilated convolution processing is performed on the low-frequency features, which can be achieved by: grouping the input channels of the residual features to obtain multiple groups, wherein each group includes the first elements corresponding to at least two channels in the residual features; and performing dilated convolution processing on the first elements in each group. When grouped convolution is applied to the causal convolution operator included in the residual unit, causal convolution processing is performed on the obtained dilated convolution results, which can be achieved by: grouping the input channels of the dilated convolution results to obtain multiple groups, wherein each group includes the second elements corresponding to at least two channels in the dilated convolution results; and performing causal convolution processing on the second elements in each group.

[0221] In some embodiments, the fourth neural network for audio decoding includes multiple cascaded decoding blocks, each decoding block includes a feature decoding block and at least one residual unit; step 203111 can be implemented by step 203111A, and step 203112 can be implemented by step 203112A: in step 203111A, feature decoding processing is performed on low-frequency features through feature decoding blocks in multiple cascaded decoding blocks to obtain residual features; correspondingly, in step 203112A, residual processing is performed on the residual features through at least one residual unit in multiple cascaded decoding blocks to obtain low-frequency coding feature estimates.

[0222] In some embodiments, step 203111A can be implemented in the following manner: perform feature decoding processing on low-frequency features through a feature decoding block in the first decoding block of multiple cascaded decoding blocks, and output the decoding result output by the feature decoding block in the first decoding block to at least one residual unit in the first decoding block; perform feature decoding processing on the residual result output by at least one residual unit in the i-1th decoding block through a feature decoding block in the i-th decoding block of multiple cascaded decoding blocks, and output the decoding result output by the feature decoding block in the i-th decoding block to at least one residual unit in the i-th decoding block; use the decoding result output by the feature decoding block in the last decoding block as the residual feature; wherein i is a successively increasing positive integer, 1<i≤I, and I is the number of decoding blocks. The decoding result output by the feature decoding block in the first decoding block is obtained by performing the following processing on the feature decoding block in the first decoding block of the multiple cascaded decoding blocks: convolution processing is performed on the low-frequency features to obtain convolution features, wherein the number of channels of the convolution features is less than the number of channels of the low-frequency features; upsampling processing is performed on the convolution features to obtain the decoding result output by the feature decoding block in the first decoding block. The decoding result output by the feature decoding block in the i-th decoding block is obtained by performing the following processing on the feature decoding block in the i-th decoding block: convolution processing is performed on the residual result output by at least one residual unit in the i-1-th decoding block to obtain convolution features, wherein the number of channels of the convolution features is less than the number of channels of the residual result output by at least one residual unit; upsampling processing is performed on the convolution features to obtain the decoding result output by the feature decoding block in the i-th decoding block.

[0223] As an example, the fourth neural network includes 4 decoding blocks, each decoding block includes 4 or 5 residual units.

[0224] In some embodiments, step 203112A may be implemented in the following manner: performing residual processing on the residual feature through at least one residual unit in the last decoding block of a plurality of cascaded decoding blocks to obtain a low-frequency coding feature estimation value.

[0225] In step 2032, high-frequency reconstruction processing is performed on the high-frequency feature estimation value included in the coding feature to obtain a high-frequency sub-band signal estimation value corresponding to the audio code stream.

[0226] It should be noted that the feature reconstruction at the decoding end is the inverse process of the extraction process at the encoding end, and the high-frequency reconstruction at the decoding end is the inverse process of the high-frequency analysis at the encoding end.

[0227] In some embodiments, high-frequency feature estimation values ​​included in the coding features are subjected to high-frequency reconstruction processing to obtain high-frequency sub-band signal estimation values ​​corresponding to the audio code stream. This can be achieved by calling a sixth neural network to perform feature reconstruction processing on the high-frequency feature estimation values ​​included in the coding features to obtain high-frequency sub-band signal estimation values ​​corresponding to the audio code stream, wherein the structure of the sixth neural network is similar to that of the third neural network.

[0228] In some embodiments, step 2032 may be implemented by steps 20321 to 20324:

[0229] In step 20321, frequency domain transform processing is performed on the first half sample points and the second half sample points included in the low-frequency subband signal estimation value to obtain first transform coefficients corresponding to the first half sample points and second transform coefficients corresponding to the second half sample points.

[0230] It should be noted that the frequency domain transformation method of the embodiments of the present application includes modified discrete cosine transform (MDCT), discrete cosine transform (DCT), fast Fourier transform (FFT), etc. The embodiments of the present application are not limited to the frequency domain transformation method.

[0231] In step 20322, based on the first transform coefficient, the high-frequency feature estimation value is subjected to inverse frequency band expansion processing to obtain a first high-frequency sub-band signal estimation value.

[0232] In some embodiments, step 20322 can be implemented in the following manner: performing spectrum replication processing on the transform coefficients of the second half of the first transform coefficients to obtain first reference transform coefficients of a reference high-frequency sub-band signal; performing gain processing on the first reference transform coefficient based on the first half of the sub-band spectrum envelope corresponding to the high-frequency feature estimation value to obtain the gained first reference transform coefficient; performing inverse frequency domain transform processing on the gained first reference transform coefficient to obtain a first high-frequency sub-band signal estimation value.

[0233] In some embodiments, based on the first half subband spectral envelope corresponding to the high-frequency feature estimation value, the first reference transform coefficient is gain processed to obtain the first reference transform coefficient after gain, including: based on the first half subband spectral envelope, the first reference transform coefficient of the reference high-frequency subband signal is divided into multiple first subbands; for any first subband among the multiple first subbands, the following processing is performed: determining the first average energy corresponding to the first subband in the first half subband spectral envelope, and determining the second average energy corresponding to the first subband; determining the gain factor based on the ratio of the first average energy to the second average energy; multiplying the gain factor by each first reference transform coefficient included in the first subband to obtain the first reference transform coefficient after gain.

[0234] As an example, firstly, the low frequency subband signal estimation value x′ generated by the decoding end is LB (n), two 320-point MDCT transforms are also performed to generate two groups of 160-point MDCT coefficients (i.e., the first transform coefficients corresponding to the sample points in the first half and the second transform coefficients corresponding to the sample points in the second half).

[0235] Then, for the MDCT coefficients of the first group of 160 points (i.e., the first transform coefficients corresponding to the sample points in the first half), x′ LB (n) The 160-point MDCT coefficients generated are copied to generate the MDCT coefficients of the high-frequency part. Referring to the basic characteristics of the speech signal, the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid simple copying, which causes the artificially generated MDCT spectrum of the high-frequency part to contain too many harmonics, the last 80 points of the 160-point MDCT coefficients that the low-frequency subband depends on can be used as the master, and the spectrum can be copied twice to generate the reference value of the MDCT coefficient of the high-frequency subband signal of 160 points (i.e., the first reference transform coefficient of the reference high-frequency subband signal).

[0236] Next, the 8 sub-band spectral envelopes obtained previously are called (i.e., the 8 sub-band spectral envelopes obtained by querying the quantization table, i.e., the sub-band spectral envelopes corresponding to the high-frequency features). These 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands, and the reference values ​​of the MDCT coefficients of the generated 160-point reference high-frequency sub-band signal are divided into 8 reference high-frequency sub-bands (i.e., the first reference transform coefficients of the reference high-frequency sub-band signal are divided into multiple first sub-bands). In a band-by-band manner, based on a high-frequency sub-band and the corresponding reference high-frequency sub-band, the reference values ​​of the MDCT coefficients of the generated 320-point reference high-frequency sub-band signal are gain controlled (multiplication in the frequency domain). For example, a gain factor is calculated based on the average energy of the high-frequency sub-band (i.e., the first average energy) and the average energy of the corresponding reference high-frequency sub-band (the second average energy), and the MDCT coefficient corresponding to each point in the corresponding reference high-frequency sub-band is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to the original coefficient energy at the encoding end.

[0237] For example, assuming that the average energy of the reference high-frequency sub-band (i.e., the first sub-band divided by the reference value of the 160-point MDCT coefficient of the generated high-frequency signal) is YL, and the average energy of the current high-frequency sub-band (i.e., the sub-band corresponding to the sub-band spectrum envelope decoded based on the bitstream) is Y_H, then a gain factor a=sqrt(Y_H / YL) is calculated, where sqrt() represents a square root calculation function, which is used to calculate the square root of (Y_H / YL). With the gain factor a, the MDCT coefficients of each point in the reference high-frequency sub-band are directly multiplied by a. The average energy of the MDCT coefficients (virtually generated) after gain control is very close to the original one at the encoding end.

[0238] Finally, the inverse MDCT transform is called to generate an estimated value of the first subframe of the high frequency subband signal (ie, a first high frequency subband signal estimated value).

[0239] In step 20323, based on the second transform coefficient, the high-frequency feature estimation value is subjected to inverse frequency band expansion processing to obtain a second high-frequency sub-band signal estimation value.

[0240] Here, step 20323 is similar to step 20322, except that the processing object of step 20323 is the second transformation coefficient, and the processing object of step 20322 is the first transformation coefficient.

[0241] In some embodiments, step 20323 can be implemented in the following manner: performing spectrum replication processing on the transform coefficients of the second half of the second transform coefficients to obtain second reference transform coefficients of the reference high-frequency sub-band signal; performing gain processing on the second reference transform coefficients based on the second half of the sub-band spectrum envelope corresponding to the high-frequency feature estimation value to obtain the gained second reference transform coefficients; performing inverse frequency domain transform processing on the gained second reference transform coefficients to obtain a second high-frequency sub-band signal estimation value.

[0242] In some embodiments, based on the second half of the subband spectrum envelope corresponding to the high-frequency feature estimation value, the second reference transform coefficient is gain processed to obtain the second reference transform coefficient after gain, including: based on the second half of the subband spectrum envelope, the second reference transform coefficient of the reference high-frequency subband signal is divided into multiple second subbands; for any second subband among the multiple second subbands, the following processing is performed: determining the first average energy corresponding to the second subband in the second half of the subband spectrum envelope, and determining the second average energy corresponding to the second subband; determining the gain factor based on the ratio of the first average energy to the second average energy; multiplying the gain factor by each second reference transform coefficient included in the second subband to obtain the second reference transform coefficient after gain.

[0243] In step 20324, the first high frequency sub-band signal estimation value and the second high frequency sub-band signal estimation value are combined to obtain a high frequency sub-band signal estimation value corresponding to the audio bit stream.

[0244] In step 2033, sub-band synthesis is performed on the low-frequency sub-band signal estimation value and the high-frequency sub-band signal estimation value to obtain a synthesized audio signal corresponding to the audio code stream.

[0245] For example, the subband synthesis process is the inverse process of the subband decomposition process. The decoding end performs subband synthesis processing on the low-frequency subband signal and the high-frequency subband signal to restore the synthesized audio signal, wherein the synthesized audio signal is the restored reconstructed signal.

[0246] In some embodiments, sub-band synthesis processing is performed on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain a synthesized audio signal, including: up-sampling the low-frequency sub-band signal to obtain a low-pass filtered signal; up-sampling the high-frequency sub-band signal to obtain a high-frequency filtered signal; and filtering and synthesizing the low-pass filtered signal and the high-frequency filtered signal to obtain a synthesized audio signal.

[0247] For example, after obtaining the low-frequency sub-band signal and the high-frequency sub-band signal, the low-frequency sub-band signal and the high-frequency sub-band signal are subjected to sub-band synthesis through a QMF synthesis filter to restore a synthesized audio signal.

[0248] In some embodiments, the frame header also includes at least one channel bit, and the channel bit is used to indicate whether a mono decoding method or a stereo decoding method is used to perform audio decoding on the audio code stream. When the channel bit in the frame header included in the audio code stream encapsulation is a first value, it indicates that a stereo decoding method needs to be used to decode the audio code stream, that is, a synthesized stereo signal is reconstructed through the decoded synthetic audio signal and the stereo feature vector in the audio code stream encapsulation; when the channel bit in the frame header included in the audio code stream encapsulation is a second value, it indicates that a mono decoding method needs to be used to decode the audio code stream, that is, the decoded synthetic audio signal is used as a mono synthetic audio signal.

[0249] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0250] The embodiments of the present application can be applied to various audio scenarios, such as voice calls, instant messaging, etc. The following is an example of a voice call:

[0251] In the related art, the principle of speech coding is roughly as follows: speech coding can directly encode the speech waveform samples, sample by sample; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0252] The above coding principles are all derived from speech signal modeling, that is, compression methods based on signal processing. In order to improve the coding quality while ensuring the efficiency of speech coding relative to the compression method based on signal processing. The embodiment of the present application provides a multi-mode multi-rate speech coding method (i.e., an audio coding method and an audio decoding method). Based on the characteristics of the audio signal, for the important part (low-frequency sub-band signal), after processing based on the neural network (NN, Neural Network) technology, a feature vector with a lower dimension than the input low-frequency sub-band signal can be obtained. Among them, the neural network adopts a similar "blocking" operation inside, which can reduce the complexity of the algorithm and improve the coding effect. For the low-frequency feature vector (i.e., the above-mentioned low-frequency feature), it can be based on the same low-frequency feature vector, using a code table with different quantization precision for quantization and encoding, to achieve a multi-rate encoding and decoding effect. For the high-frequency feature vector (i.e., the above-mentioned high-frequency feature), multi-level quantization and encoding are adopted to achieve a multi-rate encoding and decoding effect.

[0253] The embodiments of the present application can be applied to Figure 6CThe voice communication link shown takes the Voice over Internet Protocol (VoIP) conference system as an example, and deploys the voice codec technology involved in the embodiment of the present application in the encoding and decoding parts to solve the basic function of voice compression. The encoder is deployed on the uplink client 601, and the decoder is deployed on the downlink client 602. The voice is collected through the uplink client, and pre-processing enhancement, encoding and other processing are performed. The encoded code stream is transmitted to the downlink client 602 through the network, and decoding, enhancement and other processing are performed by the downlink client 602 to play back the decoded voice on the downlink client 602.

[0254] Considering forward compatibility (i.e., the new encoder is compatible with the existing encoder), a transcoder needs to be deployed in the background of the system (i.e., the server) to solve the problem of interoperability between the new encoder and the existing encoder. For example, if the sender (uplink client) is the new NN encoder and the receiver (downlink client) is the public switched telephone network (PSTN, Public Switched Telephone Network) (G.722). In the background, it is necessary to execute the NN decoder to generate the voice signal, and then call the G.722 encoder to generate a specific code stream to realize the transcoding function, so that the receiver can correctly decode based on the specific code stream.

[0255] Before specifically introducing the multi-mode multi-rate speech coding method provided in the embodiment of the present application, the QMF filter group, the dilated convolutional network, and the frequency band extension are first introduced.

[0256] The QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, which represent the QMF low-pass signal and the QMF high-pass signal respectively. Figure 8 The spectral response of the low-pass part HLow(z) and the high-pass part HHigh(z) of the QMF filter shown in FIG. Based on the relevant theoretical knowledge of the QMF analysis filter bank, the correlation between the coefficients of the low-pass filter and the high-pass filter can be easily described, as shown in formula (1):

[0257] h High (k) = -1 k h Low (k) (1)

[0258] Among them, h Low (k) represents the coefficient of low-pass filtering, h High (k) represents the coefficient of high-pass filtering.

[0259] Similarly, according to QMF related theory, the QMF synthesis filter bank can be described based on the QMF analysis filter bank HLow(z) and HHigh(z), as shown in formula (2).

[0260] G Low (z) = H Low (z)

[0261] G High (z)=(-1)*H High (z) (2)

[0262] Among them, G Low (z) represents the restored low-pass signal, G High (z) represents the recovered high-pass signal.

[0263] The low-pass and high-pass signals recovered by the decoding end are synthesized by the QMF synthesis filter bank, so that the reconstructed signal (ie, the synthesized signal) of the sampling rate Fs corresponding to the input signal can be restored.

[0264] See also Fig. 9A and Fig. 9B , Fig. 9A is a schematic diagram of a common convolution (eg, causal convolution) network provided in an embodiment of the present application, Fig. 9B Schematic diagram of a dilated convolutional network provided in an embodiment of the present application. Compared with ordinary convolutional networks, dilated convolution can increase the receptive field while keeping the size of the feature map unchanged, and can also avoid errors caused by upsampling and downsampling. Fig. 9A and Fig. 9B The convolution kernel size shown in is 3×3; however, Fig. 9A The receptive field of the ordinary convolution shown is only 3, while Fig. 9B The receptive field 902 of the dilated convolution shown reaches 5. That is, for a convolution kernel of size 3×3, Fig. 9A The receptive field of the ordinary convolution shown is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1; Fig. 9B The dilated convolution shown has a receptive field of 5 and a dilation rate of 2.

[0265] The convolution kernel can also be used in similar Fig. 9A or Fig. 9B The convolution kernel moves on the plane, which involves the concept of stride rate. For example, each time the convolution kernel shifts 1 grid, the corresponding stride rate is 1.

[0266] In addition, there is the concept of the number of convolution channels, which is the number of convolution kernel parameters used to perform convolution analysis. In theory, the more channels there are, the more comprehensive the signal analysis is and the higher the accuracy is; however, the higher the number of channels, the higher the complexity is. For example, a 1×320 tensor can use a 24-channel convolution operation, and the output is a 24×320 tensor.

[0267] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the size of the convolution kernel can be set to 1×3), the expansion rate, the shift rate and the number of channels can be defined according to actual application needs. The embodiments of the present application do not make specific limitations on this.

[0268] like Fig.10 The schematic diagram of frequency band extension (or frequency band replication) shown in the figure first reconstructs the broadband signal, then copies the broadband signal to the ultra-wideband signal, and finally performs shaping based on the ultra-wideband envelope. Fig.10 The frequency domain implementation scheme shown specifically includes:

[0269] 1) Implement a core layer encoding at a low sampling rate; 2) Select the low-frequency spectrum and copy it to the high frequency; 3) Perform gain control on the copied high-frequency spectrum based on the pre-recorded boundary information (describing the energy correlation between high and low frequencies, etc.). With only a bit rate of 1-2kbps, the sampling rate can be doubled.

[0270] The low-complexity and low-bitrate neural network speech compression method provided in the embodiment of the present application will be described below in combination with the high-frequency part and the low-frequency part in the mono signal.

[0271] Combine the following Figure 7 The multi-mode multi-rate speech coding method provided in the embodiment of the present application is described as follows:

[0272] The following processing is performed on the encoding end:

[0273] For the input audio signal x(n) of the nth frame, use the analysis filter to decompose it into the low-frequency subband signal X LB (n) and the high frequency subband signal X HB (n).

[0274] For the low frequency subband signal X LB (n), call the second NN to obtain the low-dimensional feature vector F LB (n), i.e. low-frequency features, feature vector F LB The dimension of (n) is smaller than that of the low-frequency subband signal to reduce the amount of data. LB (n), call the neural network (encoding part) to generate a lower-dimensional feature vector F LB(n). The embodiments of the present application do not limit other NN structures, such as autoencoder, full-connection (FC) network, long short-term memory (LSTM) network, convolutional neural network (CNN) + LSTM, etc. Among them, the use of similar "blocking" operations inside the neural network can reduce the complexity of the algorithm and improve the encoding effect. LB (n) Use code tables with different quantization precisions for quantization and encoding to achieve multi-rate encoding and decoding effects.

[0275] For the high frequency subband signal X HB (n), considering that high frequencies are not as important to quality as low frequencies, the high frequency subband signal X HB (n) Other schemes can be used to extract the feature vector F HB (n). For example, the frequency band extension technology based on speech signal analysis can realize the generation of high-frequency sub-band signals with only a small number of bits; it can also use the same NN structure as the low-frequency sub-band signal or a more streamlined network (for example, the output feature vector is smaller than the low-frequency feature vector F LB (n) is smaller). For the high frequency feature vector F HB (n) Multi-level quantization and encoding are used to achieve multi-rate encoding and decoding effects.

[0276] The eigenvector corresponding to the subband signal (i.e., F LB (n) and F HB (n)) performs vector quantization or scalar quantization, and performs entropy coding on the quantized value, and transmits the encoded bit stream (low-frequency bit stream and high-frequency bit stream) to the decoding end.

[0277] The decoding end performs the following processing:

[0278] Decode the code stream (including low-frequency code stream and high-frequency code stream) received by the decoding end to obtain the estimated value F′ of the low-frequency feature vector LB (n) and the estimated value of the high-frequency eigenvector F′ HB (n).

[0279] For the low-frequency part, the estimated value F′ based on the low-frequency eigenvector LB (n) Call the third NN to obtain the estimated value x′ of the low-frequency subband signal LB (n). The neural network uses a similar "blocking" operation to reduce the complexity of the algorithm and improve the encoding effect. According to the actual codeword length in the code stream, code tables with different quantization precisions can be used to generate F′ with different precisions. LB (n) and x′LB (n) to achieve multi-rate encoding and decoding effects.

[0280] For the high-frequency part, the estimated value F′ based on the high-frequency eigenvector HB (n) Call high-frequency reconstruction to generate an estimate x′ of the high-frequency subband signal HB (n). According to the actual codeword length in the code stream, code tables with different quantization precisions can be used to generate F′ with different precisions. HB (n) and x′ HB (n) to achieve multi-rate encoding and decoding effects.

[0281] Finally, the QMF synthesis filter is called to generate the reconstructed synthetic speech signal x′(n).

[0282] The multi-mode multi-rate speech encoding method provided in the embodiment of the present application is described in detail below.

[0283] In some embodiments, a speech signal with a sampling rate of Fs=32000 Hz is taken as an example (it should be noted that the method provided in the embodiment of the present application is also applicable to scenarios with other sampling rates, including but not limited to: 8000 Hz, 32000 Hz, and 48000 Hz). At the same time, assuming that the frame length is set to 20 ms, therefore, for Fs=32000 Hz, it is equivalent to containing 640 sample points per frame.

[0284] Reference below Figure 7 The flowchart shown in FIG. 1 describes the encoding end and the decoding end in detail.

[0285] The process of encoding the low-frequency part and the high-frequency part of the mono signal is as follows:

[0286] For an audio signal (mono signal) with a sampling rate of Fs=32000 Hz, the input signal of the nth frame includes 640 sample points, which are recorded as input signal x(n), ie, a mono input signal.

[0287] Step 11: Call the QMF analysis filter to decompose the signal.

[0288] Calling the QMF analysis filter (2-channel QMF) for downsampling, we can obtain two sub-band signals, namely the low-frequency sub-band signal X LB (n) and the high frequency subband signal X HB (n). Low frequency subband signal X LB (n) and the high frequency subband signal X HB The effective bandwidth of (n) is 0-8kHz and 8-16kHz respectively, and the low-frequency subband signal X LB (n) and the high frequency subband signal X HB The number of sample points of (n) is 320.

[0289] Step 12: Based on the low-frequency subband signal, call the second NN.

[0290] Based on the low-frequency subband signal x LB (n), call the second NN to generate a lower dimensional feature vector F LB (n). It should be noted that X LB The dimension of (n) is 320, and F LB The dimension of (n) is 56. From the perspective of data volume, the second NN plays a role of "dimensionality reduction" and realizes the function of data compression. LB (n) dimensions, or other dimensions smaller than X LB (n) dimension.

[0291] See also Fig.11 The network structure diagram of the second NN is shown below. The process of data compression by the second NN is described in detail below:

[0292] First, a 16-channel causal convolution is called to expand the input tensor (i.e., vector) to a 16×320 tensor.

[0293] Then, the 16×320 tensor is preprocessed. For example, after performing a convolution operation on the 16×320 tensor, a pooling operation with a factor of 2 is performed, and the activation function can be PReLU to generate a 16×160 tensor.

[0294] Next, 4 encoding blocks with different downsampling factors (Down_factor) are cascaded. Each encoding block contains a residual block, a convolution layer, and a pooling layer. Each residual block includes 5 residual units (Residual Unit) based on dilated convolution (the input and output feature dimensions of the residual unit will not change); a convolution layer is used to double the number of input channels, and the activation function can be PReLU, so as to ensure the data volume and avoid data loss; the pooling layer is a pooling operation containing Down_factor to complete downsampling and achieve data compression. Here, the Down_factor of the 4 encoding blocks is set to 2, 4, 4, and 5 respectively. Therefore, the number of output channels of the 4 encoding blocks is set to 32, 64, 128, and 256 respectively. After being processed by the 4 encoding blocks, the input 16×160 tensor is converted into 32×80, 64×20, 128×5, and 256×1 tensors respectively. The embodiment of the present application is not limited to the number of coding blocks, and may be any positive integer such as 2, 3, 4, 5, etc. In addition, the embodiment of the present application is not limited to the number of residual units in a coding block, and may be any positive integer such as 2, 3, 4, 5, 6, etc. The number of residual units in multiple coding blocks may be the same or different, for example, one coding block contains 4 residual units and another coding block contains 5 residual units.

[0295] Here we further introduce the residual unit. The residual unit refers to a module in a deep neural network. By introducing cross-layer connections in the neural network, the neural network is easier to optimize during training and avoids problems such as gradient disappearance or gradient explosion. The core idea is to perform residual learning on the input inside the module, that is, to bypass some layers through a direct path and pass the input information directly to the output, so that the network can better utilize shallow feature information during the learning process. Fig. 12A This is a schematic diagram of the residual block structure used in the encoding block in the third NN. This residual block includes 4 residual units based on dilated convolution, and each residual unit contains a dilated convolution block with a specified dilation rate (Dilation rate), that is, each dilated convolution block contains a convolution operator with a specified dilation rate (such as Dilation rate = 3). In the embodiment of the present application, the use of dilated convolution blocks with 5 progressive dilation rates is equivalent to using different receptive fields to extract input features at different resolutions, which can better analyze the data relatively comprehensively. After the residual is processed by 5 dilated convolution blocks with specified dilation rates, it is added to the input from the jump connection to obtain the output result of the residual block, and the output result is output to the convolution layer connected to the residual block.

[0296] Here, further Fig. 12A Any residual unit is described, such as Fig. 12B As shown. For any residual unit, it contains at least one dilated convolution with a specified dilation rate (used to expand the receptive field), and PReLU can be used as the activation function; in addition, one or more causal convolutions (used to extract local information) can be cascaded, and PReLU can be used as the activation function. The convolution kernel size of the dilated convolution with the above specified dilation rate can be 3, 5, 7, 9, etc., and the convolution kernel size of the causal convolution can be 1, 3, etc. This embodiment of the present application does not limit the convolution kernel size of the dilated convolution or causal convolution with the above specified dilation rate. In addition, the causal convolution or dilated convolution in the embodiment of the present application can also be implemented by other specific convolution units with similar or equivalent functions.

[0297] In addition, for the residual unit, in order to reduce the complexity of the algorithm, the grouped convolution algorithm is introduced. Grouped convolution is to divide the input channels into multiple groups for convolution operations, and only the input channels and output channels in each group are associated. Here, it is assumed that the input channels are 16 and the output channels are 32. If the number of groups is 1, each input channel is associated with 32 output channels. If the number of groups is 2, first divide the 16 input channels into two groups of 0-7 and 8-15. In the two groups, the input channels are associated with the output channels in the group. For example, the input channels 0-7 in the first group are associated with the output channels 0-15, and the input channels 8-15 in the second group are associated with the output channels 16-31. For example, the 0th output channel is only associated with the 0th-7th input channels, not with the 8th-15th input channels, and the 25th output channel is only associated with the 8th-15th input channels, not with the 0th-7th input channels. From this comparison, it can be seen that the introduction of grouped convolution can avoid the association between any input channel and all output channels, reduce the number of connections, and reduce complexity. Of course, since the larger the number of groups, the smaller the correlation between the input channel and the output channel, it will also affect the encoding effect, so the larger the number of groups, the better. In the embodiment of the present application, the hole convolution contained in the 5 residual blocks corresponding to the 4 coding blocks can use different grouping number configurations, and the specific grouping number configurations are shown in Table 1.

[0298] Table 1. Grouping configuration used by residual units in different coding blocks

[0299] Encoding Block Number of groups Coding block (Down_factor = 2) 2 Coding block (Down_factor = 4) 4 Coding block (Down_factor = 4) 4 Coding block (Down_factor = 5) 4

[0300] Finally, after similar preprocessing of the 256×1 tensor, a 56-dimensional feature vector F can be output. LB (n). According to the second NN calculation, the 56-dimensional feature vector F LB Each value in (n) is between [-1, 1].

[0301] Step 13: For the high frequency sub-band signal X HB (n) Perform high frequency analysis.

[0302] The purpose of high-frequency analysis is to extract the high-frequency subband signal x HB (n) to generate a lower-dimensional feature vector F HB (n). The present application embodiment is not limited to F HB (n) dimensions, or other dimensions smaller than X LB (n) dimension, but F HB The dimension of (n) must be smaller than that of F LB (n) dimension.

[0303] In some embodiments, referring to step 12, another structure similar to the second NN can be introduced to generate a low-dimensional feature vector. Compared with the low-frequency subband signal, the high-frequency subband signal is less important to the quality, so the NN structure for the high-frequency subband signal does not need to be as complex as the second NN. The NN structure for the high-frequency subband signal is similar to that of the second NN, but compared with the structure of the second NN, the NN structure for the high-frequency subband signal greatly reduces the number of channels.

[0304] The embodiment of the present application proposes another method to compress the high frequency sub-band signal, namely, frequency band expansion (recovering the broadband audio signal from the frequency band limited narrow-band audio signal). The application of frequency band expansion in the embodiment of the present application is described in detail below:

[0305] For the high frequency subband signal X including 320 points (ie, sample points) HB (n), the 320-point high-frequency subband signal X HB (n), divided into two subframes (the first subframe and the second subframe), each subframe includes 160 points. Compared with the 640-point 50% overlap MDCT transform, the algorithm delay of 10ms can be reduced by using 320-point 50% overlap MDCT transform by framing. Therefore, for each 20ms frame, 2 320-point MDCT transforms can be performed to extract features.

[0306] For any subframe, i.e., a high-frequency subband signal including 160 points, a modified discrete cosine transform (MDCT) is called to generate 160-point MDCT coefficients. Specifically, if there is a 50% overlap, the first subframe of the n+1th frame can be merged (joined) with the second subframe of the nth frame, and the 320-point MDCT is calculated to obtain the 160-point MDCT coefficients of the first subframe of the n+1th frame; the second subframe of the n+1th frame can be merged with the first subframe of the n+1th frame, and the 320-point MDCT is calculated to obtain the 160-point MDCT coefficients of the second subframe of the n+1th frame.

[0307] For the 160-point MDCT coefficients of any one of the above two subframes, the following processing is performed in the same manner: the 160-point MDCT coefficients are divided into N subbands, where the subband is a group of adjacent multiple MDCT coefficients, and the 160-point MDCT coefficients can be divided into 4 subbands. For example, the 160 points can be evenly distributed, that is, the number of points included in each subband is consistent. Of course, the embodiment of the present application cannot perform non-uniform division of the 160 points, such as the subband with a low frequency includes fewer MDCT coefficients (higher frequency resolution), and the subband with a high frequency includes more MDCT coefficients (lower frequency resolution).

[0308] According to Nyquist sampling theorem (to restore the original signal from the sampled signal without distortion, the sampling frequency should be greater than 2 times the highest frequency of the original signal. When the sampling frequency is less than 2 times the highest frequency of the spectrum, the spectrum of the signal is aliased. When the sampling frequency is greater than 2 times the highest frequency of the spectrum, the spectrum of the signal is not aliased), the 320-point MDCT coefficients above represent the spectrum of 8-16kHz. However, ultra-wideband voice communication does not necessarily require the spectrum to be 16kHz. For example, if the spectrum is set to 14kHz, only the MDCT coefficients of the first 240 points need to be considered. Correspondingly, the number of subbands can be controlled to 6.

[0309] For each subband, the average energy of all MDCT coefficients in the current subband is calculated as the subband spectrum envelope (the spectrum envelope is a smooth curve passing through the main peak points of the spectrum). For example, the MDCT coefficients included in the current subband are x(n), n = 1, 2, ..., 40, then the average energy Y = ((x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40). When the 160-point MDCT coefficients are divided into 4 sub-bands, 4 sub-band spectrum envelopes can be obtained. These 4 sub-band spectrum envelopes are the sub-band spectrum envelopes corresponding to the sub-bands. Where i=1 represents the first subframe, and i=2 represents the second subframe. Get the eigenvector F of the high frequency subband signal HB (n).

[0310] In summary, using any of the above two methods (NN structure and frequency band expansion), the 160-dimensional high-frequency subband signal can be output as a 4-dimensional feature vector. Therefore, for each 20ms frame, at least 8-dimensional data is needed to represent the high-frequency information, and the coding efficiency is significantly improved.

[0311] Step 14: Quantization encoding.

[0312] For the eigenvector F of the low-frequency subband signal LB (n) and the eigenvector F of the high-frequency subband signal HB (n), both scalar quantization (each component is quantized separately) and entropy coding methods can be performed. In addition, the embodiments of the present application do not limit the technical combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy coding.

[0313] Here, for the low-frequency eigenvector F LB According to the above description, after the low-frequency subband signal is processed by the second NN, a 56-dimensional feature vector F is obtained. LB (n) The present application embodiment provides a method based on scalar quantization and entropy coding, including: 1) for F LB (n) For each dimension, the interval [-1, 1] is divided into 11 equal parts to form a codebook containing 11 elements, and each dimension value is quantized into one of the 11 elements; 2) According to Shannon's entropy theorem, for a codebook containing 11 elements and uniformly distributed, the entropy (average bit) is Therefore, the average bit corresponding to each frame of low-frequency subband signal is 193.76 bits; 3) For every 20ms frame mode, there are 50 frames in 1 second, so the average bit rate is 9.69kbps. According to the entropy coding theory, the probability distribution statistics of each of the above dimensions can be performed to generate 56 code tables. Generally, each dimension is non-uniformly distributed, so the actual bit rate is around 9.69kbps, or less than 9.69kbps.

[0314] In addition, in order to achieve the low-frequency feature vector F LB The goal of multi-rate encoding and decoding of (n) can be implemented as follows:

[0315] The code table of the above implementation method, that is, the code table containing 11 elements in each dimension, is named low-frequency code table-1, and the reference bit rate is 9.69 kbps.

[0316] Similarly, for each dimension, the interval [-1, 1] is divided into 9 equal parts, the entropy is 3.17, and the average bit rate is 3.17*56*50 / 1000=8.88kbps. It is named low-frequency code table-2, and the reference bit rate is 8.88kbps. In this way, there are at least 2 bit rate modes, and different bit rates correspond to different qualities; multi-rate encoding is realized.

[0317] Similarly, for each dimension, the interval [-1, 1] is divided into 7 equal parts, with an average bit rate of 7.86 kbps, which is named low-frequency code table-3; the interval [-1, 1] is divided into 5 equal parts, with an average bit rate of 6.50 kbps, which is named low-frequency code table-4.

[0318] Thus, for the original 56-dimensional feature vector calculated for each frame of data, the coding mode of low-frequency features of at least 4 code rates is realized by configuring the quantization accuracy. It should be noted that the embodiment of the present application does not limit other multi-code rate construction methods and the number of code tables.

[0319] For each frame (including 2 subframes), the feature vector F of the 8-dimensional high-frequency subband signal HB (n), a specific implementation of its quantization coding includes: 1) scalar quantization is performed on each dimension separately, and 5 bits are allocated; thus, 40 bits are allocated for each frame; 2) for every 20ms frame mode, there are 50 frames in 1 second, so the average bit rate is 2kbps. If entropy coding (including but not limited to Huffman Coding or Range Coding) is performed on the quantized values ​​of each dimension above, the overall bit rate will be less than 2kbps. The above code table is defined as high-frequency code table-1, and the quantization value is defined as the high-frequency feature vector quantization value (i.e., the quantization value of the high-frequency feature).

[0320] In addition, in order to achieve the high frequency feature vector F HB The multi-rate encoding and decoding goal of (n) can be further introduced into residual processing, which can be implemented as follows:

[0321] 1) For each 10ms subframe, the 4 subbands of the subframe are further evenly divided into 2 subsets.

[0322] For example, the set of envelope values ​​of four sub-bands is {e1, e2, e3, e4}.

[0323] 2) For each subset of each subband, calculate the envelope value of the subset.

[0324] For example, the set of envelope values ​​of all subsets is {e 1a , e 1b , e 2a , e 2b , e 3a , e 3b , e 4a , e 4b}.

[0325] 3) Subtract the corresponding high-frequency feature vector quantization value from the envelope value of each subset to obtain the first high-frequency feature vector residual (referred to as the first residual value).

[0326] For example, the high-frequency feature vector quantization value is Then the first high-frequency eigenvector residual is

[0327] 4) The first high-frequency feature residual is subjected to similar quantization and entropy coding to obtain a first high-frequency feature vector residual quantization value (hereinafter referred to as the quantization value of the first residual value).

[0328] Here, considering that the dynamic range of the residual value is much lower than the original value, only 2 bits need to be allocated for scalar quantization of each residual. In this way, a reference bit rate of 2*16*50=1.6kbps is required (similarly, because it is non-uniformly distributed entropy coding, the actual bit rate is generally less than 1.6kbps) to obtain the residual quantization value of the first high-frequency eigenvector.

[0329] 5) Subtract the sum of the corresponding high-frequency feature vector quantization value and the first high-frequency feature vector residual quantization value from the envelope value of each subset to obtain the second high-frequency feature vector residual (referred to as the second residual value).

[0330] Here, similarly, the quantization and entropy coding of the second high-frequency eigenvector residual can be completed using a 1.6 kbps reference bit rate.

[0331] By analogy, as more residuals are calculated, it is possible to ensure that the high-frequency feature vector reconstructed at the decoding end is infinitely close to the original high-frequency feature vector. In addition, according to the presence or absence of high-frequency feature vector residuals, or the number of high-frequency feature vector residuals, a multi-rate coding effect is achieved. The embodiment of the present application may define the number of types of residual coding (no residual coding, first high-frequency feature vector residual, second high-frequency feature vector residual, third high-frequency feature vector residual, etc.) according to the bit rate requirements, and is not limited here.

[0332] In addition, an embodiment of the present application provides a solution for additionally extracting side information at the encoding end, which is used to guide more sophisticated processing at the decoding end to improve audio quality. In principle, the main function of band extension is to copy the low-frequency spectrum to the high frequency and then perform gain control. However, the high-frequency spectrum is flatter than the low-frequency spectrum and does not have so many harmonic components. If it is directly copied and then gain controlled, the artificially generated high frequency may contain too many harmonic components. Therefore, an embodiment of the present application proposes to estimate specific side information at the encoding end and write it into the bit stream. The decoding end determines whether to perform additional processing at the decoding end based on the above-mentioned flatness side information. It should be noted that the above-mentioned flatness side information will only prompt the decoding end whether the current frame needs additional processing. In a special case, no additional processing is required for the entire speech.

[0333] The above side information estimation is implemented as follows at the encoding end:

[0334] 1) For each 10 - ms sub - frame, divide the 160 - point MDCT coefficients into two blocks. It can be evenly divided, or the MDCT coefficients of the first 2 sub - bands can be used as the first block, and the MDCT coefficients of the last 2 sub - bands can be used as the second block.

[0335] 2) For each block, calculate the square of each MDCT coefficient, i.e., p(i) = c(i) 2 。

[0336] 3) Calculate the arithmetic mean of all MDCT coefficients in the current block: where I represents the number of MDCT coefficients in the current block.

[0337] 4) Calculate the geometric mean of all MDCT coefficients in the current block: where In represents the logarithmic operation, and I represents the number of MDCT coefficients in the current block.

[0338] 5) Calculate the first flatness of the current block Generally, sfpHi is a value in the range of [0, 1].

[0339] 6) Similarly, calculate the second flatness sfp of the MDCT coefficients in the specified low - frequency band Lo 。

[0340] 7) When (sfp Hi [i]<sfpL o [i]) or (sfp Hi [i]<SF_THD), Sfp_FLAG = 0; otherwise, Sfp_FLAG = 1. Here, SF_THD represents the set flatness threshold, and Sfp_FLAG represents the flatness side information.

[0341] Here, when Sfp_FLAG = 1, it means that the decoding end needs additional operations to avoid excessive harmonics artificially generated in the high - frequency. According to the above operations, 4 additional bits are required for each frame, representing the flatness side information of the two blocks in two 10 - ms sub - frames. The embodiments of the present application are not limited to the above extraction method, and other analysis methods can also be used to extract the above flatness side information to guide the decoding process at the decoding end.

[0342] After quantization coding, a bit - stream can be generated. According to experiments, in the range of 5 - 10 kbps, high - quality compression can be achieved for 16 - kHz wide - band signals; in the range of 8 - 15 kbps, high - quality compression can be achieved for 32 - kHz ultra - wide - band signals.

[0343] In addition, in order to achieve the above - mentioned goal of multi - mode and multi - rate encoding and decoding, a good bit - stream structure needs to be designed. As Figure 13A-13E shown, the implementation form of the bit - stream structure is as follows:

[0344] For each frame of code stream, it generally includes an 8-bit frame header. Among them, the 8-bit frame header at least includes a 1-bit coding mode bit. According to the 1-bit coding mode bit, it can be known whether to use wideband coding (the value of the 1-bit coding mode bit is 0) or ultra-wideband coding (the value of the 1-bit coding mode bit is 1), that is, the coding mode includes two types (wideband coding and ultra-wideband coding). In addition, it also includes a 2-bit coding rate bit. According to the 2-bit coding rate bit, it can be known which code rate mode is used in the corresponding wideband coding mode or ultra-wideband coding mode. It should be noted that the embodiments of the present application are not limited to the positions of the 1-bit coding mode bit and the 2-bit coding rate bit in the frame header. For example, the 1-bit coding mode bit can be located in the first position of the frame header, and the 2-bit coding rate bit can be located in the last two positions of the frame header.

[0345] like Fig.13A As shown in the figure, for the broadband coding mode, the frame header is followed by a 56-dimensional low-frequency feature vector index. Because it is entropy coding, the coding length is different. As mentioned above, according to the 2-bit code rate mode bit, different low-frequency code tables are used to quantize the encoded low-frequency code stream and save it in the 56-dimensional low-frequency feature vector index part.

[0346] The 2-bit coding rate in wideband coding mode is explained as follows:

[0347] 1) 00, indicating the use of low-frequency code table -4, the reference average bit rate is 6.5kbps, which is also the lowest bit rate in broadband coding mode;

[0348] 2) 01, indicating the use of low-frequency code table-3, with a reference average bit rate of 7.86 kbps;

[0349] 3) 10, indicating the use of low-frequency code table-2, with a reference average bit rate of 8.88 kbps;

[0350] 4) 11, indicating the use of low-frequency code table -1, the reference average bit rate is 9.69 kbps, which is also the highest bit rate in broadband coding mode.

[0351] For ultra-wideband coding mode, a 2-bit rate mode bit is defined according to whether residual coding is used. The 2-bit code rate bit in ultra-wideband coding mode is explained as follows:

[0352] 1) 00 (excluding residual coding), indicating that the low-frequency part uses the low-frequency code table -4, and the code stream encapsulation structure includes the use of low-frequency code table -4 quantization coding, ultra-wideband coding 4 spectral envelopes, and 4-bit flatness. The reference average bit rate is 6.5+2+0.2=8.7kbps;

[0353] 2) 01 (excluding residual coding), indicating that the low-frequency part uses the low-frequency code table -1. The code stream encapsulation structure includes quantization coding using the low-frequency code table -1, ultra-wideband coding 4 spectral envelopes, and 4-bit flatness. The reference average bit rate is 9.69+2+0.2=11.9 kbps;

[0354] 3) 10 (including the first residual coding), indicating that the low-frequency part uses the low-frequency code table-1, and the code stream encapsulation structure includes the use of the low-frequency code table-1 quantization coding, ultra-wideband coding 4 spectral envelopes, 8 first residuals and 4-bit flatness, and the reference average bit rate is 9.69+2+1.6+0.2=13.5kbps;

[0355] 4) 11 (including the first residual coding and the second residual coding), indicating that the low-frequency part uses the low-frequency code table-1, and the code stream encapsulation structure includes the use of the low-frequency code table-1 quantization coding, ultra-wideband coding 4 spectral envelopes, 8 first residuals, 8 second residuals and 4-bit flatness. The reference average bit rate is 9.69+2+1.6+1.6+0.2=15.0kbps

[0356] like Figure 13A-13C , according to the 2-bit code rate mode bit, the low-frequency feature vector index and the high-frequency feature vector index related code stream are placed immediately after the frame header. Figure 13D-13E If the code stream encapsulation structure contains 4-bit flatness edge information, the 4-bit edge information can be followed by the 8-dimensional high-dimensional feature vector index.

[0357] In summary, when using an encoder for audio encoding, the user will input the voice data to be encoded and the correct configuration parameters to the encoder, for example, when the input voice data is broadband, the sampling rate is 16000Hz, and when the input voice data is ultra-wideband, the sampling rate is 32000Hz. For the convenience of description, a broadband signal based on 16000Hz sampling and an ultra-wideband signal based on 32000Hz sampling are taken as examples, but the embodiments of the present application do not limit other combinations.

[0358] In addition, the user needs to configure the configuration parameters of the bit rate mode used, and the encoder can select a suitable code table according to the configured bit rate mode. Generally, the higher the bit rate mode, the more bits the encoder will use for encoding, and the corresponding reconstructed voice quality is higher. As shown above, the embodiment of the present application provides 4 bit rate modes for the wideband coding mode or the ultra-wideband coding mode, but the embodiment of the present application is not limited to the 4 bit rate modes.

[0359] For example, when the user inputs a wideband signal based on 16000Hz sampling, the encoder can use the wideband coding mode for encoding, directly use the second NN for the input wideband signal to extract low-frequency features, quantize and encode the extracted low-frequency features, and obtain the code stream of the wideband signal; when the user inputs an ultra-wideband signal based on 32000Hz sampling, the encoder can use the ultra-wideband coding mode for encoding, first pass the input ultra-wideband signal through the analysis filter to separate the low-frequency sub-band signal and the high-frequency sub-band signal, and directly use the second NN for the low-frequency sub-band signal to extract the low-frequency feature F LB (n), for F LB (n) Quantization encoding is performed to obtain a low-frequency code stream, and the high-frequency subband signal is subjected to high-frequency analysis to extract the high-frequency feature F HB (n), for F HB (n) Perform quantization encoding to obtain a high-frequency code stream. In addition, according to the bit rate mode input at the encoding end, the encoder can select a corresponding code table for quantization encoding.

[0360] The decoding process of the low-frequency part and the high-frequency part is as follows:

[0361] Step 21: quantization decoding.

[0362] Quantization decoding is the inverse process of quantization encoding. For the received code stream (including high-frequency code stream and low-frequency code stream), entropy decoding is first performed, and the estimated value F′ of the feature vector of the low-frequency code stream is obtained by looking up the quantization table. LB (n) and the estimated value F′ of the eigenvector of the high-frequency code stream HB (n).

[0363] refer to Fig.13E Here, we take the configuration of ultra-wideband coding mode and the bit rate mode bit being 10 as an example for the following description:

[0364] First, parse the frame header of the code stream encapsulation. When the coding mode bit is 1, it means that ultra-wideband coding is used during encoding. When the code rate mode bit is 10, it means that the broadband part uses low-frequency code table-1 during encoding. After the ultra-wideband part is encoded, the code stream encapsulation contains 8-dimensional high-frequency feature vector indexes and 16-dimensional first high-frequency feature vector residual indexes.

[0365] For the broadband part, the low-frequency code stream is parsed by entropy coding and decoding technology, and the estimated value F′ of the 56-dimensional low-frequency feature vector can be obtained through the low-frequency code table-1 LB (n).

[0366] For the ultra-wideband part, first, based on the 5-bit code table, an 8-dimensional high-frequency feature vector estimate is obtained for each frame; then, based on the 2-bit code table, a 16-dimensional first high-frequency feature vector residual estimate is obtained for each frame. Through the corresponding 8-dimensional high-frequency feature vector estimate (i.e., the above-mentioned high-frequency feature estimate) and the 16-dimensional first high-frequency feature vector residual estimate (i.e., the above-mentioned first residual feature estimate), the 16-dimensional high-frequency feature vector final estimate (i.e., the above-mentioned high-frequency feature final estimate) can be obtained. The i-th value in the high-frequency feature vector estimate is added to the 2i-1-th value and the 2i-th value in the residual estimate of the first high-frequency feature vector, respectively, to obtain the 2i-1-th value and the 2i-th value in the final estimate of the high-frequency feature vector. For example, the high-frequency feature estimate is F′ HB (n) = [e′1, e′2, ..., e′8], the first high-frequency eigenvector residual estimate is F′ C (n) = [e′ 1C , e′ 2C , ..., e′ 16C ], then the final estimated value of the high-frequency feature is F′ FHB (n) = [e′1+e′ 1C , e′1+e′ 2C , ..., e′8+e′ 16C ].

[0367] refer to Fig.13D Here, we take the configuration of ultra-wideband coding mode and the bit rate mode bit being 01 as an example for the following description:

[0368] First, parse the frame header of the code stream encapsulation. When the coding mode bit is 1, it means that ultra-wideband coding is used during encoding. When the code rate mode bit is 01, it means that the broadband part uses low-frequency code table -1 during encoding. After the ultra-wideband part is encoded, the code stream encapsulation contains an 8-dimensional high-frequency feature vector index.

[0369] For the broadband part, the low-frequency code stream is parsed by entropy coding and decoding technology, and the estimated value F′L of the 56-dimensional low-frequency feature vector can be obtained through the low-frequency code table-4. B (n).

[0370] For the ultra-wideband part, firstly, based on the 5-bit code table, an 8-dimensional high-frequency feature vector estimation value is obtained for each frame, and the 8-dimensional high-frequency feature vector estimation value is copied to obtain a 16-dimensional high-frequency feature vector final estimation value (i.e., the above-mentioned high-frequency feature final estimation value). For example, the high-frequency feature estimation value is F′ HB (n) = [e′1, e′2, ..., e′8], then the final estimated value of the high-frequency feature is F′ FHB (n)=[e′1, e′1, ..., e′8, e′8].

[0371] Here, the configuration of ultra-wideband coding mode and the bit rate mode bit being 11 is taken as an example for the following description:

[0372] First, parse the frame header of the code stream encapsulation. When the coding mode bit is 1, it means that ultra-wideband coding is used during encoding. When the code rate mode bit is 11, it means that the broadband part uses low-frequency code table-1 during encoding. After the ultra-wideband part is encoded, the code stream encapsulation contains 8-dimensional high-frequency feature vector index, 16-dimensional first high-frequency feature vector residual index and 16-dimensional second high-frequency feature vector residual index.

[0373] For the broadband part, the low-frequency code stream is parsed by entropy coding and decoding technology, and the estimated value F′ of the 56-dimensional low-frequency feature vector can be obtained through the low-frequency code table-1 LB (n).

[0374] For the ultra-wideband part, firstly, based on the 5-bit code table, an 8-dimensional high-frequency feature vector estimation value is obtained for each frame; then, based on the 2-bit code table, a 16-dimensional first high-frequency feature vector residual estimation value and a 16-dimensional second high-frequency feature vector residual estimation value are obtained for each frame. The 16-dimensional high-frequency feature vector final estimation value (i.e., the above-mentioned high-frequency feature estimation value) can be obtained through the corresponding 8-dimensional high-frequency feature vector estimation value (i.e., the above-mentioned high-frequency feature estimation value), the 16-dimensional first high-frequency feature vector residual estimation value (i.e., the above-mentioned first residual feature estimation value) and the 16-dimensional second high-frequency feature vector residual estimation value (i.e., the above-mentioned second residual feature estimation value). The ith value in the high-frequency eigenvector estimate is added to the 2i-1th value in the first high-frequency eigenvector residual estimate and the 2i-1th value in the second high-frequency eigenvector residual estimate to obtain the 2i-1th value in the high-frequency eigenvector final estimate. The ith value in the high-frequency eigenvector estimate is added to the 2i-th value in the first high-frequency eigenvector residual estimate and the 2i-th value in the second high-frequency eigenvector residual estimate to obtain the 2i-th value in the high-frequency eigenvector final estimate. For example, the high-frequency eigenvector estimate is F′ HB (n) = [e′1, e′2, ..., e′8], the first high-frequency eigenvector residual estimate is F′ C (n) = [e′ 1C , e′ 2C , ..., e′ 16C ], the second high-frequency eigenvector residual estimate is F′ B (n) = [e′ 1B , e′ 2B , ..., e′ 16B ], then the final estimated value of the high-frequency feature is F′ FHB (n) = [e′1+e′ 1C +e′ 1B , e′1+e′ 2C +e′ 2B,…,e′8+e′ 16C +e′ 16B ].

[0375] Step 22: Estimated value F′ based on the feature vector of the low-frequency code stream LB (n), call the third NN.

[0376] First, based on the estimated value F′ of the feature vector of the low-frequency code stream LB (n), calling Fig.14 The third NN shown generates an estimate x′ of the low-frequency subband signal LB (n). The third NN is similar to the second NN, such as causal convolution, the post-processing structure is similar to the pre-processing structure in the second NN, and the decoding block structure is symmetrical with the encoding block on the encoding side. The encoding block on the encoding side first performs dilated convolution and then pooling to complete downsampling, while the decoding block on the decoding side first performs pooling to complete upsampling and then performs dilated convolution. The specific process of the third NN is as follows:

[0377] First, call a causal convolution, which can transform the input tensor F′ LB (n), a tensor expanded from 56×1 to 256×1.

[0378] Next, cascade four decoding blocks with different upsampling factors (Up_factor). Each decoding block includes a convolution layer, an upsampling module, and a residual block, wherein a convolution layer is used to halve the number of input channels; an upsampling module includes a specific Up_factor for completing upsampling; a residual block includes 5 residual units (Residual Unit) based on hole convolution. The Up_factor of the four decoding blocks is set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels of the four decoding blocks is set to 128, 64, 32, and 16, respectively. After processing by four decoding blocks, the 256×1 tensor is converted into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. Among them, the embodiments of the present application are not limited to the number of decoding blocks, and can be any positive integers such as 2, 3, 4, 5, etc.

[0379] Here, for the upsampling module including a specific Up_factor, a copy (Repeat) operation can be used to complete the upsampling operation by repeated filling, thereby saving complexity.

[0380] Here, the configuration of the five residual units based on dilated convolution at the decoding end is similar to that of the residual unit at the encoding end, including but not limited to the internal structure of the residual unit, the convolution kernel size, the dilation rate, etc. The number of groups used for dilated convolution in the decoding block is shown in Table 2. In the decoding block here, a larger number of groups of 2 is used to associate more input channels and output channels to improve the quality of speech reconstruction.

[0381] Table 2. Grouping configurations used by residual units in different decoding blocks

[0382] Decoding Blocks Number of groups Decoding block (Up_factor = 5) 4 Decoding block (Up_factor = 4) 4 Decoding block (Up_factor = 4) 2 Decoding block (Up_factor = 2) 2

[0383] Then, the 16×160 tensor output by the cascade decoding block is post-processed. For example, a Repeat operation with a factor of 2 is performed on the 16×160 tensor output by the cascade decoding block to complete upsampling, and then a convolution operation is performed and an activation function is used for the PReLU operation to generate a 16×320 tensor.

[0384] Finally, a causal convolution is called to convert the input 16×320 tensor into a 1×320 tensor to reconstruct the low-frequency subband signal.

[0385] Step 23: Estimation of the eigenvector F′ of the high frequency subband signal HB (n) Perform high-frequency reconstruction.

[0386] Similar to the high frequency analysis at the encoding end, the high frequency reconstruction in the embodiment of the present application includes two schemes.

[0387] The first implementation of high frequency reconstruction corresponds to the first implementation of high frequency analysis in the encoding end. The estimated value F′ based on the eigenvector of the high frequency subband signal HB (n), call the neural network to generate an estimate x′ of the high-frequency subband signal HB (n).

[0388] The second implementation of high-frequency reconstruction corresponds to the band extension technology of high-frequency analysis at the encoding end. Based on the 16 MDCT sub-band spectrum envelopes decoded from the high-frequency bitstream (each 10ms sub-frame contains 8 sub-band spectrum envelopes), that is, the final estimated value of the high-frequency feature vector, the following operations are performed:

[0389] First, the estimated value x′ of the low-frequency subband signal generated by the decoding end LB (n), two 320-point MDCT transforms are also performed to generate two groups of 160-point MDCT coefficients (ie, MDCT coefficients of the low-frequency part, including the first group of 160-point MDCT coefficients and the second group of 160-point MDCT coefficients).

[0390] Then, x′ LB(n) The 160-point MDCT coefficients generated are copied to generate the MDCT coefficients of the high-frequency part. Referring to the basic characteristics of the speech signal, the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid simple copying, which causes the artificially generated MDCT spectrum of the high-frequency part to contain too many harmonics, the last 80 points of the 160-point MDCT coefficients that the low-frequency subband depends on can be used as the master, and the spectrum can be copied twice to generate the reference value of the 160-point MDCT coefficients of the high-frequency subband signal.

[0391] It should be noted that if the bitstream encapsulation contains flatness edge information, additional operations are required. When SfpFLAG==0, no operation is performed. When SfpFLAG==1, additional flattening operations are required. The implementation of the flattening operation is as follows:

[0392] Taking the MDCT coefficient c(i) at position i as an example, calculate the average power spectrum e of the neighborhood of c(i) ave (i), as shown in formula (3).

[0393]

[0394] Here, c(i-1) represents the MDCT coefficient at the i-1th position, and c(i+1) represents the MDCT coefficient at the i+1th position.

[0395] When e ave When (i)==0, the MDCT coefficient c(i) at the i-th position is not smoothed, that is, c(i)=c(i); otherwise, the MDCT coefficient c(i) at the i-th position is smoothed, that is, c(i) is updated to c(i) and e ave (i), that is, c(i) = c(i) / e ave (i).

[0396] After the flattening operation, the 8 sub-band spectral envelopes obtained previously are called (i.e., the 8 sub-band spectral envelopes obtained by querying the quantization table). These 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands, and the reference values ​​of the MDCT coefficients of the generated 160-point high-frequency sub-band signals are divided into 8 reference high-frequency sub-bands. Based on a high-frequency sub-band and the corresponding reference high-frequency sub-band, the reference values ​​of the MDCT coefficients of the generated 160-point high-frequency sub-band signals are gain controlled (multiplication in the frequency domain). For example, the gain factor is calculated based on the average energy of the high-frequency sub-band and the average energy of the corresponding reference high-frequency sub-band, and the MDCT coefficient corresponding to each point in the corresponding reference high-frequency sub-band is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to the original coefficient energy at the encoding end.

[0397] For example, assuming that the average energy of the reference high-frequency sub-band (i.e., the sub-band divided by the reference value of the 160-point MDCT coefficient of the generated high-frequency signal) is Y_L, and the average energy of the current high-frequency sub-band (i.e., the sub-band corresponding to the sub-band spectrum envelope decoded based on the bitstream) is Y_H, then a gain factor a=sqrt(Y_H / Y_L) is calculated, where sqrt() represents a square root calculation function, which is used to calculate the square root of (Y_H / Y_L). With the gain factor a, the MDCT coefficients of each point in the reference high-frequency sub-band are directly multiplied by a. The average energy of the MDCT coefficients (virtually generated) after gain control is very close to the original one at the encoding end.

[0398] Finally, the inverse MDCT transform is called to generate the estimated value of the first subframe of the high-frequency subband signal (the estimated value of the subframe calculated by the MDCT coefficients of the first group of 160 points) and the estimated value of the second subframe of the high-frequency subband signal (the estimated value of the subframe calculated by the MDCT coefficients of the second group of 160 points). The estimated value of the first subframe and the estimated value of the second subframe are combined to obtain the estimated value x′ of the high-frequency subband signal. HB (n) Perform an inverse MDCT transform on the 320-point MDCT coefficients after gain to generate 640-point estimated values. By overlapping, take the first 320 effective estimated values ​​as x′ HB (n).

[0399] Step 24: Synthesize the filter

[0400] At the decoding end, the estimated value x′ of the low-frequency subband signal is obtained LB (n) and the estimated value x′ of the high frequency subband signal HB After that, we only need to upsample and call the QMF synthesis filter to generate a 640-point reconstructed signal x′(n).

[0401] The embodiment of the present application can collect data and jointly train the relevant networks of the encoding end and the decoding end to obtain the optimal parameters. The user only needs to prepare the data and set the corresponding network structure. After the training is completed in the background, the trained model can be put into use.

[0402] In the voice communication system, the communication modes include mono voice communication and stereo communication. Therefore, the embodiment of the present application is not limited to the above mono voice communication codec technology, but can also be a stereo codec technology, wherein the stereo codec technology can enable the encoder to receive signals from the left and right channels (i.e., the input signal of the left channel and the input signal of the right channel), and the user can have a certain left and right spatial sense when wearing headphones for a call.

[0403] The low-complexity and low-bitrate neural network speech compression method provided in the embodiment of the present application will be described below in conjunction with a stereo signal.

[0404] It should be noted that the stereo encoding method of the embodiment of the present application includes but is not limited to the following two methods: 1) the left and right channels are respectively encoded and decoded using the above-mentioned mono signal encoding and decoding technology, and the stereo signal is restored at the decoding end by combining the decoded single-channel reconstructed signal; 2) parametric stereo technology is used for encoding and decoding.

[0405] Among them, the parametric stereo coding technology is a technology that effectively encodes a stereo signal by treating it as a mono signal plus a small amount of parameter overhead describing the stereo image information. At the encoding end, the input signals of the left and right channels (i.e., stereo signals) are down-mixed into one signal, and the one signal is regarded as the above-mentioned input signal x(n), and then the encoding method of the mono signal in the embodiment of the present application is used for encoding. In addition, the characteristic information of the correlation between the left and right channels (i.e., stereo feature vector, generally 1-2kbps) is extracted by parametric stereo coding, and the stereo feature vector is used to mix the side information of the mono channel. At the decoding end, a down-mix signal (i.e., the above-mentioned reconstructed signal) is obtained by decoding, and the stereo signals of the left and right channels are reconstructed by the decoded down-mix signal and the stereo feature vector describing the correlation between the left and right channels.

[0406] Following the above embodiment, for the 8-bit frame header in the code stream encapsulation, 1-bit channel bit can be allocated, and according to the 1-bit stereo coding bit, it can be known whether mono coding or stereo coding is adopted. The specific implementation method is as follows:

[0407] On the encoding side, if the number of input channels is 1, the channel flag is set to 0, and the channel bit of the corresponding frame header is written to 0, indicating that the input signal is encoded in mono, that is, the encoding method for the input signal x(n) is reused for encoding; if the number of input channels is 2, that is, stereo, the channel Flag is set to 1, and the channel bit of the corresponding frame header is written to 1, indicating that the input signal is encoded in stereo, that is, the parametric stereo encoding method introduced above is reused for encoding.

[0408] At the decoding end, the channel bit corresponding to the frame header is parsed. If the channel bit is 0, it indicates mono decoding, and the above decoding method is reused to decode the reconstructed signal; if the channel bit is 1, it indicates stereo decoding, and the above-mentioned parameter stereo decoding method is reused for decoding.

[0409] like Figures 15A-15E The code stream encapsulation shown in the above Figures 13A-13EThe code stream encapsulation shown carries a stereo feature vector. At the decoding end, the stereo signal of the left and right channels is reconstructed through the decoded downmix signal and the stereo feature vector describing the correlation between the left and right channels.

[0410] In addition, for the 8-bit frame header in the code stream encapsulation, the channel bit may not be allocated. For example, if there is no channel bit, mono encoding is used by default.

[0411] Please note that the frame header in each embodiment of the present application is not limited to 8 bits, and can be more or less bits. In another embodiment, more bits can be allocated to the channel bit, such as 2 bits of the channel bit, to represent the encoding and decoding of more channels, such as 3.1 channels or 4.1 channels, etc.

[0412] In summary, the multi-mode and multi-rate encoding and decoding method provided in the embodiment of the present application significantly improves the audio quality while ensuring acceptable complexity compared to signal processing schemes through the organic combination of signal decomposition, signal processing technology and deep neural network.

[0413] So far, the audio encoding method or audio decoding method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the terminal device provided by the embodiment of the present application. The embodiment of the present application also provides an audio encoding device and an audio decoding device. In actual applications, each functional module in the audio encoding device and the audio decoding device can be implemented by hardware resources of an electronic device (such as a terminal device, a server or a server cluster), such as computing resources such as a processor, communication resources (such as for supporting various communication methods such as optical cables and cellular), and a memory. Figure 3A The audio encoding device 555 and the Figure 3B An audio decoding device 655 stored in a memory 650 is shown, which may be software in the form of a program or plug-in, for example, a software module designed in a programming language such as C / C++ or Java, an application software designed in a programming language such as C / C++ or Java, or a dedicated software module in a large software system, an application program interface, a plug-in, a cloud service, and the like. Different implementation methods are described below with examples.

[0414] The audio encoding device 555 includes a series of modules, including a feature extraction module 5551, an encoding module 5552, and a signal encoding module 5553. The following further describes the scheme for implementing audio encoding by cooperating among the modules in the audio encoding device 555 provided in the embodiment of the present application.

[0415] The second acquisition module 5551 is used to respond to the encoding request for the audio signal, obtain the target encoding mode for the audio signal from multiple encoding modes, and obtain the target bit rate mode for the audio signal from multiple bit rate modes; the extraction module 5552 is used to extract the encoding features of the audio signal from the audio signal through the target encoding mode; the signal encoding module 5553 is used to perform signal encoding processing on the encoding features of the audio signal through the target bit rate mode to obtain the audio code stream of the audio signal; the construction module 5554 is used to determine the frame header based on the target encoding mode and the target bit rate mode; the generation module 5555 is used to generate the audio code stream encapsulation of the audio signal based on the audio code stream and the frame header.

[0416] In some embodiments, when the target coding mode is a wideband coding mode, the extraction module 5552 is further used to call a first neural network based on the wideband coding mode; and extract the coding features of the audio signal from the audio signal through the first neural network.

[0417] In some embodiments, the extraction module 5552 is further used to perform feature extraction processing on the audio signal to obtain audio features of the audio signal; and perform residual processing on the audio features using at least one residual unit included in the first neural network to obtain coding features of the audio signal.

[0418] In some embodiments, the first neural network includes 4 encoding blocks, each of which includes 4 or 5 residual units.

[0419] In some embodiments, the target bit rate mode is used to indicate the use of a target code table to perform signal encoding on the coding features of the audio signal; the signal encoding module 5553 is also used to quantize the coding features of the audio signal through the target code table to obtain the quantized value of the coding features; and perform entropy encoding on the quantized value through the target code rate corresponding to the target code table to obtain the audio code stream of the audio signal.

[0420] In some embodiments, when the target coding mode is an ultra-wideband coding mode, the extraction module 5552 is also used to perform sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal; extract the low-frequency features of the low-frequency sub-band signal from the low-frequency sub-band signal through a second neural network; perform high-frequency analysis processing on the high-frequency sub-band signal to obtain high-frequency features of the high-frequency sub-band signal; and determine the low-frequency features and the high-frequency features as coding features of the audio signal.

[0421] In some embodiments, the extraction module 5552 is also used to perform frame processing on the high-frequency sub-band signal to obtain multiple sub-frames of the high-frequency sub-band signal; perform frequency band expansion processing on each of the sub-frames to obtain the sub-band spectrum envelope of each of the sub-frames; and use the sub-band spectrum envelopes corresponding to the multiple sub-frames as the high-frequency features of the high-frequency sub-band signal.

[0422] In some embodiments, the extraction module 5552 is also used to perform frequency domain transform processing based on multiple sample points included in the subframe to obtain transform coefficients corresponding to the multiple sample points respectively; divide the transform coefficients corresponding to the multiple sample points respectively into multiple sub-bands; perform mean processing on the transform coefficients included in each of the sub-bands to obtain the average energy corresponding to each of the sub-bands, and use the average energy as the sub-band spectrum envelope corresponding to each of the sub-bands.

[0423] In some embodiments, the target bit rate mode includes an instruction to use a first code table to perform signal encoding on the low-frequency feature, and to use a second code table to perform signal encoding on the high-frequency feature; the signal encoding module 5553 is also used to quantize the low-frequency feature through the first code table to obtain the quantized value of the low-frequency feature, and to perform entropy encoding on the quantized value of the low-frequency feature through the code rate corresponding to the first code table to obtain the low-frequency code stream of the low-frequency sub-band signal; to quantize the high-frequency feature through the second code table to obtain the quantized value of the high-frequency feature, and to perform entropy encoding on the quantized value of the high-frequency feature through the code rate corresponding to the second code table to obtain the high-frequency code stream of the high-frequency sub-band signal; and to construct the audio code stream of the audio signal based on the low-frequency code stream and the high-frequency code stream.

[0424] In some embodiments, the target bit rate mode also includes an instruction to use a third code table to perform residual encoding on the residual features of the high-frequency features; the signal encoding module 5553 is also used to determine the subband corresponding to the subframe of the high-frequency subband signal, and divide the subband into multiple subsets; perform mean processing on the transform coefficients included in each of the subsets to obtain the average energy corresponding to each of the subsets, and use the average energy as the envelope value corresponding to each of the subsets; determine the quantization value of the subband spectrum envelope of the subband corresponding to the subset in the quantization value of the high-frequency feature, and use the envelope value corresponding to the subset as the quantization value of the subband spectrum envelope. The difference between the quantized value of the sub-band spectral envelope value and the quantized value of the sub-band spectral envelope is used as the first residual value; based on the first residual value, the residual feature of the high-frequency feature is determined; the residual feature of the high-frequency feature is quantized through the third code table to obtain the quantized value of the residual feature of the high-frequency feature, and the quantized value of the residual feature of the high-frequency feature is entropy encoded through the code rate corresponding to the third code table to obtain the residual code stream of the residual feature; the construction module 5554 is also used to use the low-frequency code stream, the high-frequency code stream and the residual code stream as the audio code stream of the audio signal.

[0425] In some embodiments, when the number of residual values ​​is N, N is a positive integer greater than 1, and the signal encoding module 5553 is also used to determine the quantization value of the nth residual value, and determine the sum of the quantization value of the nth residual value and the quantization value of the subband spectrum envelope of the corresponding subband; the difference between the envelope value corresponding to the subset and the sum is used as the n+1th residual value; the N residual values ​​are determined as the residual features of the high-frequency features; wherein n is a successively increasing positive integer, 1≤n≤N.

[0426] In some embodiments, before generating the audio code stream encapsulation of the audio signal based on the audio code stream and the frame header, the signal encoding module 5553 is also used to determine the flatness edge information of the high-frequency sub-band signal; the generation module 5555 is also used to combine the audio code stream, the frame header and the flatness edge information to obtain the audio code stream encapsulation of the audio signal.

[0427] In some embodiments, the signal encoding module 5553 is also used to divide the transform coefficients included in the subframe of the high-frequency subband signal into multiple blocks; determine the first flatness of each of the blocks, and determine the second flatness of the low-frequency specified frequency band of the audio signal; when the first flatness is less than the second flatness, or the first flatness is less than a flatness threshold, set the flatness side information to a first value, wherein the first value indicates that flattening processing is required during audio decoding; when the first flatness is greater than or equal to the second flatness, and the first flatness is greater than or equal to the flatness threshold, set the flatness side information to a second value, wherein the second value indicates that flattening processing is not required during audio decoding.

[0428] In some embodiments, the frame header further includes at least one channel bit, and the channel bit is used to indicate whether mono encoding or stereo encoding is used to perform audio encoding on the audio signal.

[0429] The audio decoding device 655 includes a series of modules, including a first acquisition module 6551, a signal decoding module 6552, and a reconstruction module 6553. The following further describes the solution of implementing audio encoding by cooperating among the modules in the audio encoding device 555 provided in the embodiment of the present application.

[0430] The first acquisition module 6551 is used to respond to a decoding request for an audio code stream encapsulation and obtain a target coding mode and a target bit rate mode from a frame header included in the audio code stream encapsulation; wherein the audio code stream included in the audio code stream encapsulation is obtained by audio encoding an audio signal through the target coding mode and the target bit rate mode, the target coding mode is obtained from multiple coding modes, and the target bit rate mode is obtained from multiple bit rate modes; the signal decoding module 6552 is used to perform signal decoding processing on the audio code stream through the target coding mode and the target bit rate mode to obtain a coding feature estimation value corresponding to the audio code stream; the reconstruction module 6553 is used to reconstruct the coding feature estimation value corresponding to the audio code stream through the target coding mode to obtain a synthetic audio signal corresponding to the audio code stream.

[0431] In some embodiments, when the target coding mode is a broadband coding mode, the target bit rate mode is used to indicate the use of a target code table to perform signal encoding on the coding features of the audio signal; the signal decoding module 6552 is also used to perform entropy decoding on the audio bit stream using the target bit rate corresponding to the target code table to obtain a quantization value corresponding to the audio bit stream; and perform inverse quantization on the quantization value corresponding to the audio bit stream using the target code table to obtain an estimated value of the coding features corresponding to the audio bit stream.

[0432] In some embodiments, when the target coding mode is a wideband coding mode, the reconstruction module 6553 is also used to call a third neural network based on the wideband coding mode; through the third neural network, the coding feature estimation value corresponding to the audio bit stream is reconstructed to obtain a synthetic audio signal corresponding to the audio bit stream.

[0433] In some embodiments, the reconstruction module 6553 is also used to perform residual processing on the coding feature estimation value corresponding to the audio code stream using at least one residual unit included in the third neural network to obtain the audio feature estimation value corresponding to the audio code stream; and perform feature reconstruction processing on the audio feature estimation value corresponding to the audio code stream to obtain the synthetic audio signal corresponding to the audio code stream.

[0434] In some embodiments, the third neural network includes 4 decoding blocks, each of which includes 4 or 5 residual units.

[0435] In some embodiments, when the target coding mode is an ultra-wideband coding mode, the target bit rate mode includes an indication of using a first code table to perform signal encoding on the low-frequency features of the audio signal, and using a second code table to perform signal encoding on the high-frequency features of the audio signal, and the audio code stream includes a low-frequency code stream and a high-frequency code stream; the signal decoding module 6552 is further used to obtain the low-frequency code stream and the high-frequency code stream from the audio code stream through the target coding mode; perform entropy decoding processing on the low-frequency code stream through the code rate corresponding to the first code table to obtain a quantization value corresponding to the low-frequency code stream, and perform inverse quantization processing on the quantization value corresponding to the low-frequency code stream through the first code table to obtain a low-frequency feature estimation value corresponding to the low-frequency code stream; perform entropy decoding processing on the high-frequency code stream through the code rate corresponding to the second code table to obtain a quantization value corresponding to the high-frequency code stream, and perform inverse quantization processing on the quantization value corresponding to the high-frequency code stream through the second code table to obtain a high-frequency feature estimation value corresponding to the high-frequency code stream; determine the coding feature estimation value corresponding to the audio code stream based on the low-frequency feature estimation value and the high-frequency feature estimation value.

[0436] In some embodiments, the audio code stream also includes a residual code stream, and the target bit rate mode also includes an instruction to use a third code table to perform residual encoding on the residual features of the high-frequency features corresponding to the audio code stream; the signal decoding module 6552 is also used to perform entropy decoding processing on the residual code stream at the code rate corresponding to the third code table to obtain a quantization value corresponding to the residual code stream, and perform inverse quantization processing on the quantization value corresponding to the residual code stream through the third code table to obtain a residual feature estimation value corresponding to the residual code stream; the high-frequency feature estimation value and the residual feature estimation value are summed to determine the high-frequency feature final estimation value; the low-frequency feature estimation value and the high-frequency feature final estimation value are determined as the coding feature estimation value corresponding to the audio code stream.

[0437] In some embodiments, the reconstruction module 6553 is also used to, when the target coding mode is ultra-wideband coding, perform feature reconstruction processing on the low-frequency feature estimation value included in the coding feature through a fourth neural network to obtain a low-frequency sub-band signal estimation value corresponding to the audio code stream; perform high-frequency reconstruction processing on the high-frequency feature estimation value included in the coding feature to obtain a high-frequency sub-band signal estimation value corresponding to the audio code stream; perform sub-band synthesis on the low-frequency sub-band signal estimation value and the high-frequency sub-band signal estimation value to obtain a synthesized audio signal corresponding to the audio code stream.

[0438] In some embodiments, the reconstruction module 6553 is also used to perform frequency domain transform processing on the sample points of the first half and the sample points of the second half included in the low-frequency sub-band signal estimation value to obtain a first transform coefficient corresponding to the sample points of the first half and a second transform coefficient corresponding to the sample points of the second half; based on the first transform coefficient, the high-frequency feature estimation value is inversely processed by frequency band expansion to obtain a first high-frequency sub-band signal estimation value; based on the second transform coefficient, the high-frequency feature estimation value is inversely processed by frequency band expansion to obtain a second high-frequency sub-band signal estimation value; and the first high-frequency sub-band signal estimation value and the second high-frequency sub-band signal estimation value are combined to obtain a high-frequency sub-band signal estimation value corresponding to the audio code stream.

[0439] In some embodiments, the reconstruction module 6553 is also used to perform spectrum replication processing on the transform coefficients of the second half of the first transform coefficients to obtain first reference transform coefficients of a reference high-frequency sub-band signal; based on the first half of the sub-band spectrum envelope corresponding to the high-frequency feature estimation value, perform gain processing on the first reference transform coefficient to obtain a first reference transform coefficient after gain; perform inverse frequency domain transform processing on the first reference transform coefficient after gain to obtain a first high-frequency sub-band signal estimation value.

[0440] In some embodiments, the audio bitstream encapsulation includes flatness side information; after performing spectral replication processing on the latter half of the transform coefficients in the first transform coefficients to obtain the first reference transform coefficients of the reference high-frequency subband signal, the reconstruction module 6553 is further configured to determine the (i-1)th first reference transform coefficient, the ith first reference transform coefficient, and the (i+1)th first reference transform coefficient when the flatness side information indicates that flattening processing is required during audio decoding; determine the average power spectrum based on the (i-1)th first reference transform coefficient, the ith first reference transform coefficient, and the (i+1)th first reference transform coefficient; use the ratio of the ith first reference transform coefficient to the average power spectrum as the new ith first reference transform coefficient; where 1 < i < I, i is a positive integer, and I is the number of the first reference transform coefficients.

[0441] In some embodiments, the frame header further includes at least 1 channel bit, and the channel bit is used to indicate whether to use a mono decoding method or a stereo decoding method to perform audio decoding on the audio bitstream.

[0442] An embodiment of the present application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions, so that the electronic device executes the audio encoding method or the audio decoding method described above in the embodiments of the present application.

[0443] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, where the computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the processor will be caused to execute the audio encoding method or the audio decoding method provided in the embodiments of the present application. For example, Figure 4A the audio encoding method shown.

[0444] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or it may be various electronic devices including one or any combination of the above memories.

[0445] In some embodiments, computer-executable instructions (referred to as executable instructions for short) may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0446] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0447] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0448] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0449] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An audio encoding method, characterized in that: The method comprises: In response to an encoding request for an audio signal, acquiring a target bit rate mode for the audio signal from a plurality of bit rate modes; Extracting a coding feature of the audio signal from the audio signal; Performing signal encoding processing on the encoding feature of the audio signal through the target bit rate mode to obtain an audio bit stream of the audio signal; Based on the target bit rate mode, determining a frame header; An audio code stream encapsulation of the audio signal is generated based on the audio code stream and the frame header.

2. The method according to claim 1, characterized in that: The step of extracting the coding feature of the audio signal from the audio signal comprises: Call the first neural network; The encoding features of the audio signal are extracted from the audio signal through the first neural network.

3. The method according to claim 1 or 2, characterized in that: The target bit rate mode is used to indicate that a target code table is used to perform signal encoding on the coding feature of the audio signal; The step of performing signal encoding processing on the encoding feature of the audio signal through the target bit rate mode to obtain an audio bit stream of the audio signal includes: quantizing the coding feature of the audio signal using the target code table to obtain a quantized value of the coding feature; The quantized value is entropy encoded using a target bit rate corresponding to the target code table to obtain an audio bit stream of the audio signal.

4. An audio encoding method, characterized in that: The method comprises: In response to an encoding request for an audio signal, acquiring a target bit rate mode for the audio signal from a plurality of bit rate modes; Extracting low-frequency features from the low-frequency sub-band signal of the audio signal, and extracting high-frequency features from the high-frequency sub-band signal of the audio signal; By using the target bit rate mode, signal encoding processing is performed on the low-frequency feature and the high-frequency feature respectively to obtain a low-frequency code stream of the low-frequency sub-band signal and a high-frequency code stream of the high-frequency sub-band signal; Based on the target bit rate mode, determining a frame header; An audio code stream encapsulation of the audio signal is generated based on the low-frequency code stream, the high-frequency code stream and the frame header.

5. The method according to claim 4, characterized in that The extracting the low-frequency feature from the low-frequency sub-band signal of the audio signal comprises: extracting low-frequency features of the low-frequency sub-band signal from the low-frequency sub-band signal through a second neural network; The extracting high-frequency features from the high-frequency sub-band signal of the audio signal comprises: Perform high-frequency analysis processing on the high-frequency sub-band signal to obtain high-frequency features of the high-frequency sub-band signal.

6. The method according to claim 4, characterized in that The target bit rate mode includes an instruction to use a first code table to encode the low-frequency feature and a second code table to encode the high-frequency feature; The step of performing signal encoding processing on the low-frequency feature and the high-frequency feature respectively through the target bit rate mode to obtain a low-frequency code stream of the low-frequency sub-band signal and a high-frequency code stream of the high-frequency sub-band signal comprises: quantizing the low-frequency feature using the first code table to obtain a quantized value of the low-frequency feature, and entropy encoding the quantized value of the low-frequency feature using a code rate corresponding to the first code table to obtain a low-frequency code stream of the low-frequency subband signal; The high-frequency feature is quantized by the second code table to obtain the quantized value of the high-frequency feature, and the quantized value of the high-frequency feature is entropy encoded by the code rate corresponding to the second code table to obtain the high-frequency code stream of the high-frequency subband signal.

7. The method according to claim 6, characterized in that The target bit rate mode further includes an instruction to use a third code table to perform residual coding on the residual features of the high-frequency features; The method further comprises: Determine a subband corresponding to a subframe of the high frequency subband signal, and divide the subband into a plurality of subsets; Performing mean processing on the transform coefficients included in each of the subsets to obtain average energy corresponding to each of the subsets, and using the average energy as the envelope value corresponding to each of the subsets; Determine the quantized value of the subband spectrum envelope of the subband corresponding to the subset in the quantized value of the high-frequency feature, and use the difference between the envelope value corresponding to the subset and the quantized value of the subband spectrum envelope as the first residual value; Determining a residual feature of the high-frequency feature based on the first residual value; quantizing the residual feature of the high-frequency feature by using the third code table to obtain a quantized value of the residual feature of the high-frequency feature, and entropy encoding the quantized value of the residual feature of the high-frequency feature by using a code rate corresponding to the third code table to obtain a residual code stream of the residual feature; The step of generating an audio code stream encapsulation of the audio signal based on the low-frequency code stream, the high-frequency code stream and the frame header includes: An audio code stream encapsulation of the audio signal is generated based on the low-frequency code stream, the high-frequency code stream, the residual code stream and the frame header.

8. The method according to claim 7, characterized in that When the number of the residual values ​​is N, N is a positive integer greater than 1, the determining the residual feature of the high-frequency feature based on the first residual value includes: Determine a quantized value of an nth residual value, and determine a sum of the quantized value of the nth residual value and a quantized value of a sub-band spectral envelope of a corresponding sub-band; Taking the difference between the envelope value corresponding to the subset and the sum as the (n+1)th residual value; Determine the N residual values ​​as residual features of the high-frequency features; Wherein, n is a positive integer which increases successively, 1≤n≤N.

9. The method according to any one of claim 4, characterized in that: Before generating the audio code stream encapsulation of the audio signal based on the low-frequency code stream, the high-frequency code stream and the frame header, the method further includes: Determining flatness edge information of the high frequency sub-band signal; The step of generating an audio code stream encapsulation of the audio signal based on the low-frequency code stream, the high-frequency code stream and the frame header includes: The low-frequency code stream, the high-frequency code stream, the frame header, and the flatness edge information are combined to obtain an audio code stream encapsulation of the audio signal.

10. The method according to claim 9, characterized in that The step of determining the flatness edge information of the high frequency sub-band signal comprises: Dividing the transform coefficients included in the subframe of the high frequency subband signal into a plurality of blocks; determining a first flatness of each of the blocks, and determining a second flatness of a low-frequency designated frequency band of the audio signal; When the first flatness is less than the second flatness, or the first flatness is less than a flatness threshold, setting the flatness edge information to a first value, wherein the first value indicates that flattening processing is required during audio decoding; When the first flatness is greater than or equal to the second flatness, and the first flatness is greater than or equal to the flatness threshold, the flatness edge information is set to a second value, wherein the second value indicates that flattening processing is not required during audio decoding.

11. An audio decoding method, characterized in that: The method comprises: In response to a decoding request for an audio code stream encapsulation, obtaining a target bit rate mode from a frame header included in the audio code stream encapsulation; The audio code stream included in the audio code stream encapsulation is obtained by audio encoding an audio signal through the target code rate mode, and the target code rate mode is obtained from a plurality of code rate modes; By using the target bit rate mode, signal decoding processing is performed on the audio bit stream to obtain a coding feature estimation value corresponding to the audio bit stream; The coding feature estimation value corresponding to the audio code stream is reconstructed to obtain a reconstructed audio signal corresponding to the audio code stream.

12. The method according to claim 11, characterized in that The target bit rate mode is used to indicate that a target code table is used to perform signal encoding on the coding feature of the audio signal; The step of performing signal decoding processing on the audio bitstream through the target bitrate mode to obtain a coding feature estimation value corresponding to the audio bitstream includes: Performing entropy decoding processing on the audio code stream according to the target code rate corresponding to the target code table to obtain a quantization value corresponding to the audio code stream; The target code table is used to perform inverse quantization processing on the quantized value corresponding to the audio code stream to obtain the coding feature estimation value corresponding to the audio code stream.

13. The method according to claim 11 or 12, characterized in that: The reconstructing the estimated value of the coding feature corresponding to the audio code stream to obtain a reconstructed audio signal corresponding to the audio code stream includes: Call the third neural network; The third neural network is used to reconstruct the estimated value of the coding feature corresponding to the audio code stream to obtain a reconstructed audio signal corresponding to the audio code stream.

14. An audio decoding method, characterized in that: The method comprises: In response to a decoding request for an audio code stream encapsulation, obtaining a target bit rate mode from a frame header included in the audio code stream encapsulation; The low-frequency code stream and the high-frequency code stream included in the audio code stream encapsulation are obtained by respectively performing audio encoding on the low-frequency sub-band signal and the high-frequency sub-band signal of the audio signal through the target code rate mode, and the target code rate mode is obtained from multiple code rate modes; By using the target bit rate mode, signal decoding processing is performed on the low-frequency bit stream and the high-frequency bit stream respectively to obtain a low-frequency feature estimation value and a high-frequency feature estimation value; Reconstructing the low-frequency feature estimation value and the high-frequency feature estimation value to obtain a low-frequency sub-band signal estimation value and a high-frequency sub-band signal estimation value; A reconstructed audio signal is determined based on the low frequency subband signal estimate value and the high frequency subband signal estimate value.

15. The method according to claim 14, characterized in that The target bit rate mode includes an instruction to use a first code table to encode the low-frequency characteristics of the audio signal, and use a second code table to encode the high-frequency characteristics of the audio signal; The step of performing signal decoding processing on the low-frequency code stream and the high-frequency code stream respectively through the target code rate mode to obtain a low-frequency feature estimation value and a high-frequency feature estimation value includes: Performing entropy decoding processing on the low-frequency code stream according to the code rate corresponding to the first code table to obtain a quantization value corresponding to the low-frequency code stream, and performing inverse quantization processing on the quantization value corresponding to the low-frequency code stream according to the first code table to obtain a low-frequency feature estimation value corresponding to the low-frequency code stream; The high-frequency code stream is entropy decoded using the code rate corresponding to the second code table to obtain a quantization value corresponding to the high-frequency code stream, and the quantization value corresponding to the high-frequency code stream is inversely quantized using the second code table to obtain a high-frequency feature estimation value corresponding to the high-frequency code stream.

16. The method according to claim 15, characterized in that The audio code stream encapsulation further includes a residual code stream, and the target code rate mode further includes an instruction to use a third code table to perform residual coding on the residual features of the high-frequency features; The method further comprises: The residual code stream is entropy decoded using the code rate corresponding to the third code table to obtain a quantization value corresponding to the residual code stream, and the quantization value corresponding to the residual code stream is inversely quantized using the third code table to obtain a residual feature estimation value corresponding to the residual code stream.

17. The method according to claim 14, characterized in that The reconstructing the low-frequency feature estimation value and the high-frequency feature estimation value to obtain a low-frequency sub-band signal estimation value and a high-frequency sub-band signal estimation value respectively includes: Performing feature reconstruction processing on the low-frequency feature estimation value through a fourth neural network to obtain a low-frequency sub-band signal estimation value; The high-frequency feature estimation value is subjected to high-frequency reconstruction processing to obtain a high-frequency sub-band signal estimation value.

18. The method according to claim 17, characterized in that The step of performing high-frequency reconstruction processing on the high-frequency feature estimation value to obtain a high-frequency sub-band signal estimation value includes: Performing frequency domain transform processing on the sample points of the first half and the sample points of the second half included in the low-frequency subband signal estimation value to obtain first transform coefficients corresponding to the sample points of the first half and second transform coefficients corresponding to the sample points of the second half; Based on the first transform coefficient, performing inverse processing of frequency band expansion on the high-frequency feature estimation value to obtain a first high-frequency sub-band signal estimation value; Based on the second transform coefficient, performing inverse processing of frequency band expansion on the high-frequency feature estimation value to obtain a second high-frequency sub-band signal estimation value; The first high frequency sub-band signal estimation value and the second high frequency sub-band signal estimation value are combined to obtain a high frequency sub-band signal estimation value.

19. The method according to claim 18, characterized in that The step of performing inverse processing of frequency band expansion on the high frequency feature estimation value based on the first transform coefficient to obtain a first high frequency subband signal estimation value comprises: Perform spectral replication processing on the transformation coefficients in the second half of the first transformation coefficients to obtain first reference transformation coefficients of a reference high-frequency subband signal; Based on the subband spectral envelope of the first half corresponding to the high-frequency feature estimation value, perform gain processing on the first reference transformation coefficients to obtain the first reference transformation coefficients after gain; Perform inverse frequency-domain transformation processing on the first reference transformation coefficients after gain to obtain an estimated value of the first high-frequency subband signal.

20. The method according to claim 19, wherein The audio bitstream encapsulation includes flatness side information; After performing spectral replication processing on the transformation coefficients in the second half of the first transformation coefficients to obtain first reference transformation coefficients of a reference high-frequency subband signal, the method further includes: When the flatness side information indicates that flattening processing is required during audio decoding, determine the (i - 1)-th first reference transformation coefficient, the i-th first reference transformation coefficient, and the (i + 1)-th first reference transformation coefficient; Based on the (i - 1)-th first reference transformation coefficient, the i-th first reference transformation coefficient, and the (i + 1)-th first reference transformation coefficient, determine the average power spectrum; Use the ratio of the i-th first reference transformation coefficient to the average power spectrum as the new i-th first reference transformation coefficient; where 1 < i < I, i is a positive integer, and I is the number of the first reference transformation coefficients.

21. A method for processing an audio code stream, characterized in that: Store an audio bitstream on a non-volatile computer-readable storage medium, where the audio bitstream is generated based on the audio encoding method according to any one of claims 1 to 10, or is decoded according to the audio decoding method according to any one of claims 11 to 20.

22. An audio decoding device, characterized in that: The apparatus includes: A first acquisition module, configured to respond to a decoding request for an audio bitstream encapsulation, and obtain a target bitrate mode from a frame header included in the audio bitstream encapsulation; wherein the audio bitstream included in the audio bitstream encapsulation is obtained by performing audio encoding on an audio signal through the target bitrate mode, and the target bitrate mode is obtained from a plurality of bitrate modes; A signal decoding module, configured to perform signal decoding processing on the audio bitstream through the target bitrate mode to obtain an estimated value of the encoding features corresponding to the audio bitstream; A reconstruction module, configured to perform reconstruction processing on the estimated value of the encoding features corresponding to the audio bitstream to obtain a reconstructed audio signal corresponding to the audio bitstream.

23. A computer-readable storage medium, characterized in that: Store computer-executable instructions that, when executed by a processor, implement the audio encoding method according to any one of claims 1 to 10, or the audio decoding method according to any one of claims 11 to 20.

Images