Audio encoding method, audio decoding method, device, electronic device, and computer program

Abstract

In relation to artificial intelligence technology, an audio encoding method, an audio decoding method, an apparatus, an electronic device, and a storage medium are provided, which include the steps of: performing a downsampling process on an audio signal to obtain a low-frequency signal of the audio signal; performing a low-frequency feature extraction process on the audio signal to obtain a low-frequency feature of the audio signal; performing a high-frequency analysis process on the audio signal to obtain a high-frequency feature of the audio signal, where a feature dimension of the high-frequency feature is lower than that of the low-frequency feature; performing an encoding process on the low-frequency feature to obtain a low-frequency bitstream of the audio signal; and performing an encoding process on the high-frequency feature to obtain a high-frequency bitstream of the audio signal.

Description

[Technical Field]

[0001] The embodiments of the present invention claim priority based on Chinese patent application having application number 202310597138.3 and filing date May 24, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to artificial intelligence technology, and in particular to an audio encoding method, an audio decoding method, an apparatus, an electronic device, and a storage medium. [Background technology]

[0003] Artificial intelligence (AI) is a comprehensive technology in computer science that aims to equip devices with sensing, reasoning, and decision-making capabilities through research into the design principles and implementation methods of various intelligent devices. AI technology is a comprehensive discipline that covers a wide range of fields, including natural language processing, machine learning, and deep learning. With the development of technology, AI technology is being applied in more and more fields, demonstrating increasingly important value.

[0004] Audio coding / decoding technology is an important application in the field of artificial intelligence and a key technology for communication services, including remote audio / video calls. Simply put, audio coding technology aims to transmit as much audio information as possible using limited network bandwidth resources. From the perspective of Shannon information theory, audio coding is a type of source coding. The objective of source coding is to compress the amount of data to be transmitted as much as possible on the encoding side, eliminate redundancy in the information, and restore the data losslessly (or nearly so) on the decoding side.

[0005] In the related art, in order to guarantee the audio quality during the encoding process, the efficiency of audio encoding is greatly reduced. Summary of the Invention

[0006] Embodiments of the present invention provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product that can ensure audio quality and improve the efficiency of audio encoding.

[0007] The technical solutions of the embodiments of the present invention are as follows:

[0008] An embodiment of the present invention provides an audio encoding method executed by an electronic device, the method comprising: performing a downsampling process on an audio signal to obtain low-frequency features of the audio signal; performing a low-frequency feature extraction process on the audio signal to obtain low-frequency features of the audio signal; performing a high-frequency analysis process on the audio signal to obtain high-frequency features of the audio signal, wherein a feature dimension of the high-frequency features is lower than a feature dimension of the low-frequency features; performing an encoding process on the low-frequency features to obtain a low-frequency bitstream of the audio signal; and performing an encoding process on the high-frequency features to obtain a high-frequency bitstream of the audio signal.

[0009] An embodiment of the present invention provides an audio decoding method, executed by an electronic device, comprising the steps of: performing a decoding process on a low-frequency bitstream of an audio signal to obtain low-frequency features corresponding to the low-frequency bitstream; performing a decoding process on a high-frequency bitstream of the audio signal to obtain high-frequency features corresponding to the high-frequency bitstream, where the low-frequency bitstream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and a feature dimension of the high-frequency features is lower than a feature dimension of the low-frequency features; performing a low-frequency feature reconstruction process on the low-frequency features to obtain low-frequency signals corresponding to the low-frequency features; performing an upsampling process on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal; and performing a signal reconstruction process on the high-frequency features and the up-sampled signal to obtain a synthesized audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

[0010] In the above aspect, the upsampling process is realized by a plurality of cascaded decoding layers, and the step of performing the upsampling process on the convolutional features and obtaining the upsampling feature of the low-frequency features includes the steps of performing the upsampling process on the convolutional features by a first decoding layer of the cascaded plurality of decoding layers, outputting the upsampling result of the first decoding layer to a subsequent cascaded decoding layer, and continuing the upsampling process and output of the upsampling result by the subsequent cascaded decoding layer until it is output to the last decoding layer, and determining the upsampling result output by the last decoding layer as the upsampling feature of the low-frequency features.

[0011] An embodiment of the present invention provides an audio encoding device, including: a downsampling module that performs a downsampling process on an audio signal to obtain a low-frequency signal of the audio signal; a low-frequency extraction module that performs a low-frequency feature extraction process on the audio signal to obtain low-frequency features of the audio signal; a high-frequency analysis module that performs a high-frequency analysis process on the audio signal to obtain high-frequency features of the audio signal, wherein a feature dimension of the high-frequency features is lower than a feature dimension of the low-frequency features; and an encoding module that performs an encoding process on the low-frequency features to obtain a low-frequency bitstream of the audio signal, and performs an encoding process on the high-frequency features to obtain a high-frequency bitstream of the audio signal.

[0012] In an embodiment of the present invention, there is provided an audio decoding device, the device including: a decoding module that performs a decoding process on a low-frequency bitstream of an audio signal to obtain low-frequency features corresponding to the low-frequency bitstream; and a decoding process on a high-frequency bitstream of the audio signal to obtain high-frequency features corresponding to the high-frequency bitstream, where the low-frequency bitstream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and a feature dimension of the high-frequency features is lower than a feature dimension of the low-frequency features; a low-frequency reconstruction module that performs a low-frequency feature reconstruction process on the low-frequency features to obtain a low-frequency signal corresponding to the low-frequency features; an upsampling module that performs an upsampling process on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal; and a signal reconstruction module that performs a signal reconstruction process on the high-frequency features and the up-sampled signal to obtain a synthesized audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

[0013] In an embodiment of the present invention, there is provided an electronic device including a memory having a computer program or computer-executable instructions stored therein, and a processor that, when executing the computer program or computer-executable instructions stored in the memory, implements an audio encoding method or an audio decoding method according to an embodiment of the present invention.

[0014] An embodiment of the present invention provides a computer-readable storage medium having stored thereon a computer program or computer-executable instructions, the computer program or computer-executable instructions being adapted to implement an audio encoding method or an audio decoding method according to an embodiment of the present invention when executed by a processor.

[0015] In an embodiment of the present invention, there is provided a computer program product comprising a computer program or computer-executable instructions, which, when executed by a processor, implements an audio encoding method or an audio decoding method according to an embodiment of the present invention.

[0016] The embodiment of the present invention has the following advantageous effects.

[0017] The audio signal is downsampled to obtain a low-frequency signal. Since the low-frequency signal has a greater impact on audio coding than the high-frequency signal in the audio signal, differentiated signal processing is used to extract low-frequency features and high-frequency features of the audio signal, so that the feature dimension of the high-frequency feature is lower than that of the low-frequency feature, and the low-frequency features and high-frequency features with reduced feature dimension are coded, thereby ensuring audio quality and improving audio coding efficiency. [Brief explanation of the drawings]

[0018] [Figure 1]FIG. 2 is a schematic diagram of a comparison of frequency spectra at different bit rates according to an embodiment of the present invention; [Figure 2] 1 is a schematic diagram of the architecture of an audio encoding / decoding system according to an embodiment of the present invention; [Figure 3A] 3A and 3B are schematic diagrams illustrating the configuration of an electronic device according to an embodiment of the present invention. [Figure 3B] 3A and 3B are schematic diagrams illustrating the configuration of an electronic device according to an embodiment of the present invention. [Figure 4A] FIG. 1 is a schematic diagram of a typical convolutional network according to an embodiment of the present invention. [Figure 4B] FIG. 1 is a schematic diagram of a dilated convolutional network according to an embodiment of the present invention. [Figure 5] FIG. 1 is a schematic diagram of frequency band extension according to an embodiment of the present invention. [Figure 6A] 1 is a first exemplary flowchart of an audio encoding method according to an embodiment of the present invention. [Figure 6B] 10 is a second exemplary flowchart of an audio encoding method according to an embodiment of the present invention. [Figure 6C] 10 is a third exemplary flowchart of an audio encoding method according to an embodiment of the present invention. [Figure 6D] 10 is a fourth exemplary flowchart of an audio encoding method according to an embodiment of the present invention. [Figure 7A] 1 is a first exemplary flowchart of an audio decoding method according to an embodiment of the present invention. [Figure 7B] 10 is a second exemplary flowchart of an audio decoding method according to an embodiment of the present invention. [Figure 7C] 10 is a third exemplary flowchart of an audio decoding method according to an embodiment of the present invention. [Figure 7D] 10 is a fourth exemplary flowchart of an audio decoding method according to an embodiment of the present invention. [Figure 8] 1 is a schematic diagram of an end-to-end voice communication link according to an embodiment of the present invention; [Figure 9]1 is an exemplary flowchart of a low-rate neural network encoding / decoding method according to an embodiment of the present invention. [Figure 10] FIG. 2 is a schematic diagram of a first neural network according to an embodiment of the present invention. [Figure 11] FIG. 4 is a schematic diagram of a second neural network according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] In order to make the objectives, technical means and advantages of the present invention clearer, the following will further describe in detail the embodiments of the present invention with reference to the drawings. The described embodiments do not limit the present invention, and all other embodiments that can be obtained by those skilled in the art without creative work belong to the protection scope of the present invention.

[0020] In the following description, references to the terms "first" and "second" are merely intended to distinguish between similar objects and do not represent a particular order of the objects. It should be noted that "first" and "second" may be interchanged with respect to a particular order or order of precedence, where permitted, such that the described embodiments of the invention may be practiced in orders other than those shown or described herein.

[0021] In the following description, references to "some embodiments" describe a subset of all possible embodiments, but the "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other unless inconsistent.

[0022] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terms used herein are for the purpose of describing only the embodiments of the present invention and are not intended to limit the present invention.

[0023] Before describing the embodiments of the present invention in more detail, the nouns and terms contained in the embodiments of the present invention will be explained, and the nouns and terms contained in the embodiments of the present invention are suitable for the following interpretation.

[0024] (1) Neural Network (NN): A mathematical model of an algorithm that performs distributed parallel information processing, imitating the behavioral characteristics of animal neural networks. Such neural networks rely on the complexity of the system to achieve the goal of information processing by adjusting the interconnections between the many internal nodes.

[0025] (2) Deep Learning (DL): This is a new research direction in the field of machine learning (ML). Deep learning learns the inherent rules and representation levels of sample data, and the information gained during the learning process is useful for interpreting data such as text, images, and voice. The ultimate goal is for machines to have the analytical learning ability of humans and be able to recognize data such as text, images, and voice.

[0026] (3) Quantization: A process of approximating continuous values ​​(or a large number of discrete values) of a signal to a finite number (or fewer) of discrete values. Here, quantization includes vector quantization (VQ) and scalar quantization.

[0027] Vector quantization is an effective lossy compression technique, and its theoretical basis is Shannon's rate-distortion theory. The basic principle of vector quantization is to use the index of the codeword in a codebook that best matches the input vector instead of the input vector for transmission and storage, and only a simple table lookup operation is required during decoding. For example, several scalar data are organized into a vector space, and the vector space is divided into several subregions. During quantization, for vectors that fall into the subregions, the corresponding index is used instead of the input vector.

[0028] Scalar quantization is the quantization of a scalar, i.e., one-dimensional vector quantization, which divides the dynamic range into several small regions, each of which has a representative value (index). If the input signal falls into a certain range, the input signal is quantized to that representative value.

[0029] (4) Entropy coding: This is a lossless coding method that does not lose any information during the coding process according to the entropy principle. It is also an important module in lossless coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exponential-Golomb coding, and arithmetic coding.

[0030] Speech coding technology aims to transmit as much audio information as possible using fewer network bandwidth resources. Speech coding can achieve compression rates of 10x or more. In other words, compressing 10MB of original audio data with an encoder reduces the bandwidth required for transmission to just 1MB, significantly reducing the bandwidth consumed for transmitting information. For example, for a wideband audio signal with a sampling rate of 16,000 Hz, using a 16-bit sampling depth (the resolution of the audio intensity recording in the sampling), the uncompressed version would have a bitrate of 256 kbps. Even with lossy coding, within a bitrate range of 10–20 kbps, the quality of the reconstructed audio signal is close to the uncompressed version, and is considered perceptually indistinguishable. For services requiring higher sampling rates, such as ultra-wideband audio at 32,000 Hz, a bitrate range of at least 30 kbps is required.

[0031] To ensure smooth communication in communication systems, industry operators have deployed standard speech encoding / decoding protocols, such as those established by international or national standardization organizations (e.g., ITU-T, 3GPP, IETF, AVS, CCSA), as well as standardized protocols such as G.711, G.722, AMR series, EVS, and OPUS. To illustrate the relationship between compression bit rate and quality, Figure 1 is a schematic diagram comparing frequency spectra at different bit rates. Curve 101 is the original speech spectrum curve, i.e., the uncompressed signal. Curve 102 is the OPUS encoder spectrum curve at a bit rate of 20 kbps. Curve 103 is the OPUS encoder spectrum curve at a bit rate of 6 kbps. As can be seen from Figure 1, as the encoding bit rate increases, the compressed signal becomes closer to the original signal.

[0032] In the related art, the principle of speech coding is as follows: speech coding can directly code speech waveform samples sample by sample, or according to the principle of human speech production, extract relevant low-dimensional features, encode the features at the encoding side, and reconstruct the speech signal based on these parameters at the decoding side.

[0033] The above coding principles all originate from audio signal modeling, i.e., they are compression methods based on signal processing, and cannot guarantee audio coding quality. Embodiments of the present invention provide an audio coding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product based on artificial intelligence, in order to guarantee audio quality and improve coding efficiency. The following describes exemplary applications of an electronic device according to an embodiment of the present invention. The electronic device according to an embodiment of the present invention may be implemented as a terminal device, a server, or a combination of a terminal device and a server. The following describes an example in which the electronic device is implemented as a terminal device.

[0034] 2 is a schematic diagram of the architecture of an audio encoding / decoding system 100 according to an embodiment of the present invention. For example, as shown in FIG. 2, the audio encoding / decoding system 100 includes a server 200, a network 300, a terminal device 400 (i.e., encoding side), and a terminal device 500 (i.e., decoding side). Here, the network 300 may be a local area network, a wide area network, or a combination of both.

[0035] In some embodiments, the terminal device 400 executes a client 410. The client 410 may be various types of clients, such as an instant messaging client, a web conferencing client, a live streaming client, a browser, etc. In response to an audio collection command triggered by a sender (e.g., a web conference initiator, a host, an audio call initiator, etc.), the client 410 calls a microphone provided in the terminal device 400 to collect an audio signal, performs audio encoding processing on the collected audio signal, and obtains bitstreams (a high-frequency bitstream and a low-frequency bitstream).

[0036] For example, the client 410 invokes an audio encoding method according to an embodiment of the present invention to encode the collected audio signal, i.e., performs a downsampling process on the audio signal to obtain a low-frequency signal of the audio signal; performs a low-frequency feature extraction process on the low-frequency signal to obtain a low-frequency feature of the audio signal; performs a high-frequency analysis process on the audio signal to obtain a high-frequency feature of the audio signal, where the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature; performs an encoding process on the low-frequency feature to obtain a low-frequency bitstream of the audio signal; and performs an encoding process on the high-frequency feature to obtain a high-frequency bitstream of the audio signal. The encoding side (i.e., the terminal device 400) combines signal processing technology and artificial intelligence technology to downsample the audio signal to obtain the low-frequency signal. Since low-frequency signals have a greater impact on audio coding than high-frequency signals in an audio signal, differentiated signal processing is used to extract low-frequency features and high-frequency features of the audio signal, respectively, so that the feature dimension of the high-frequency features is lower than that of the low-frequency features, and the low-frequency features and high-frequency features with reduced feature dimension are coded, respectively, thereby ensuring audio quality and improving audio coding efficiency.

[0037] The client 410 transmits bitstreams of the audio signal (i.e., high-frequency and low-frequency bitstreams) to the server 200 via the network 300. This allows the server 200 to transmit the bitstreams (high-frequency and low-frequency bitstreams) to terminal devices 500 associated with recipients (e.g., participants in a web conference, viewers, recipients of a voice call, etc.).

[0038] A client 510 (e.g., an instant messaging client, a web conferencing client, a live streaming client, a browser, etc.) running on the terminal device 500 can receive the bitstreams (high-frequency bitstream and low-frequency bitstream) sent by the server 200, and then perform audio decoding processing on the bitstreams to obtain an audio signal (i.e., a synthesized voice signal) to realize audio communication.

[0039] For example, the client 510 invokes an audio decoding method according to an embodiment of the present invention to decode the received bitstreams (high-frequency bitstream and low-frequency bitstream), i.e., performs a decoding process on the low-frequency bitstream of the audio signal to obtain low-frequency features corresponding to the low-frequency bitstream, and performs a decoding process on the high-frequency bitstream of the audio signal to obtain high-frequency features corresponding to the high-frequency bitstream; performs a low-frequency feature reconstruction process on the low-frequency features to obtain a low-frequency signal corresponding to the low-frequency features; performs an upsampling process on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal; and performs a signal reconstruction process on the high-frequency features and the up-sampled signal to obtain a synthesized audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

[0040] In some embodiments, embodiments of the present invention may be implemented using Cloud Technology, which is a host technology that integrates a set of resources, such as hardware, software, and networks, within a wide area network or a local area network to realize data computation, storage, processing, and sharing.

[0041] Cloud technology is a collective term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model, which can be used as needed to form a flexible and convenient resource pool. Cloud technology is an important supporting technology. The above-mentioned service interaction function between servers 200 may be realized by cloud technology.

[0042] For example, the server 200 shown in FIG. 2 may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, big data, and artificial intelligence platforms. The terminal devices 400 and 500 shown in FIG. 2 may be, but are not limited to, various types of user terminals, such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable game devices, and in-vehicle devices), smartphones, smart speakers, smart watches, smart TVs, and in-vehicle terminals. The terminal devices (e.g., the terminal devices 400 and 500) and the server 200 may be directly or indirectly connected via wired or wireless communication, and this is not limited to the embodiments of the present invention.

[0043] In some embodiments, the terminal device or server 200 may implement the audio encoding method or audio decoding method according to the embodiments of the present invention by executing a computer program. For example, the computer program may be a native program or software module in an operating system. The computer program may be a native application (APP), i.e., a program that needs to be installed in an operating system, such as a live streaming APP, a web conferencing APP, or an instant messaging APP. Alternatively, the computer program may be an applet that can be incorporated into any APP, i.e., a program that can be executed simply by downloading it to a browser environment. In short, the computer program may be any type of application program, module, or plug-in.

[0044] In some embodiments, multiple servers may constitute a blockchain, and server 200 may be a node on the blockchain, and information connections may exist between nodes in the blockchain, and information may be transmitted between nodes. Here, data related to the audio encoding method or audio decoding method according to embodiments of the present invention (e.g., audio encoding logic, audio decoding logic, high-frequency bitstream, and low-frequency bitstream) may be stored on the blockchain.

[0045] FIG. 3A is a schematic diagram of an electronic device according to an embodiment of the present invention. As shown in FIG. 3A, the electronic device 500 is a terminal device, as an example. The electronic device 500 shown in FIG. 3A includes at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The components within the electronic device 500 are connected via a bus system 550. The bus system 550 is used to realize communication between these components. The bus system 550 includes a data bus, a power bus, a control bus, and a status signal bus. For ease of explanation, the buses in FIG. 3A are collectively referred to as the bus system 550.

[0046] Processor 520 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. Here, a general-purpose processor may be a microprocessor or any conventional processor, etc.

[0047] The user interface 540 includes one or more output devices 541 capable of displaying media content, including one or more speakers and / or one or more visual displays, and one or more input devices 542, including user interface components for providing user input, such as a keyboard, mouse, microphone, touchscreen display, camera, other input buttons and controls, etc.

[0048] Memory 550 may be removable, non-removable, or a combination thereof. Examples of hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Preferably, memory 550 includes one or more storage devices that are physically separate from processor 520.

[0049] Memory 550 may include volatile or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). Memory 550 according to embodiments of the present invention may include any suitable type of memory.

[0050] In some embodiments, memory 550 may store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as described below.

[0051] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, and a driver layer, for processing basic system services and performing hardware-based tasks.

[0052] The network communications module 552 reaches other computing devices through one or more (wired or wireless) network interfaces 530. Examples of network interfaces 530 include Bluetooth, wireless communication (WiFi), and Universal Serial Bus (USB).

[0053] In some embodiments, the audio encoding device or audio decoding device according to the present invention may be implemented in software. As shown in Fig. 3A, the audio encoding device 555 stored in the memory 550 may be software in the form of a program, a plug-in, or the like, and includes a downsampling module 5551, a low-frequency extraction module 5552, a high-frequency analysis module 5553, and an encoding module 5554. Here, the downsampling module 5551, the low-frequency extraction module 5552, the high-frequency analysis module 5553, and the encoding module 5554 are for implementing audio encoding functions. These modules are logical, and therefore can be arbitrarily combined or further divided according to the functions to be implemented.

[0054] 3B is a schematic diagram of a configuration of electronic device 600 according to an embodiment of the present invention. As shown in FIG. 3B, electronic device 600 has a configuration similar to that of electronic device 500. Electronic device 600 shown in FIG. 3B includes at least one processor 520, memory 550, at least one network interface 530, and a user interface 540. In some embodiments, the components in electronic device 500 are connected via bus system 550.

[0055] The user interface 540 includes one or more output devices 541 that allow for the display of media content. The user interface 540 includes one or more input devices 542.

[0056] In some embodiments, memory 550 may store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as described below.

[0057] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, and a driver layer, for processing basic system services and performing hardware-based tasks.

[0058] The network communications module 552 reaches other computing devices through one or more (wired or wireless) network interfaces 530. Examples of network interfaces 530 include Bluetooth, wireless communication (WiFi), and Universal Serial Bus (USB).

[0059] 3B , the audio decoding device 556 stored in the memory 550 may be software in the form of a program, a plug-in, or the like, and includes a decoding module 5561, a low-frequency reconstruction module 5562, an up-sampling module 5563, and a signal reconstruction module 5564. Here, the decoding module 5561, the low-frequency reconstruction module 5562, the up-sampling module 5563, and the signal reconstruction module 5564 are for realizing an audio decoding function, and since these modules are logical, they can be arbitrarily combined or further divided according to the functions to be implemented.

[0060] Before describing the audio encoding method and audio decoding method according to an embodiment of the present invention in detail, the dilated convolutional network and frequency band extension technology will be described below.

[0061] The following description will be given with reference to FIGS. 4A and 4B. FIG. 4A is a schematic diagram of a conventional convolutional network according to an embodiment of the present invention. FIG. 4B is a schematic diagram of a dilated convolutional network according to an embodiment of the present invention. Compared to a conventional convolutional network, a dilated convolution can increase the receptive field while maintaining the size of the feature map and avoid errors caused by upsampling and downsampling. The convolution kernel sizes shown in FIGS. 4A and 4B are both 3×3. However, the receptive field 401 of the conventional convolution shown in FIG. 4A is only 3, while the receptive field 402 of the dilated convolution shown in FIG. 4B reaches 5. In other words, for a 3×3 convolution kernel, the receptive field of the conventional convolution shown in FIG. 4A is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1. The receptive field of the dilated convolution shown in FIG. 4B is 5, and the dilation rate is 2.

[0062] The convolution kernel can also move on a plane as shown in Figure 4A or 4B. Here, we introduce the concept of stride rate (stride size). For example, every time the convolution kernel shifts by one square, the corresponding stride rate is 1.

[0063] There is also the concept of the number of convolution channels, which refers to the number of parameters corresponding to the convolution kernel used in the convolution analysis. In theory, the more channels there are, the more complete and accurate the signal analysis will be. However, the more channels there are, the higher the complexity will be. For example, a 1x320 tensor can be convolved using 24 channels, resulting in a 24x320 tensor.

[0064] Furthermore, depending on the needs of actual applications, the size of the dilated convolution kernel (for example, for a voice signal, the size of the convolution kernel may be set to 1×3), the dilation rate, the stride rate, and the number of channels may be defined by oneself, and the embodiments of the present invention are not limited thereto.

[0065] FIG. 5 is a schematic diagram of frequency band expansion (or frequency band duplication) according to an embodiment of the present invention. As shown in FIG. 5, a wideband signal is reconstructed, then the wideband signal is duplicated into an ultra-wideband signal, and finally, the signal is shaped based on the ultra-wideband envelope. The frequency domain aspect shown in FIG. 5 specifically includes (1) implementing one kernel layer coding at a low sampling rate, (2) selecting and duplicating a low-frequency portion of the frequency spectrum to a high frequency, and (3) performing gain control on the duplicated high-frequency spectrum based on pre-recorded boundary information (which describes, for example, the energy correlation between high and low frequencies). The effect of doubling the sampling rate can be achieved with a bit rate of only 1 to 2 kbps.

[0066] As described above, the audio encoding method according to the embodiment of the present invention may be implemented by various types of electronic devices, such as a terminal, a server, or a combination of both. Therefore, the following description will not be repeated regarding the entity that performs each step. Figure 6A is a first exemplary flowchart of the audio encoding method according to the embodiment of the present invention. As shown in Figure 6A, the audio encoding function is realized by the audio encoding method. The steps shown in Figure 6A will be described below.

[0067] Step 101: Perform a downsampling process on the audio signal to obtain a low frequency signal of the audio signal.

[0068] As an example of acquiring an audio signal, the encoding side responds to an audio capture instruction triggered by the sending side (e.g., the initiator, organizer, or initiator of a web conference, a voice call, etc.) by calling a microphone provided in the encoding side's terminal device to capture an audio signal, and acquires an audio signal x(n) (also referred to as an input signal).

[0069] After the audio signal is acquired, the audio signal is downsampled to extract low-frequency signals from the audio signal. Since low-frequency signals have a greater effect on audio coding than high-frequency signals, signal processing can be performed that distinguishes between low-frequency signals and high-frequency signals.

[0070] Here, the audio signal includes a low-frequency portion and a high-frequency portion. The low-frequency portion is a low-frequency portion of an audio signal separated by a filter from an audio signal with a specific sampling rate based on the characteristics of the audio signal. The high-frequency portion is a high-frequency portion of an audio signal separated from an audio signal with a specific sampling rate. For example, if the effective bandwidth of the audio signal x(n) is 0-16 kHz, the effective bandwidth of the low-frequency signal is 0-8 kHz, and the effective bandwidth of the high-frequency signal x(n) is 0-16 kHz. HB The effective bandwidth of (n) may be 6 to 16 kHz. Note that the embodiments of the present invention are not limited to dividing the frequency band of the audio signal, and may divide the audio signal into average or non-average bands, for example, to obtain uniform low-frequency and high-frequency signals.

[0071] The audio signal is a discrete digital signal and includes a plurality of first sampling points obtained by sampling, where the first sampling points are sampling values ​​sampled from a continuous analog signal.

[0072] In some embodiments, the audio signal includes a plurality of first sampling points obtained by sampling. Step 101 may be realized by the following steps: performing a downsampling process on the first sampling points included in the audio signal using a downsampling filter to obtain a low-frequency signal of the audio signal.

[0073] In the field of digital signal processing, a downsampling filter is used to reduce the sampling rate of an audio signal in order to reduce the amount of data, reduce the system complexity, or meet specific application requirements, where the downsampling factor of the downsampling filter may be a multiple of 2, such as 2, 4, 8, etc.

[0074] Here, the downsampling filter performs the following processing: filtering a first sampling point included in the audio signal based on a digital signal to obtain a filtered audio signal, and downsampling the filtered audio signal based on the digital signal to obtain a low-frequency component of the audio signal. Here, the filtering based on the digital signal may be low-pass filtering or band-pass filtering.

[0075] For example, in the field of digital signal processing, an audio signal is first filtered by a downsampling filter to remove high-frequency components and aliasing interference from the audio signal so that the downsampled audio signal does not lose necessary information, and then the downsampling filter maintains the sampling points of the filtered audio signal at a certain interval to reduce the sampling rate of the audio signal.

[0076] Taking an example where an audio signal x(n) contains 640 sampling points, the downsampling filter extracts the low frequency signal x(n) from the audio signal x(n). LB (n) is extracted. The downsampling filter uses a coefficient of 1 / 2. The effective bandwidth of the low-frequency signal is 0 to 8 kHz, and the number of sampling points per frame is 320.

[0077] Step 102: Perform a low-frequency feature extraction process on the audio signal to obtain the low-frequency features of the audio signal.

[0078] Here, since low-frequency signals have a greater impact on audio coding than high-frequency signals, a low-frequency feature extraction process may be performed on the low-frequency signals using the first neural network model to obtain low-frequency features. This allows the feature dimensions of the low-frequency features to be reduced as much as possible while ensuring the integrity of the low-frequency features, thereby improving the effectiveness of audio coding. Here, the low-frequency features are features that represent the low-frequency signals, and the feature dimensions of the low-frequency features are smaller than the feature dimensions of the low-frequency signals.

[0079] It should be noted that the embodiments of the present invention are not limited to the model structure of the first neural network, and for example, the first neural network may be a deep neural network, a convolutional neural network, or the like.

[0080] 6B is a second exemplary flowchart of an audio encoding method according to an embodiment of the present invention. As shown in FIG. 6B, step 102 of FIG. 6A may be implemented by steps 1021 to 1024. In step 1021, a convolution process is performed on the low-frequency signal to obtain convolution features of the low-frequency signal. In step 1022, a pooling process is performed on the convolution features to obtain pooled features of the low-frequency signal. In step 1023, a downsampling process is performed on the pooled features to obtain downsampling features of the low-frequency signal. In step 1024, a convolution process is performed on the downsampling features to obtain low-frequency features of the audio signal.

[0081] In the field of audio encoding / decoding, neural network (NN) operations such as convolution, pooling, and downsampling play an important role in processing audio signals and extracting audio signal features. In audio encoding / decoding, convolution operations can be used to extract local features of an audio signal. By applying a convolution kernel (a learnable filter), a convolution operation can be performed across the time dimension of an audio signal to obtain signal patterns and resonances. Convolution extracts time- and frequency-domain features of an audio signal and is used for tasks such as noise reduction, feature extraction, and signal separation. Pooling operations are used to reduce the time dimension of an audio signal and reduce data complexity and computational effort. Pooling operations sample local regions of the input signal and aggregate information such as the maximum and average values ​​of the regions to generate a more compact feature representation. In audio signals, pooling operations can improve the robustness and generalization ability of a network and reduce the risk of overfitting. Downsampling operations reduce the sampling rate of an audio signal, i.e., reduce the frequency of the signal. The downsampling operation can reduce the amount of data, save storage and transmission overhead, and reduce complexity. Downsampling may be performed in the frequency domain or the time domain to reduce the sampling rate or reduce the dimensionality of a signal. In the field of audio encoding / decoding, operations such as convolution, pooling, and downsampling can achieve tasks such as feature extraction, encoding, and decoding of audio signals by constructing an appropriate neural network structure. These operations can improve the efficiency and quality of audio signal processing and expand the application range of audio encoding / decoding technology in fields such as audio processing, speech recognition, and music generation.

[0082] As shown in Figure 10, the low frequency signal x LB (n) and generate a lower-dimensional feature vector F LB(n), i.e., low-frequency features are generated. First, in step 1021, the input low-frequency signal x is subjected to causal convolution 1001. LB A convolution process is performed on (n) to obtain 16×320 convolution features. Next, in step 1022, a pooling process (i.e., preprocessing 1002) with a coefficient of 2 is performed on the 24×320 convolution features to obtain 16×160 pooled features. Next, in step 1023, a downsampling process is performed on the 24×160 pooled features by the coding block to obtain 256×1 downsampled features. Finally, in step 1024, a convolution process is performed on the 256×1 downsampled features by causal convolution 1004 to obtain a 56-dimensional feature vector F LB Get (n).

[0083] Here, since the downsampling process in step 1023 is realized by a plurality of cascaded coding layers, step 1023 may be realized by the following steps: a first coding layer of the plurality of cascaded coding layers performs a downsampling process on the pooling feature, and outputs the downsampling result of the first coding layer to a subsequent cascaded coding layer, and the downsampling process and output of the downsampling result are continued by the subsequent cascaded coding layer until the downsampling result is output to the last coding layer, and the downsampling result output by the last coding layer is determined as the downsampling feature of the low-frequency signal.

[0084] As shown in Figure 10, four coding blocks (coding layers) with different downsampling coefficients (Down_factor) are cascaded to perform downsampling on the pooled features. That is, first, the coding block 1003-1 with a downsampling coefficient (Down_factor = 2) performs downsampling on the 16x160 pooled features to obtain a 32x80 downsampling result, then the coding block 1003-2 with a downsampling coefficient (Down_factor = 4) performs downsampling on the 32x80 downsampling result to obtain a 64x20 downsampling result, then the coding block 1003-3 with a downsampling coefficient (Down_factor = 4) performs downsampling on the 64x20 downsampling result to obtain a 128x5 downsampling result, and finally, the coding block 1003-4 with a downsampling coefficient (Down_factor = 5) performs downsampling on the 128x5 downsampling result to obtain a 256x1 downsampling feature. Taking a coding block (Down_factor=4) as an example, to achieve the downsampling effect, one or more dilation convolutions may be performed first, and then pooling may be performed based on the Down_factor.

[0085] Furthermore, each coding layer process provides a deeper understanding of the downsampling characteristics, and the learning of multiple coding layers allows the downsampling characteristics of low-frequency signals to be learned more and more accurately. Cascaded coding layers can obtain the downsampling characteristics of low-frequency signals with gradually increasing accuracy.

[0086] Step 103: Perform high frequency analysis processing on the audio signal to obtain high frequency features of the audio signal.

[0087] Here, the feature dimension of the high-frequency features is lower than that of the low-frequency features. For example, the feature dimension of the low-frequency features is 56, and the feature dimension of the high-frequency features is 8. Because low-frequency signals have a greater impact on audio coding than high-frequency signals, differentiated signal processing is used to extract low-frequency features and high-frequency features of the audio signal so that the feature dimension of the high-frequency features is lower than that of the low-frequency features. Here, high-frequency analysis processing is used to reduce the dimension of the high-frequency signals in the audio signal and achieve the function of data compression.

[0088] 6C is a third exemplary flowchart of an audio encoding method according to an embodiment of the present invention. As shown in FIG. 6C, step 103 in FIG. 6A may be realized by step 1031. In step 1031, a frequency band extension process is performed on the audio signal to obtain high-frequency features of the audio signal.

[0089] Here, in step 1031, high frequency signals in the audio signal are compressed at high speed and high frequency features are extracted.

[0090] 6D is a fourth exemplary flowchart of the audio encoding method according to an embodiment of the present invention. As shown in FIG. 6D, step 1031 in FIG. 6C may be realized by steps 10311 to 10314.

[0091] Step 10311: Perform a frequency domain transform process on a plurality of second sampling points included in the audio signal to obtain transform coefficients corresponding to the plurality of second sampling points, respectively.

[0092] The frequency domain transformation method (i.e., the time-frequency transformation method) in step 10311 may be a modified discrete cosine transform (MDCT), a discrete cosine transform (DCT), a fast Fourier transform (FFT), etc., and the embodiment of the present invention is not limited to the form of frequency domain transformation.

[0093] In some embodiments, step 10311 may be implemented as follows: obtain a plurality of third sampling points included in a reference audio signal, where the reference audio signal is an audio signal adjacent to the audio signal; and perform a discrete cosine transform process on the plurality of second sampling points included in the audio signal based on the plurality of third sampling points included in the reference audio signal and the plurality of second sampling points included in the audio signal, to obtain transform coefficients corresponding to each of the plurality of second sampling points.

[0094] As an example of step 10311, for an audio signal x(n) including 640 second sampling points, a modified discrete cosine transform (MDCT) method is invoked to generate MDCT coefficients of the 640 second sampling points (i.e., transform coefficients corresponding to each of the second sampling points). For example, if the modified discrete cosine transform method has a 50% overlap, the audio signal of the (n+1)th frame (i.e., reference audio signal) and the audio signal of the nth frame (i.e., audio signal) may be merged (concatenated) to calculate an MDCT of 1280 sampling points, and the MDCT coefficients of the first 640 valid sampling points may be used as the transform coefficients corresponding to each of the second sampling points.

[0095] Step 10312: Divide high-frequency transform coefficients among the transform coefficients corresponding to the plurality of second sampling points into a plurality of sub-bands.

[0096] Continuing to refer to the example of step 10311 above, step 10312 specifically includes the following steps: The last 320 MDCT coefficients of the 640 second sampling points (i.e., the transform coefficients corresponding to the second sampling points) represent high-frequency signals, i.e., high-frequency transform coefficients; the high-frequency transform coefficients are divided into N subbands, where a subband is a set of adjacent MDCT coefficients; the 320 MDCT coefficients of the second sampling points are divided into 8 subbands; for example, the 320 MDCT coefficients of the second sampling points may be evenly distributed, i.e., the number of sampling points included in each subband may be the same, in which case each subband includes 40 MDCT coefficients of the second sampling points. However, the embodiment of the present invention is not limited thereto, and the 320 second sampling points may be allocated non-uniformly, for example, such that the low frequency sub-bands contain fewer MDCT coefficients (higher frequency resolution) and the high frequency sub-bands contain more MDCT coefficients (lower frequency resolution).

[0097] According to the Nyquist sampling law (to recover the original signal from the sampled signal without distortion, the sampling frequency must be greater than twice the highest frequency of the original signal. If the sampling frequency is less than twice the highest frequency of the frequency spectrum, aliasing will occur in the signal's frequency spectrum; if the sampling frequency is greater than twice the highest frequency of the frequency spectrum, aliasing will not occur in the signal's frequency spectrum), the MDCT coefficients of the 320 second sampling points (abbreviated as points) represent a frequency spectrum from 8 to 16 kHz. However, ultra-wideband voice communication does not necessarily require a frequency spectrum up to 16 kHz. For example, if the frequency spectrum is set to 14 kHz, only the MDCT coefficients of the first 240 second sampling points need to be considered, and the number of subbands may be controlled to six accordingly.

[0098] Step 10313: Average the transform coefficients included in each subband to obtain an average energy corresponding to each subband, and determine the average energy as the subband spectral envelope corresponding to each subband.

[0099] The averaging process in step 10313 may be arithmetic averaging or geometric averaging, and the embodiment of the present invention is not limited to the mode of averaging process.

[0100] Taking geometric averaging as an example, step 10313 may be implemented as follows: determine the sum of squares of the transform coefficients corresponding to the second sampling points included in each sub-band, and determine the ratio of the sum of squares to the number of second sampling points included in the sub-band as the average energy corresponding to each sub-band.

[0101] Continuing to refer to the example of step 10312 above, step 10313 specifically includes the following steps: For each subband, calculate the average energy of all MDCT coefficients included in the current subband as a subband spectral envelope (the subband spectral envelope is a smooth curve passing through each main peak point of the frequency spectrum). For example, if the MDCT coefficients included in the current subband are x(n), n=1, 2, ..., 40, calculate the average energy Y=(x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40).

[0102] Step 10314: Determine a subband spectral envelope corresponding to each of the plurality of subbands as a high-frequency feature of the audio signal.

[0103] Continuing to refer to the example of step 10313 above, step 10314 specifically includes the following steps: When the 320-point MDCT coefficients are divided into eight sub-bands, eight sub-band spectral envelopes may be obtained, which are used to extract the high-frequency features F of the audio signal. HB(n), i.e., the high frequency feature vector.

[0104] Step 104: Perform an encoding process on the low-frequency features to obtain a low-frequency bitstream of the audio signal, and perform an encoding process on the high-frequency features to obtain a high-frequency bitstream of the audio signal.

[0105] Here, in the field of digital signal processing, a digital signal-based encoding process is performed on low-frequency features to obtain a low-frequency bitstream of an audio signal, and a digital signal-based encoding process is performed on high-frequency features to obtain a high-frequency bitstream of an audio signal.

[0106] In some embodiments, step 104 may be implemented as follows: performing a quantization operation on the low-frequency features to obtain index values ​​of the low-frequency features, performing an entropy coding operation on the index values ​​of the low-frequency features to obtain a low-frequency bitstream of the audio signal, performing a quantization operation on the high-frequency features to obtain index values ​​of the high-frequency features, and performing an entropy coding operation on the index values ​​of the high-frequency features to obtain a high-frequency bitstream of the audio signal.

[0107] As an example of step 104, the low frequency feature F LB (n) and high-frequency features F HB For both (n), a method of scalar quantization (individual quantization of each component) and entropy coding may be adopted. However, the embodiment of the present invention is not limited to this, and a technical combination of vector quantization (joint quantization of adjacent components by combining them into one vector) and entropy coding may also be adopted, and the high-frequency bit stream and the low-frequency bit stream obtained by encoding are merged and transmitted to the decoding side, where they are decoded.

[0108] In view of the above, an audio coding method according to an embodiment of the present invention obtains a low-frequency signal by downsampling an audio signal. Since the low-frequency signal has a greater impact on audio coding than the high-frequency signal in the audio signal, differentiated signal processing is performed to extract low-frequency features and high-frequency features from the audio signal, so that the feature dimension of the high-frequency feature is lower than that of the low-frequency feature, and the low-frequency features and high-frequency features with reduced feature dimension are coded, thereby ensuring audio quality and improving audio coding efficiency.

[0109] As described above, the audio decoding method according to the embodiment of the present invention may be implemented by various types of electronic devices. Figure 7A is a first exemplary flowchart of the audio decoding method according to the embodiment of the present invention. As shown in Figure 7A, the audio decoding method realizes an audio encoding function. The following describes the steps shown in Figure 7A.

[0110] Step 201: Perform a decoding process on a low-frequency bitstream of an audio signal to obtain low-frequency features corresponding to the low-frequency bitstream, and perform a decoding process on a high-frequency bitstream of the audio signal to obtain high-frequency features corresponding to the high-frequency bitstream.

[0111] Here, the low-frequency bitstream is obtained by encoding the low-frequency signal obtained by downsampling the audio signal, and the feature dimension of the high-frequency features is lower than the feature dimension of the low-frequency features.

[0112] For example, after obtaining a high-frequency bitstream and a low-frequency bitstream by encoding using the audio encoding method shown in Figure 6A, the high-frequency bitstream and the low-frequency bitstream obtained by encoding are merged and transmitted to a decoding side. Upon receiving the high-frequency bitstream and the low-frequency bitstream, the decoding side decodes the low-frequency bitstream to obtain low-frequency features corresponding to the low-frequency bitstream, and decodes the high-frequency bitstream to obtain high-frequency features corresponding to the high-frequency bitstream.

[0113] Here, the decoding in step 201 is the reverse process of the encoding in step 104. In the field of digital signal processing, a low-frequency bitstream of an audio signal is subjected to a digital signal-based decoding process to obtain low-frequency features corresponding to the low-frequency bitstream, and a high-frequency bitstream of the audio signal is subjected to a digital signal-based decoding process to obtain high-frequency features corresponding to the high-frequency bitstream.

[0114] In some embodiments, step 201 may be implemented as follows: perform entropy decoding on the low-frequency bitstream to obtain index values ​​corresponding to the low-frequency bitstream; perform inverse quantization on the index values ​​corresponding to the low-frequency bitstream to obtain low-frequency features corresponding to the low-frequency bitstream; perform entropy decoding on the high-frequency bitstream to obtain index values ​​corresponding to the high-frequency bitstream; and perform inverse quantization on the index values ​​corresponding to the high-frequency bitstream to obtain high-frequency features corresponding to the high-frequency bitstream, where the inverse quantization is implemented by consulting a quantization table, which is a mapping table generated by quantization in the encoding process.

[0115] For example, entropy decoding is performed on the received bitstreams (high-frequency bitstream and low-frequency bitstream), an index value corresponding to the bitstream is obtained, and the index value corresponding to the codestream is referenced in the quantization table. The referenced feature vector is taken as the feature vector corresponding to the bitstream, i.e., the estimated value F' of the low-frequency feature vector is obtained. LB (n) (i.e., low-frequency features corresponding to the low-frequency bitstream) and the estimate of the high-frequency feature vector F' HB (n) (i.e., high-frequency features corresponding to the high-frequency bitstream). Note that the decoding process on the decoding side for the received bitstream is the reverse process of the encoding process on the encoding side, so the values ​​generated in the decoding process are estimates for the values ​​in the encoding process, and for example, the high-frequency features generated in the decoding process are estimates for the high-frequency features in the encoding process.

[0116] Step 202: Perform low-frequency feature reconstruction processing on the low-frequency features to obtain low-frequency signals corresponding to the low-frequency features.

[0117] For example, the low-frequency feature reconstruction in step 202 is the inverse process of the low-frequency feature extraction in step 102, in which a low-frequency feature reconstruction process is performed on the low-frequency features using a second neural network model to obtain low-frequency signals corresponding to the low-frequency features (estimated values ​​corresponding to the low-frequency signals obtained in the encoding process).

[0118] It should be noted that the embodiments of the present invention are not limited to the model structure of the second neural network, and for example, the second neural network may be a deep neural network, a convolutional neural network, or the like.

[0119] 7B is a second exemplary flowchart of an audio decoding method according to an embodiment of the present invention. As shown in FIG. 7B, step 202 in FIG. 7A may be implemented by steps 2021 to 2024. In step 2021, a convolution process is performed on the low-frequency features to obtain convolutional features of the low-frequency features. In step 2022, an upsampling process is performed on the convolutional features to obtain upsampling features of the low-frequency features. In step 2023, a pooling process is performed on the upsampling features to obtain pooled features of the low-frequency features. In step 2024, a convolution process is performed on the pooled features to obtain a low-frequency signal corresponding to the low-frequency features.

[0120] In the field of audio coding / decoding, upsampling operations are used to increase the resolution of feature maps to more accurately reconstruct audio signals. Upsampling involves interpolation or other forms of upsampling techniques to generate more accurate feature maps, which can better restore the original details and characteristics of the audio signal during the decoding process. By using neural network techniques such as convolution, pooling, and upsampling in audio decoding, useful features can be effectively extracted, computational complexity can be reduced, and the original content of the audio signal can be more accurately reconstructed. These techniques can improve the performance and efficiency of audio decoding and promote the development and application of audio coding / decoding technology.

[0121] As shown in Figure 11, the low-frequency feature F' LB Based on (n), the neural network model shown in FIG. 11 is called to generate the low-frequency signal x' LB(n) is generated. Here, the neural network model shown in FIG. 11 is similar to the neural network model shown in FIG. 10, for example, causal convolution, and the configuration of the post-processing is similar to that of the pre-processing. The configuration of the decoding block is symmetrical to the coding block on the coding side. The coding block on the coding side first performs dilation convolution and then pooling to complete downsampling, while the decoding block on the decoding side first performs pooling to complete upsampling and then performs dilation convolution.

[0122] First, in step 2021, the input low-frequency feature F' is convolved by the causal convolution 1101. LB (n) to obtain 256×1 convolution features. Next, in step 2022, an upsampling process is performed on the 256×1 convolution features to obtain 16×160 upsampled features. Next, in step 2023, a pooling process (i.e., post-processing 1103) is performed on the 16×160 undampled features to obtain 16×320 pooled features. Next, in step 2024, a convolution process is performed on the 16×320 pooled features using causal convolution 1104 to obtain a 320-dimensional low-frequency signal x' LB Get (n).

[0123] Here, since the unsampling process in step 2022 is realized by a plurality of cascaded decoding layers, step 2022 may be realized by the following steps: a first decoding layer among a plurality of cascaded decoding layers performs an upsampling process on the convolutional features, and the upsampling result of the first decoding layer is output to a subsequent cascaded decoding layer, and the subsequent cascaded decoding layers continue to perform the upsampling process and output the upsampling result until it is output to the last decoding layer, and the upsampling result output by the last decoding layer is determined as the upsampling feature of the low-frequency features.

[0124] As an example of step 2022, as shown in FIG. 11, four decoding blocks (decoding layers) with different upsampling coefficients (Up_factor) are cascaded to perform upsampling on convolutional features. That is, first, a decoding block 1102-1 with an upsampling coefficient (Up_factor=5) performs upsampling on the 256×1 convolutional features to obtain a 128×5 upsampling result. Next, a decoding block 1102-2 with an upsampling coefficient (Up_factor=4) performs upsampling on the 128×5 upsampling result to obtain a 64×20 upsampling result. Next, a decoding block 1102-3 with an upsampling coefficient (Up_factor=4) performs upsampling on the 64×20 upsampling result to obtain a 32×80 upsampling result. Finally, a decoding block 1102-4 with an upsampling coefficient (Up_factor=2) performs upsampling on the 32×80 upsampling result to obtain a 16×160 upsampling feature. Take the decoding block (Up_factor=4) as an example, to achieve the effect of upsampling, pooling is performed based on the Up_factor, and then one or more dilation convolutions are performed.

[0125] Furthermore, each processing step of the decoding layer provides a deeper understanding of the upsampling features, and the learning of multiple decoding layers can gradually improve the accuracy of the upsampling features of the low-frequency features. Cascaded decoding layers can obtain the upsampling features of the low-frequency features with gradually improved accuracy.

[0126] Step 203: Perform an upsampling process on the low-frequency signal to obtain an upsampling signal of the low-frequency signal.

[0127] For example, when the sampling rate on the encoding side is 1 / 2, the low-frequency signal x' obtained on the decoding side is LBThe sampling rate of the low-frequency signal x'(n) is only half that of the audio signal x(n) at the encoding side. LB (n) and double the sampling rate to obtain the upsampled signal x up (n) to ensure that the sampling rate of the upsampled signal is the same as the sampling rate of the audio signal.

[0128] In some embodiments, step 203 may be implemented as follows: an upsampling filter performs an upsampling process on the fourth sampling point included in the low-frequency signal to obtain an upsampling signal of the low-frequency signal;

[0129] In the field of digital signal processing, an upsampling filter is used to increase the sampling rate of a signal input to the upsampling filter in order to better capture detail information in the signal or extend the frequency spectrum range, where the upsampling factor of the upsampling filter may be a multiple of 2, such as 2, 4, 8, etc.

[0130] Here, the upsampling filter performs the following processing: upsampling processing based on a digital signal is performed on the fourth sampling point included in the low-frequency signal to obtain the up-sampled low-frequency signal, and filtering processing based on the digital signal is performed on the up-sampled low-frequency signal to obtain an up-sampled signal of the low-frequency signal. Here, the filtering processing based on the digital signal may be low-pass filtering or band-pass filtering.

[0131] For example, in the field of digital signal processing, an upsampling filter performs a digital signal-based upsampling process on every fourth sampling point in a low-frequency signal. Here, upsampling refers to inserting zero values ​​into the low-frequency signal to increase the sampling rate. To remove aliasing interference caused by interpolation, a digital signal-based filtering process is performed on the upsampled low-frequency signal.

[0132] low frequency signal x' LB Taking (n) as an example, where (n) contains 320 sampling points, the upsampling filter can be used to filter the low frequency signal x' LB The upsampling filter is used to double the sampling rate of the low frequency signal x' (n). LB The sampling rate of (n) is doubled to obtain the upsampled signal x up (n), where x for each frame up (n) contains 640 sampling points (abbreviated as points).

[0133] Step 204: Perform a signal reconstruction process on the high-frequency features and the up-sampled signal to obtain a synthesized audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

[0134] In signal reconstruction, extension is performed based on the high-frequency features and the upsampled signal band to obtain a high-frequency signal, and merging is performed based on the upsampled signal and the reconstructed high-frequency signal to obtain a merged audio signal x'(n).

[0135] 7C is a third exemplary flowchart of the audio decoding method according to an embodiment of the present invention. As shown in FIG. 7C, step 204 in FIG.

[0136] Step 2041: Perform a frequency domain transform process on a plurality of fifth sampling points included in the up-sampled signal to obtain transform coefficients corresponding to each of the plurality of fifth sampling points.

[0137] The frequency domain transformation method (i.e., the time-frequency transformation method) in step 2041 may be a modified discrete cosine transform (MDCT), a discrete cosine transform (DCT), a fast Fourier transform (FFT), or the like, and the embodiment of the present invention is not limited to the form of frequency domain transformation.

[0138] In some embodiments, step 2041 may be implemented as follows: obtain a plurality of sampling points included in a reference upsampling signal, where the reference upsampling signal is an upsampling signal adjacent to the upsampling signal; and, based on referring to the plurality of sampling points included in the upsampling signal and a plurality of fifth sampling points included in the upsampling signal, perform a discrete cosine transform process on the plurality of fifth sampling points included in the upsampling signal to obtain transform coefficients corresponding to each of the plurality of fifth sampling points.

[0139] As an example of step 2041, an upsampled signal x containing 640 sampling points is upA modified discrete cosine transform (MDCT) method is invoked for (n) to generate MDCT coefficients of 640 sampling points (i.e., transform coefficients corresponding to a plurality of fifth sampling points). Specifically, when the modified discrete cosine transform method has 50% overlap, the upsampling signal of the n+1th frame (i.e., the reference upsampling signal) and the upsampling signal of the nth frame (i.e., the upsampling signal) are merged (concatenated) to calculate an MDCT of 1280 sampling points (including a plurality of sampling points included in the reference upsampling signal and a plurality of fifth sampling points included in the upsampling signal). The calculated MDCT of 1280 sampling points is an upsampling signal format, and high-frequency content is missing. Therefore, only the low-frequency content of the calculated MDCT of 1280 sampling points is retained. That is, the MDCT coefficients of the effective top 640 sampling points are used to generate transform coefficients X corresponding to a plurality of fifth sampling points. up (n).

[0140] Step 2042: Perform frequency band extension processing on the transform coefficients and high-frequency features corresponding to each of the plurality of fifth sampling points to obtain high-frequency transform coefficients of the high-frequency signal.

[0141] Here, a frequency band extension technique is used to reconstruct high-frequency transform coefficients of the high-frequency signal based on the transform coefficients respectively corresponding to the plurality of fifth sampling points and the high-frequency features.

[0142] 7D is a fourth exemplary flowchart of the audio decoding method according to an embodiment of the present invention. As shown in FIG. 7D, step 2042 in FIG. 7C may be realized by the following steps 20421 to 20422.

[0143] Step 20421: Perform a frequency spectrum replication process on at least some of the transform coefficients in the first half of the transform coefficients corresponding to each of the plurality of fifth sampling points to obtain reference high-frequency transform coefficients of the reference high-frequency signal.

[0144] In some embodiments, step 20421 may be implemented as follows: perform a frequency spectrum replication process on the transform coefficients of the latter part of the first part of the transform coefficients corresponding to each of a plurality of fifth sampling points to obtain reference high-frequency transform coefficients of the reference high-frequency signal.

[0145] As an example of step 20421, the transform coefficients corresponding to each of the plurality of fifth sampling points (including MDCT coefficients of 640 points) are up (n) and X up The low-frequency MDCT coefficients in (n) (i.e., the first 320 MDCT coefficients) are replicated to generate reference values ​​of the MDCT coefficients in the high-frequency part, i.e., the reference high-frequency transform coefficients of the reference high-frequency signal. The basic feature of the reference audio signal is that there are many harmonics in the low-frequency part and few harmonics in the high-frequency part. Therefore, in order to avoid excessive harmonics being included in the generated high-frequency MDCT spectrum due to simple replication, X up The high-frequency part of the frequency spectrum may be duplicated twice based on the last 160 points of the low-frequency MDCT coefficients in (n) (i.e., the transform coefficients in the latter half of the first half of the transform coefficients corresponding to the plurality of fifth sampling points, respectively) to generate reference values ​​of the MDCT coefficients of the 320 points of the high-frequency part signal (i.e., the reference high-frequency transform coefficients of the reference high-frequency signal). Therefore, X after frequency spectrum duplication up The last 320 points in (n) are non-zero coefficients.

[0146] In addition, the embodiment of the present invention is X up (n) is not limited to being based on the last 160 points of the low frequency MDCT coefficients in X upAlternatively, the frequency spectrum may be replicated once based on the low-frequency MDCT coefficients in (n) to generate reference values ​​of the MDCT coefficients of the 320 points of the high-frequency partial signal. up Using the last 80 points of the low-frequency MDCT coefficients in (n) as a base, the frequency spectrum may be replicated four times to generate reference values ​​of the 320 points of the MDCT coefficients of the high-frequency partial signal.

[0147] Step 20422: According to the sub-band spectral envelope corresponding to the high-frequency feature, perform gain processing on the reference high-frequency transform coefficients of the reference high-frequency signal to obtain the high-frequency transform coefficients of the high-frequency signal.

[0148] In some embodiments, step 20422 may be implemented as follows: Divide reference high-frequency transform coefficients of a reference high-frequency signal into a plurality of sub-bands based on sub-band spectral envelopes corresponding to high-frequency features; For each sub-band among the plurality of sub-bands, determine a first average energy of the high-frequency sub-band corresponding to the sub-band in the sub-band spectral envelope, determine a second average energy of the sub-band, determine a gain factor based on a ratio between the first average energy and the second average energy, multiply each reference high-frequency transform coefficient included in the sub-band by the gain factor to obtain a high-frequency transform coefficient corresponding to the sub-band; and determine the high-frequency transform coefficients corresponding to each of the plurality of sub-bands as high-frequency transform coefficients of the high-frequency signal.

[0149] Continuing to refer to the example of step 20421 above, step 20422 specifically includes the following steps: The sub-band spectral envelopes (e.g., eight sub-band spectral envelopes) corresponding to the high-frequency features are decoded to obtain an estimated value F' corresponding to the high-frequency features obtained by decoding. HB(n), and the eight subband spectral envelopes correspond to eight high-frequency subbands. The reference values ​​of 320 points of MDCT coefficients of the generated high-frequency partial signal (i.e., the reference high-frequency transform coefficients of the reference high-frequency signal) are divided into eight subbands, and the reference values ​​of 320 points of MDCT coefficients of the high-frequency partial signal are gain-adjusted (multiplication is performed in the frequency domain) based on one high-frequency subband and the corresponding subband. For example, a gain coefficient is calculated based on the average energy of the high-frequency subband and the average energy of the corresponding subband, and the MDCT coefficients corresponding to each point in the corresponding subband are multiplied by the gain coefficient, ensuring that the energy of the high-frequency MDCT coefficients generated by decoding is close to the original coefficient energy of the encoding side.

[0150] For example, suppose the average energy of a subband (i.e., a subband divided by the reference value of the 320-point MDCT coefficient of the generated high-frequency partial signal) is Y_L, and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectral envelope decoded based on the bitstream) is Y_H. Then, calculate the gain coefficient a=sqrt(Y_H / Y_L). Once the gain coefficient a is obtained, the MDCT coefficient of each point in the subband is directly multiplied by a. The average energy of the gain-controlled MDCT coefficients (i.e., the high-frequency transform coefficients of the high-frequency signal) is very close to the original average energy on the encoding side, so that the original high-frequency signal can be restored as much as possible.

[0151] Step 2043: Determine a composite audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream based on the high-frequency transform coefficients and the transform coefficients corresponding to each of some fifth sampling points among the plurality of fifth sampling points.

[0152] In some embodiments, step 2043 may be implemented as follows: performing a merging process on the high-frequency transform coefficients and the transform coefficients corresponding to each of the first half fifth sampling points of the plurality of fifth sampling points to obtain full transform coefficients; and performing an inverse transform process of the frequency spectrum transform on the full transform coefficients to obtain a composite audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

[0153] Continuing to refer to the example of step 20422 above, step 2043 specifically includes the following steps: up (n) (i.e., the MDCT coefficients of the first 320 points, i.e., the transform coefficients corresponding to some fifth sampling points among the plurality of fifth sampling points) and the MDCT coefficients after gain control (i.e., X up The reference values ​​of the last 320 MDCT coefficients after gain in (n), i.e., the high-frequency transform coefficients, are merged to obtain the full transform coefficients (i.e., 640 MDCT coefficients including the low-frequency MDCT coefficients and the high-frequency MDCT coefficients), and an inverse MDCT transform (i.e., the inverse transform process of the frequency spectrum transform) is performed on the full transform coefficients to generate a 1280-point MDCT estimate, and the first 640 valid MDCT estimates are used as the synthesized audio signal through overlapping.

[0154] From the above, an audio decoding method according to an embodiment of the present invention decodes a received bitstream (i.e., a low-frequency bitstream and a high-frequency bitstream), obtains low-frequency features and high-frequency features, and invokes the corresponding inverse processing of the encoding side on the low-frequency feature vector to reconstruct the low-frequency part. Then, the reconstructed low-frequency part (low-frequency signal) is up-sampled and filtered to restore the same sampling rate as the original input audio signal. The high-frequency transform coefficients of the high-frequency signal are reconstructed by merging the low-frequency part after up-sampling filtering with the high-frequency features, and the reconstructed low-frequency part and the high-frequency transform coefficients are merging to ensure that the energy of the reconstructed high-frequency signal is close to the energy of the high-frequency signal of the encoding side. In this way, the frequency domain coefficients from low to high frequencies can be completely reconstructed.

[0155] The following describes an example of applying an embodiment of the present invention to a real application scenario.

[0156] The embodiments of the present invention can be applied to various audio scenarios, such as voice calls, instant messaging, etc. The following will take voice calls as an example.

[0157] In the related art, the principle of speech coding is generally as follows: speech coding can directly code speech waveform samples sample by sample, or according to the principle of human speech production, extract relevant low-dimensional features, encode the features at the encoding side, and reconstruct the speech signal based on these parameters at the decoding side.

[0158] The principles of the above-mentioned speech coding all originate from compression methods based on speech signal modeling, i.e., signal processing. To improve coding efficiency while ensuring speech quality compared to compression methods based on speech processing, an embodiment of the present invention provides a low-rate neural network encoding / decoding method (i.e., speech processing method) that isolates the most important low-frequency portion (i.e., low-frequency signal) of a speech signal at a specific sampling rate through downsampling filtering based on the characteristics of the speech signal. The important low-frequency portion (i.e., low-frequency signal) is processed using neural network (NN) technology to obtain a feature vector with lower dimensions than the input low-frequency signal. The relatively unimportant portion (i.e., high-frequency portion) of the speech signal is encoded using fewer bits. On the encoding side, the low-frequency portion and the high-frequency portion of the speech signal are processed separately to obtain feature vectors with lower dimensions than the input speech signal (low-frequency feature vector and high-frequency feature vector), and the feature vectors are compression-encoded. On the decoding side, the received bitstream (i.e., low-frequency bitstream and high-frequency bitstream) is decoded to obtain low-frequency feature vectors and high-frequency feature vectors, and the low-frequency feature vectors are subjected to the inverse process corresponding to the encoding side to reconstruct the low-frequency part. The reconstructed low-frequency part is then restored to the same sampling rate as the original input audio signal using an upsampling filter. The high-frequency part is reconstructed by combining the low-frequency part and the high-frequency feature vector after the upsampling filter, and the reconstructed low-frequency part and high-frequency part are merged and decoded.

[0159] An embodiment of the present invention can be applied to the voice communication link shown in Figure 8. A Voice over Internet Protocol (VoIP) conferencing system will be used as an example. To solve the basic function of voice compression, the voice encoding / decoding technique according to the embodiment of the present invention is implemented in the encoding and decoding parts. An encoder is implemented in an uplink client 801, and a decoder is implemented in a downlink client 802. The uplink client captures voice, performs pre-processing enhancement, encoding, and other processes, transmits the encoded bitstream to the downlink client 802 via a network, and the downlink client 802 performs decoding, enhancement, and other processes, and plays the decoded voice.

[0160] Considering forward compatibility (i.e., compatibility between the new encoder and the existing encoder), a transcoder needs to be deployed in the background of the system (i.e., the server) to solve the problem of interconnection between the new encoder and the existing encoder. For example, the sender (uplink client) is a new NN encoder, and the receiver (downlink client) is a public switched telephone network (PSTN) (G.722). To realize the transcoding function, an NN decoder needs to be executed in the background to generate an audio signal, and then a G.722 encoder needs to be invoked to generate a specific bitstream so that the receiver can accurately decode it based on the specific bitstream.

[0161] The following describes a low-rate neural network encoding / decoding method (realized by an audio encoding method and an audio decoding method) according to an embodiment of the present invention with reference to FIG.

[0162] The encoding side performs the following process.

[0163] For the input speech signal x(n) of the nth frame, a downsampling filter is used to extract the low-frequency part signal x(n) of the speech signal x(n). LB (n) (abbreviated as low frequency signal).

[0164] Low frequency partial signal x LB For (n), the first NN is called and a low-dimensional feature vector F LB (n) (abbreviated as low frequency features) to obtain the feature vector F LB Since the dimension of (n) is lower than that of the low-frequency subband signal, the amount of data can be reduced. For example, for each frame x LB For (n), we call a dilated convolutional network (Dilated CNN) and generate a lower-dimensional feature vector F LB (n). The embodiment of the present invention is not limited to the NN structure, and may be other NN structures, such as an autoencoder, a fully-connected (FC) network, a long short-term memory (LSTM) network, or a convolutional neural network (CNN) + LSTM.

[0165] Also, the speech signal x(n) is analyzed at high frequencies to generate a high-frequency feature vector F HB In this embodiment, high-frequency features x(n) (abbreviated as high-frequency features) are extracted. Considering that high frequencies are not as important to quality as low frequencies, an embodiment of the present invention may generate a high-frequency signal at only 1-2 kbps by using a frequency band extension technique based on speech signal analysis. Here, an embodiment of high-frequency analysis based on frequency band extension includes: performing a time-frequency transform on the speech signal x(n) to obtain frequency-domain coefficients; merging adjacent coefficients of the frequency-domain coefficients into multiple sub-bands; and calculating the average energy of each sub-band as the spectral envelope of the sub-band. Here, only the frequency-domain coefficients related to high frequencies are processed, and the extracted spectral envelope related to high frequencies is called F. HB It may be extracted as (n).

[0166] The feature vectors corresponding to the sub-bands (i.e., F LB (n) and F HB (n)) is vector-quantized or scalar-quantized, the quantized index values ​​are entropy-coded, and the bit stream obtained by the coding is transmitted to the decoding side.

[0167] The decoding side performs the following process.

[0168] The decoder decodes the received bitstream and estimates the low-frequency feature vector (abbreviated as low-frequency feature) F LB '(n) and the estimate of the high-frequency feature vector (abbreviated as high-frequency feature) F HB '(n) respectively.

[0169] For the low frequency part, estimates of the low frequency feature vector F LB '(n) to call a second NN to generate an estimate of the low frequency signal x LB 'Generate (n).

[0170] The generated low-frequency signal estimate x LB '(n) is upsampled and filtered to x up (n), where x up The sampling rate of (n) matches the sampling rate of the audio signal x(n) input from the encoding side.

[0171] Regarding signal reconstruction, the signal reconstruction mainly includes reconstructing the high-frequency part based on frequency band extension, and merging the low-frequency part signal and the reconstructed high-frequency signal into the final output signal x'(n). First, x up(n) to obtain frequency domain coefficients. As mentioned above, the resolution of the frequency domain coefficients is consistent with the resolution of the input speech signal x(n). Second, since the frequency domain coefficients lack high-frequency components, a band extension technique is used to generate high-frequency components (i.e., frequency domain coefficients of the high-frequency part), and the high-frequency components and the frequency domain coefficients of the low-frequency part are merged to generate the complete frequency domain coefficients. Third, the complete frequency domain coefficients are subjected to an inverse time-frequency transform to obtain the final output signal x'(n). Here, the frequency band extension technique involves replicating the frequency domain coefficients of the low-frequency part to the high-frequency part, and then adding the resulting high-frequency feature vector F HB and energy adjusting the frequency domain coefficients of the high frequency part for each subband based on '(n) to ensure that the energy of the reconstructed high frequency part is close to the high frequency energy of the encoding side. In this way, the frequency domain coefficients from low frequency to high frequency can be perfectly reconstructed.

[0172] The low-rate neural network coding method according to the embodiment of the present invention will be specifically described below.

[0173] In some embodiments, an audio signal with a sampling rate of Fs=32000 Hz (note that the method according to the embodiments of the present invention is also applicable to other sampling rate scenarios, including but not limited to 8000 Hz, 16000 Hz, and 48000 Hz) is taken as an example. Assuming that the frame length is 20 ms, when Fs=32000 Hz, each frame contains 640 sampling points.

[0174] The encoding side and the decoding side will be described in detail below with reference to the flowchart shown in FIG.

[0175] The flow on the encoding side is as follows:

[0176] For an audio signal with a sampling rate of Fs=32000 Hz, the input signal of the nth frame contains 640 sampling points (abbreviated as points), and is denoted as input signal x(n).

[0177] Step 11: Perform downsampling filtering.

[0178] In the embodiment of the present invention, downsampling filtering refers to converting the original input signal x(n) into a low frequency signal x(n). LB (n). In this embodiment of the present invention, when downsampling filtering with a factor of 1 / 2 is adopted, the effective bandwidth of the low-frequency signal is 0-8 kHz, and the number of sampling points per frame is 320.

[0179] Step 12: Low-frequency signal x LB (n) is input to the first NN to perform data compression.

[0180] low frequency signal x LB (n), and then call the first NN based on the lower-dimensional feature vector F LB (n) is generated. Note that x LB The dimension of (n) is 320, and F LB The dimension of (n) is 56, and in terms of the amount of data, the first NN plays the role of "dimensionality reduction" and realizes the function of data compression.

[0181] 10 is a schematic diagram of the first NN. The flow of data compression by the first NN will be specifically described below with reference to FIG.

[0182] First, we call the 16-channel causal convolution 1001 and take the input tensor (i.e., the low-frequency signal x LB (n)) may be expanded to a 16x320 tensor.

[0183] Next, preprocessing 1002 is performed on the 16×320 tensor. For example, a pooling operation with a coefficient of 2 is performed on the 16×320 tensor, and the activation function may be ReLU to generate a 16×160 tensor.

[0184] Next, four coding blocks (i.e., coding block 1003-1, coding block 1003-2, coding block 1003-3, and coding block 1003-4) with different downsampling factors (Down_factor) are cascaded. Taking the coding block (Down_factor=2) as an example, one or more dilation convolutions may be performed first, with each convolution kernel size fixed at 1×3 and a stride rate of 1. The dilation rate of one or more dilation convolutions may be set to, for example, 3 according to actual needs. However, embodiments of the present invention are not limited thereto and may use dilation rates of different dilation convolution settings. Next, the Down_factors of the four coding blocks are set to 2, 4, 4, and 5, respectively. This is equivalent to setting pooling factors of different sizes and plays a role in downsampling. Finally, the number of channels of the four coding blocks is set to 32, 64, 128, and 256, respectively. Therefore, according to the four coding blocks, the 16×160 tensor is converted into 32×80, 64×20, 128×5, and 256×1 tensors in turn. Note that the embodiment of the present invention is not limited to other implementations of the coding blocks, such as the number of dilation convolutions (including the dilation rate), the value of the downsampling coefficient in each coding block, and the number of output channels of each coding block. With the above configuration, it is only necessary to ensure that the input of the first NN is 1×320 and the output is 56×1 dimensional.

[0185] Finally, the 256 × 1 tensor undergoes the same preprocessing causal convolution 1004 to obtain a 56-dimensional feature vector F LB (n) may be output.

[0186] Step 13: Perform high frequency analysis on the input signal x(n).

[0187] The purpose of high-frequency analysis is to analyze the input signal x(n), extract key information about the high-frequency signal, and generate a lower-dimensional feature vector F HB The purpose is to generate (n).

[0188] The embodiment of the present invention proposes a high-frequency analysis of an input signal based on frequency band extension (recovering a wideband speech signal from a band-limited narrowband speech signal). The application of frequency band extension according to the embodiment of the present invention will be specifically described below.

[0189] For an input signal x(n) containing 640 points, a modified discrete cosine transform (MDCT) is invoked to generate 640-point MDCT coefficients. For example, if the modified discrete cosine transform method is 50% overlap, the input signal data of the n+1th frame and the input signal of the nth frame may be merged (concatenated) to calculate 1280-point MDCT and obtain 640-point MDCT coefficients.

[0190] For high-frequency analysis, an embodiment of the present invention represents the last 320 MDCT coefficients of the 640 MDCT coefficients as the high-frequency signal. The 320 MDCT coefficients (i.e., the high-frequency signal) are divided into N subbands, where a subband is a group of adjacent MDCT coefficients, and the 320 MDCT coefficients may be divided into eight subbands. For example, the 320 points may be allocated evenly, i.e., the number of points included in each subband is the same. However, the embodiment of the present invention is not limited thereto, and the 320 points may be allocated non-uniformly, for example, low-frequency subbands include fewer MDCT coefficients (higher frequency resolution) and high-frequency subbands include more MDCT coefficients (lower frequency resolution).

[0191] According to the Nyquist sampling law (to recover the original signal from a sampled signal without distortion, the sampling frequency must be greater than twice the highest frequency of the original signal. If the sampling frequency is less than twice the highest frequency of the spectrum, the signal's frequency spectrum will exhibit aliasing; if the sampling frequency is greater than twice the highest frequency of the spectrum, the signal's frequency spectrum will not exhibit aliasing), the 320 MDCT coefficients described above represent a frequency spectrum from 8 to 16 kHz. However, ultra-wideband voice communication does not necessarily require a frequency spectrum up to 16 kHz. For example, if the frequency spectrum is set to 14 kHz, only the first 240 MDCT coefficients need to be considered, and the number of subbands can be controlled to six.

[0192] For each subband, the average energy of all MDCT coefficients included in the current subband is calculated as the subband spectral envelope (the subband spectral envelope is a smooth curve that passes through each main peak point of the frequency spectrum). For example, if the MDCT coefficients included in the current subband are x(n), n=1, 2, ..., 40, the average energy Y=((x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40). When the 320-point MDCT coefficients are divided into eight sub-bands, eight sub-band spectral envelopes are obtained, which are used to generate the feature vector F of the high-frequency signal. HB (n).

[0193] As a result, the high-frequency analysis unit outputs an eight-dimensional feature vector that represents the important features of the high-frequency portion of the input signal. This allows high-frequency information to be expressed with a small amount of data, significantly improving coding efficiency.

[0194] Step 14: Quantization coding is performed.

[0195] Feature vector F of the low frequency signal LB(n) and the feature vector F of the high-frequency signal HB Regarding (n), both scalar quantization (quantizing each component individually) and entropy coding methods can be implemented. In addition, embodiments of the present invention are not limited thereto, and may also employ a technical combination of vector quantization (combining multiple adjacent components into one vector and performing joint quantization) and entropy coding.

[0196] Alternatively, the feature vector may be quantized and encoded to generate a corresponding bitstream. Experiments have shown that a 32 kHz ultra-wideband signal can be compressed with high quality at a bit rate of 6-10 kbps.

[0197] Therefore, in an embodiment of the present invention, a low-frequency signal is obtained by downsampling an audio signal. Since the low-frequency signal has a greater impact on audio coding than the high-frequency signal in the audio signal, differentiated signal processing is used to extract low-frequency features and high-frequency features of the audio signal, so that the feature dimension of the high-frequency feature is lower than that of the low-frequency feature, and the low-frequency features and high-frequency features with reduced feature dimension are coded, thereby ensuring audio quality and improving audio coding efficiency.

[0198] The following is a detailed description of a low-rate neural network decoding method according to an embodiment of the present invention.

[0199] Corresponding to the above example, the flow on the decoding side is as follows.

[0200] Step 21: Perform quantization decoding.

[0201] Quantization decoding is the reverse process of quantization encoding. First, entropy decoding is performed on the received bitstream to obtain the index value corresponding to the bitstream, and then the quantization table is consulted based on the index value corresponding to the bitstream to obtain the estimated low-frequency feature vector F'. LB(n) and the estimated high-frequency feature vector F' HB Get (n).

[0202] Step 22: Estimate the low frequency feature vector F' LB (n) is input to the second NN.

[0203] Low frequency feature vector estimate F' LB Based on (n), the second NN shown in FIG. 11 is called to obtain the low-frequency signal estimate x' LB Generate (n).

[0204] Here, the second NN is similar to the first NN, such as causal convolution, and the structure of the post-processing in the second NN is similar to the pre-processing in the first NN. The structure of the decoding block is symmetrical to the coding block on the encoding side, where the coding block on the encoding side first performs dilation convolution, then pooling and downsampling, while the decoding block on the decoding side first performs pooling and upsampling, and then dilation convolution.

[0205] Step 23: Estimate of the low-frequency signal x' LB Perform upsampling filtering on (n).

[0206] The generated low-frequency signal estimate x' LB The sampling rate of (n) is only half that of the input signal x(n) at the encoding side. Therefore, the estimated value x' of the low-frequency signal LB (n) is upsampled and filtered to double the sampling rate, resulting in the upsampled signal x up (n), where x for each frame up (n) contains 640 points. The upsampling filtering according to the embodiment of the present invention corresponds to an upsampling operation by a factor of 2.

[0207] Step 24: Upsample the signal and estimate the high frequency feature vector F'HB (n) to perform signal reconstruction.

[0208] The first stage of signal reconstruction is the resulting upsampled signal x up Similarly to the encoding side configuration, a modified discrete cosine transform (MDCT) is performed on (n), i.e., the MDCT of 1280 points is calculated, and the MDCT coefficients of 640 points, i.e., x up (n). For example, an upsampled signal x containing 640 points is obtained. up For (n), we call the Modified Discrete Cosine Transform (MDCT) and compare the upsampled signal of the previous frame with the upsampled signal of the current frame x up (n) and (n) are merged to calculate the MDCT of 1280 points, and the MDCT coefficients of 640 points are obtained. up (n) is an upsampling signal format, so x up (n) retains only the content in the 0 to 8 kHz part, and the content in the 8 to 16 kHz part is missing.

[0209] The second stage of signal reconstruction is the estimation of the resulting high-frequency feature vector F' HB Based on (n), we invoke the frequency band extension technique to generate the missing high frequency components, where F' HB (n) is the sub-band spectral envelope of the eight high frequency parts decoded from the bitstream. The specific operation of the second stage of signal reconstruction is as follows:

[0210] x up The low-frequency MDCT coefficients (the first 320 MDCT coefficients) in (n) are replicated to generate reference values ​​for the MDCT coefficients in the high-frequency part. The basic feature of the reference speech signal is that it contains many harmonics in the low-frequency part and few harmonics in the high-frequency part. Therefore, to avoid excessive harmonics being included in the generated high-frequency MDCT spectrum due to simple replication, X upAlternatively, the frequency spectrum may be duplicated twice based on the last 160 points of the low-frequency MDCT coefficients in (n) to generate reference values ​​of 320 points of MDCT coefficients of the high-frequency partial signal (abbreviated as reference values ​​of 320 points of high-frequency MDCT coefficients of the reference high-frequency signal). up Alternatively, the frequency spectrum may be replicated once based on the low-frequency MDCT coefficients in (n) to generate reference values ​​of the MDCT coefficients of the 320 points of the high-frequency partial signal. up Based on the last 80 points of the low-frequency MDCT coefficients in (n), the frequency spectrum may be replicated four times to generate reference values ​​of the 320 points of the MDCT coefficients of the high-frequency partial signal. up The last 320 points in (n) are non-zero coefficients.

[0211] Next, the eight subband spectral envelopes corresponding to the eight high-frequency subbands previously obtained (i.e., the estimated high-frequency feature vector F' obtained by consulting the quantization table) are calculated. HB (n)), and the eight subband spectral envelopes correspond to eight high-frequency subbands. The reference values ​​of the 320 points of the MDCT coefficients of the generated high-frequency partial signal are divided into eight reference high-frequency subbands, and the reference values ​​of the 320 points of the MDCT coefficients of the high-frequency partial signal are gain-controlled (multiplication is performed in the frequency domain) based on one high-frequency subband and the corresponding reference high-frequency subband. For example, a gain coefficient is calculated based on the average energy of the high-frequency subband and the average energy of the corresponding reference high-frequency subband, and the MDCT coefficients at each point in the corresponding reference high-frequency subband are multiplied by the gain coefficient, ensuring that the energy of the virtually generated high-frequency MDCT coefficients in decoding is close to the original coefficient energy in encoding.

[0212] For example, assuming that the average energy of the reference high-frequency subband (i.e., the subband divided by the reference values ​​of the 320-point MDCT coefficients of the generated high-frequency partial signal) is Y_L and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectral envelope decoded based on the bitstream) is Y_H, calculate the gain coefficient a=sqrt(Y_H / Y_L). Here, sqrt() represents the square root calculation function for calculating the square root of (Y_H / Y_L). Once the gain coefficient a is obtained, the MDCT coefficient of each point in the reference high-frequency subband is directly multiplied by a. The (virtually generated) average energy of the MDCT coefficients after gain control is very close to the original average energy on the encoding side.

[0213] Finally, X up The low-frequency MDCT coefficients in (n) (i.e., the MDCT coefficients of the first 320 points) and the MDCT coefficients after gain control (i.e., X up The inverse MDCT transform is performed on the last 320 MDCT coefficients after gain in (n)) to generate a 1280-point MDCT estimate, and by overlapping, the first valid 640-point MDCT estimate is taken as an estimate of the original input signal, i.e., the output signal x'(n).

[0214] Thus, an embodiment of the present invention decodes the received bitstreams (i.e., low-frequency bitstream and high-frequency bitstream), obtains low-frequency features and high-frequency features, and invokes the corresponding inverse processing of the encoding side on the low-frequency feature vector to reconstruct the low-frequency part. Then, the reconstructed low-frequency part is up-sampled and filtered to restore the same sampling rate as the original input audio signal. The high-frequency transform coefficients of the high-frequency signal are reconstructed by merging the low-frequency part after up-sample filtering with the high-frequency features, and the reconstructed low-frequency part and the high-frequency transform coefficients are merged to ensure that the energy of the reconstructed high-frequency signal is close to the energy of the high-frequency signal of the encoding side. In this way, the frequency domain coefficients from low to high frequencies can be completely reconstructed.

[0215] The embodiment of the present invention can take in data, jointly train the associated networks on the encoding and decoding sides, and obtain optimal parameters. Users only need to prepare data and set the corresponding network structure, and the trained model can be used after the training is completed in the background.

[0216] From the above, it can be seen that an embodiment of the present invention provides a low-rate neural network encoding / decoding method that effectively combines signal decomposition, signal processing techniques and deep neural networks to ensure audio quality, achieve acceptable complexity, and significantly improve coding efficiency over signal processing schemes.

[0217] The above describes an audio encoding method or an audio decoding method according to an embodiment of the present invention, with reference to exemplary applications and implementations of a terminal device according to an embodiment of the present invention. An embodiment of the present invention also provides an audio encoding device and an audio decoding device. In practical applications, each functional module in the audio encoding device and the audio decoding device can be implemented in cooperation with hardware resources of electronic equipment (e.g., a terminal device, a server, a server cluster), such as computing resources such as a processor, communication resources (e.g., those used to support various communications such as optical cables and cellular networks), and memory. The audio encoding device 555 stored in the memory 550 shown in FIG. 3A and the audio decoding device 556 stored in the memory 550 shown in FIG. 3B may be, for example, a software module designed in a programming language such as software C / C++ or Java, application software designed in a programming language such as C / C++ or Java, or a dedicated software module in a larger software system, an application interface, a plug-in, a program and plug-in for a cloud service, etc. Various embodiments will be described below.

[0218] Here, the audio encoding device 555 includes a series of modules, including a downsampling module 5551, a low-frequency extraction module 5552, a high-frequency analysis module 5553, and an encoding module 5554. The following describes a scheme in which the modules cooperate to achieve audio encoding in the audio encoding device 555 according to an embodiment of the present invention.

[0219] The downsampling module 5551 performs a downsampling process on the audio signal to obtain a low-frequency signal of the audio signal. The low-frequency extraction module 5552 performs a low-frequency feature extraction process on the audio signal to obtain a low-frequency feature of the audio signal. The high-frequency analysis module 5553 performs a high-frequency analysis process on the audio signal to obtain a high-frequency feature of the audio signal, where the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature. The encoding module 5554 performs an encoding process on the low-frequency feature to obtain a low-frequency bitstream of the audio signal, and an encoding process on the high-frequency feature to obtain a high-frequency bitstream of the audio signal.

[0220] In some embodiments, the audio signal includes a plurality of first sampling points obtained by sampling, and the downsampling module 5551 performs a downsampling process on the first sampling points included in the audio signal using a downsampling filter to obtain a low-frequency signal of the audio signal.

[0221] In some embodiments, the downsampling module 5551 performs a digital signal-based filtering process on a first sampling point included in the audio signal using a downsampling filter to obtain the filtered audio signal, and a digital signal-based downsampling process on the filtered audio signal to obtain a low-frequency signal of the audio signal.

[0222] In some embodiments, the low frequency extraction module 5552 performs a convolution operation on the low frequency signal to obtain convolution features of the low frequency signal, performs a pooling operation on the convolution features to obtain pooled features of the low frequency signal, performs a downsampling operation on the pooled features to obtain downsampled features of the low frequency signal, and performs a convolution operation on the downsampled features to obtain low frequency features of the audio signal.

[0223] In some embodiments, the downsampling process is realized by a plurality of cascaded coding layers, and the low-frequency extraction module 5552 performs a downsampling process on the pooling feature by a first coding layer of the cascaded coding layers, outputs the downsampling result of the first coding layer to a subsequent cascaded coding layer, and continues the downsampling process and output of the downsampling result by the subsequent cascaded coding layer until it is output to the last coding layer, and determines the downsampling result output by the last coding layer as the downsampling feature of the low-frequency signal.

[0224] In some embodiments, the high frequency analysis module 5553 performs a frequency band extension process on the audio signal to obtain high frequency features of the audio signal.

[0225] In some embodiments, the high-frequency analysis module 5553 performs a frequency domain transform process on a plurality of second sampling points included in the audio signal, obtains transform coefficients corresponding to each of the plurality of second sampling points, divides the high-frequency transform coefficients among the transform coefficients corresponding to each of the plurality of second sampling points into a plurality of sub-bands, averages the transform coefficients included in each sub-band, obtains an average energy corresponding to each sub-band, determines the average energy as a sub-band spectral envelope corresponding to each sub-band, and determines the sub-band spectral envelope corresponding to each of the plurality of sub-bands as a high-frequency feature of the audio signal.

[0226] In some embodiments, the high frequency analysis module 5553 obtains a plurality of third sampling points included in a reference audio signal, where the reference audio signal is an audio signal adjacent to the audio signal, and performs a discrete cosine transform process on the plurality of second sampling points included in the audio signal based on the plurality of third sampling points included in the reference audio signal and the plurality of second sampling points included in the audio signal, and obtains transform coefficients corresponding to each of the plurality of second sampling points.

[0227] In some embodiments, the high frequency analysis module 5553 determines the sum of squares of the transform coefficients corresponding to the second sampling points included in each sub-band, and determines the ratio of the sum of squares to the number of second sampling points included in the sub-band as the average energy corresponding to each sub-band.

[0228] Here, the audio decoding device 556 includes a series of modules, including a decoding module 5561, a low-frequency reconstruction module 5562, an up-sampling module 5563, and a signal reconstruction module 5564. The following describes a scheme in which the modules cooperate to realize audio encoding in the audio decoding device 556 according to an embodiment of the present invention.

[0229] The decoding module 5561 performs a decoding process on a low-frequency bitstream of the audio signal to obtain low-frequency features corresponding to the low-frequency bitstream, and performs a decoding process on a high-frequency bitstream of the audio signal to obtain high-frequency features corresponding to the high-frequency bitstream. Here, the low-frequency bitstream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and the feature dimension of the high-frequency features is lower than the feature dimension of the low-frequency features. The low-frequency reconstruction module 5562 performs a low-frequency feature reconstruction process on the low-frequency features to obtain a low-frequency signal corresponding to the low-frequency features. The upsampling module 5563 performs an upsampling process on the low-frequency signal to obtain an upsampling signal of the low-frequency signal. The signal reconstruction module 5564 performs a signal reconstruction process on the high-frequency features and the upsampling signal to obtain a synthesized audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

[0230] In some embodiments, the audio signal includes a plurality of sampled fourth sampling points, and the upsampling module 5563 performs an upsampling process on the fourth sampling points included in the low-frequency signal using an upsampling filter to obtain an upsampling signal of the low-frequency signal.

[0231] In some embodiments, the upsampling module 5563 performs, by an upsampling filter, a digital signal-based upsampling process on the fourth sampling point included in the low-frequency signal to obtain the low-frequency signal after upsampling, and a digital signal-based filtering process on the up-sampled low-frequency signal to obtain an upsampling signal of the low-frequency signal.

[0232] In some embodiments, the low-frequency reconstruction module 5562 performs a convolution operation on the low-frequency features to obtain convolved features of the low-frequency features, performs an upsampling operation on the convolved features to obtain up-sampled features of the low-frequency features, performs a pooling operation on the up-sampled features to obtain pooled features of the low-frequency features, and performs a convolution operation on the pooled features to obtain a low-frequency signal corresponding to the low-frequency features.

[0233] In some embodiments, the upsampling process is realized by a plurality of cascaded decoding layers, and the low-frequency reconstruction module 5562 performs an upsampling process on the convolutional features by a first decoding layer of the cascaded decoding layers, outputs the upsampling result of the first decoding layer to a subsequent cascaded decoding layer, and continues the upsampling process and output of the upsampling result by the subsequent cascaded decoding layer until it is output to the last decoding layer, and determines the upsampling result output by the last decoding layer as the upsampling feature of the low-frequency features.

[0234] In some embodiments, the signal reconstruction module 5564 performs a frequency domain transform process on a plurality of fifth sampling points included in the upsampled signal to obtain transform coefficients corresponding to each of the plurality of fifth sampling points, performs a frequency band extension process on the transform coefficients corresponding to each of the plurality of fifth sampling points and the high-frequency features to obtain high-frequency transform coefficients of the high-frequency signal, and determines a synthesized audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream based on the high-frequency transform coefficients and the transform coefficients corresponding to each of some of the plurality of fifth sampling points.

[0235] In some embodiments, the signal reconstruction module 5564 performs a frequency spectrum replication process on at least some of the transform coefficients of the first half portion of the transform coefficients corresponding to each of the plurality of fifth sampling points to obtain reference high-frequency transform coefficients of the reference high-frequency signal, and performs a gain process on the reference high-frequency transform coefficients of the reference high-frequency signal based on a sub-band spectral envelope corresponding to the high-frequency feature to obtain high-frequency transform coefficients of the high-frequency signal.

[0236] In some embodiments, the signal reconstruction module 5564 performs the following processes: divide reference high-frequency transform coefficients of the reference high-frequency signal into a plurality of sub-bands based on sub-band spectral envelopes corresponding to the high-frequency features; for each sub-band among the plurality of sub-bands, determine a first average energy of the high-frequency sub-band corresponding to the sub-band in the sub-band spectral envelope; determine a second average energy of the sub-band; determine a gain factor based on a ratio between the first average energy and the second average energy; multiply each reference high-frequency transform coefficient included in the sub-band by the gain factor to obtain a high-frequency transform coefficient corresponding to the sub-band; and determine the high-frequency transform coefficients corresponding to each of the plurality of sub-bands as the high-frequency transform coefficients of the high-frequency signal.

[0237] In some embodiments, the signal reconstruction module 5564 performs a frequency spectrum replication process on the transform coefficients of the latter part of the first part of the transform coefficients corresponding to each of the plurality of fifth sampling points to obtain reference high-frequency transform coefficients of the reference high-frequency signal.

[0238] In some embodiments, the signal reconstruction module 5564 performs a merging process on the high-frequency transform coefficients and the transform coefficients corresponding to each of the first half of the plurality of fifth sampling points to obtain full transform coefficients, and performs an inverse transform process of the frequency spectrum transform on the full transform coefficients to obtain a composite audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

[0239] An embodiment of the present invention further provides a computer program or computer program product including computer-executable instructions or a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads the computer-executable instructions or the computer program from the computer-readable storage medium, and the processor executes the computer-executable instructions or the computer program to cause the electronic device to perform the audio encoding method or the audio decoding method according to the embodiment of the present invention.

[0240] An embodiment of the present invention further provides a computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, perform the above-described audio encoding method or audio decoding method according to the embodiment of the present invention, such as the audio encoding method shown in Figures 6A to 6D and the audio decoding method shown in Figures 7A to 7D.

[0241] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM, or may be any device including one or any combination of the above memories.

[0242] In some embodiments, the computer-executable instructions may be created in any type of programming language (including compiled or interpreted languages, or declarative or procedural languages) in the form of a program, software, software module, script, or code, and may be implemented in any form, for example, as a stand-alone program, a module, component, subroutine, or other unit suitable for use in a computing environment.

[0243] As an example, the executable instructions may correspond to a file in a file system, or may be stored as part of a file for holding other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program being referenced, or in multiple cooperating files (e.g., files for storing one or more modules, subprograms, or code sections).

[0244] As an example, the computer-executable instructions may be implemented and executed on a single electronic device, may be executed on multiple electronic devices located at a single location, or may be executed on multiple electronic devices distributed at multiple locations and interconnected via a communications network.

[0245] It should be noted that in the embodiments of the present invention, when applying the embodiments of the present invention to specific products or technologies, with regard to related data such as user information, permission or consent from the user must be obtained, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0246] The above merely describes exemplary embodiments of the present invention, and does not limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are included within the protection scope of the present invention.

Claims

1. An audio encoding method performed by an electronic device, comprising: performing a downsampling process on the audio signal to obtain a low frequency component of the audio signal; performing a low-frequency feature extraction process on the audio signal to obtain low-frequency features of the audio signal; performing a high-frequency analysis process on the audio signal to obtain high-frequency features of the audio signal, wherein a feature dimension of the high-frequency features is lower than a feature dimension of the low-frequency features; performing an encoding process on the low frequency features to obtain a low frequency bitstream of the audio signal, and performing an encoding process on the high frequency features to obtain a high frequency bitstream of the audio signal.

2. the audio signal includes a plurality of first sampling points obtained by sampling; The step of performing a downsampling process on the audio signal to obtain a low frequency component of the audio signal includes: The method of claim 1 , further comprising: performing a downsampling process on a first sampling point included in the audio signal using a downsampling filter to obtain a low-frequency signal of the audio signal.

3. The step of performing a downsampling process on a first sampling point included in the audio signal by the downsampling filter to obtain a low frequency signal of the audio signal includes: The downsampling filter a process of performing a filtering process based on a digital signal on first sampling points included in the audio signal, and obtaining the filtered audio signal; and The method of claim 2 , further comprising: performing a downsampling process based on a digital signal on the filtered audio signal to obtain low-frequency components of the audio signal.

4. The step of performing a low-frequency feature extraction process on the audio signal to obtain low-frequency features of the audio signal includes: performing a convolution process on the low-frequency signal to obtain a convolution feature of the low-frequency signal; performing a pooling process on the convolution features to obtain pooled features of the low-frequency signal; performing a downsampling process on the pooled features to obtain downsampling features of the low-frequency signal; and performing a convolution process on the downsampled features to obtain low frequency features of the audio signal.

5. the downsampling process is achieved by multiple cascaded coding layers; performing a downsampling process on the pooled features to obtain downsampling features of the low-frequency signal, performing a downsampling operation on the pooled features by a first coding layer of the cascaded coding layers; outputting the downsampling result of the first coding layer to a subsequent cascaded coding layer, and continuing the downsampling process and output of the downsampling result by the subsequent cascaded coding layer until the downsampling process is output to the last coding layer; and determining the downsampling result output by the last coding layer as the downsampling characteristic of the low-frequency signal.

6. performing a high frequency analysis process on the audio signal to obtain high frequency features of the audio signal, The method of claim 1 , comprising performing a frequency band extension process on the audio signal to obtain high frequency features of the audio signal.

7. performing a frequency band extension process on the audio signal to obtain high frequency features of the audio signal, performing a frequency domain transform process on a plurality of second sampling points included in the audio signal to obtain transform coefficients corresponding to the plurality of second sampling points; dividing high frequency transform coefficients among the transform coefficients corresponding to each of the plurality of second sampling points into a plurality of sub-bands; averaging transform coefficients included in each of the sub-bands to obtain an average energy corresponding to each of the sub-bands, and determining the average energy as a sub-band spectral envelope corresponding to each of the sub-bands; and determining a subband spectral envelope corresponding to each of the plurality of subbands as a high frequency feature of the audio signal.

8. The step of performing a frequency domain transform process on a plurality of second sampling points included in the audio signal and obtaining transform coefficients corresponding to each of the plurality of second sampling points includes: obtaining a plurality of third sampling points included in a reference audio signal, the reference audio signal being an audio signal adjacent to the audio signal; and performing a discrete cosine transform process on the second sampling points included in the audio signal based on the third sampling points included in the reference audio signal and the second sampling points included in the audio signal, to obtain transform coefficients corresponding to each of the second sampling points.

9. Averaging transform coefficients included in each of the sub-bands to obtain an average energy corresponding to each of the sub-bands comprises: determining the sum of squares of transform coefficients corresponding to second sampling points included in each of said sub-bands; and determining a ratio of the sum of squares to the number of second sampling points included in the subband as an average energy corresponding to each of the subbands.

10. 1. An audio decoding method executed by an electronic device, comprising: performing a decoding process on a low-frequency bitstream of an audio signal to obtain low-frequency features corresponding to the low-frequency bitstream, and performing a decoding process on a high-frequency bitstream of the audio signal to obtain high-frequency features corresponding to the high-frequency bitstream, wherein the low-frequency bitstream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and a feature dimension of the high-frequency features is lower than a feature dimension of the low-frequency features; performing a low-frequency feature reconstruction process on the low-frequency features to obtain low-frequency signals corresponding to the low-frequency features; performing an upsampling process on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal; performing a signal reconstruction process on the high-frequency features and the up-sampled signal to obtain a synthesized audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

11. The step of performing an upsampling process on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal includes: The method of claim 10 , further comprising: performing an upsampling process on every fourth sampling point included in the low-frequency signal using an upsampling filter to obtain an up-sampled signal of the low-frequency signal.

12. The step of performing an upsampling process on a fourth sampling point included in the low-frequency signal by the upsampling filter to obtain an up-sampled signal of the low-frequency signal includes: The upsampling filter performing an upsampling process based on a digital signal on a fourth sampling point included in the low-frequency signal, and acquiring the low-frequency signal after upsampling; and The method of claim 11 , further comprising: performing a filtering process based on a digital signal on the upsampled low-frequency signal to obtain an upsampled signal of the low-frequency signal.

13. performing a low-frequency feature reconstruction process on the low-frequency features to obtain a low-frequency signal corresponding to the low-frequency features, performing a convolution process on the low-frequency features to obtain convolution features of the low-frequency features; performing an upsampling process on the convolved features to obtain upsampled features of the low frequency features; performing a pooling operation on the upsampled features to obtain pooled features of the low-frequency features; and performing a convolution process on the pooled features to obtain a low-frequency signal corresponding to the low-frequency features.

14. The upsampling process is realized by a plurality of cascaded decoding layers; performing an upsampling process on the convolved features to obtain upsampled features of the low-frequency features, performing an upsampling operation on the convolutional features by a first decoding layer of the cascaded plurality of decoding layers; outputting the upsampling result of the first decoding layer to a subsequent cascaded decoding layer, and continuing the upsampling process and outputting the upsampling result by the subsequent cascaded decoding layer until the upsampling result is output to the last decoding layer; and determining the upsampling result output by the last decoding layer as the upsampling feature of the low-frequency feature.

15. performing a signal reconstruction process on the high-frequency features and the up-sampled signal to obtain a synthesized audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream, performing a frequency domain transform process on a plurality of fifth sampling points included in the upsampled signal to obtain transform coefficients corresponding to the plurality of fifth sampling points; performing a frequency band extension process on the transform coefficients corresponding to each of the fifth sampling points and the high-frequency features to obtain high-frequency transform coefficients of a high-frequency signal; and determining a composite audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream based on the high-frequency transform coefficients and transform coefficients corresponding to each of a portion of the plurality of fifth sampling points.

16. performing a frequency band extension process on the transform coefficients corresponding to each of the plurality of fifth sampling points and the high-frequency features to obtain high-frequency transform coefficients of a high-frequency signal; performing a frequency spectrum replication process on at least some transform coefficients in a first half portion of the transform coefficients corresponding to each of the plurality of fifth sampling points to obtain reference high-frequency transform coefficients of a reference high-frequency signal; and performing a gain operation on reference high-frequency transform coefficients of the reference high-frequency signal based on a sub-band spectral envelope corresponding to the high-frequency feature to obtain high-frequency transform coefficients of the high-frequency signal.

17. performing a gain process on reference high-frequency transform coefficients of the reference high-frequency signal according to a sub-band spectral envelope corresponding to the high-frequency feature to obtain high-frequency transform coefficients of the high-frequency signal; Dividing reference high-frequency transform coefficients of the reference high-frequency signal into a plurality of sub-bands based on sub-band spectral envelopes corresponding to the high-frequency features; For each sub-band of the plurality of sub-bands, determining a first average energy of a high frequency subband corresponding to said subband in said subband spectral envelope; and determining a second average energy of said subband; determining a gain factor based on a ratio of the first average energy to the second average energy; multiplying the gain factor by each of the reference high-frequency transform coefficients included in the sub-bands to obtain high-frequency transform coefficients corresponding to the sub-bands; and and performing a process for determining high-frequency transform coefficients corresponding to each of the plurality of sub-bands as high-frequency transform coefficients of the high-frequency signal.

18. The step of performing a frequency spectrum replication process on at least some transform coefficients in a first half portion of the transform coefficients corresponding to each of the plurality of fifth sampling points to obtain reference high-frequency transform coefficients of a reference high-frequency signal includes:

17. The method of claim 16, further comprising: performing a frequency spectrum replication process on transform coefficients of a second half of the first half of the transform coefficients corresponding to each of the plurality of fifth sampling points to obtain reference high-frequency transform coefficients of a reference high-frequency signal.

19. determining a composite audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream based on the high-frequency transform coefficients and transform coefficients corresponding to each of a portion of the plurality of fifth sampling points, performing a merging process on the high-frequency transform coefficients and the transform coefficients corresponding to the first half fifth sampling points of the plurality of fifth sampling points to obtain complete transform coefficients; and performing an inverse transform operation of a frequency spectrum transform on the full transform coefficients to obtain a composite audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

20. An audio encoding device, comprising: a downsampling module for performing a downsampling process on the audio signal to obtain a low frequency component of the audio signal; a low-frequency extraction module that performs a low-frequency feature extraction process on the audio signal to obtain low-frequency features of the audio signal; a high-frequency analysis module that performs a high-frequency analysis process on the audio signal to obtain high-frequency features of the audio signal, wherein a feature dimension of the high-frequency features is lower than a feature dimension of the low-frequency features; an encoding module that performs an encoding process on the low-frequency features to obtain a low-frequency bitstream of the audio signal, and that performs an encoding process on the high-frequency features to obtain a high-frequency bitstream of the audio signal.

21. An audio decoding device, a decoding module that performs a decoding process on a low-frequency bitstream of an audio signal to obtain low-frequency features corresponding to the low-frequency bitstream, and performs a decoding process on a high-frequency bitstream of the audio signal to obtain high-frequency features corresponding to the high-frequency bitstream, wherein the low-frequency bitstream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and a feature dimension of the high-frequency features is lower than a feature dimension of the low-frequency features; a low-frequency reconstruction module that performs a low-frequency feature reconstruction process on the low-frequency features to obtain low-frequency signals corresponding to the low-frequency features; an upsampling module that performs an upsampling process on the low-frequency signal to obtain an upsampling signal of the low-frequency signal; a signal reconstruction module that performs a signal reconstruction process on the high-frequency features and the upsampled signal to obtain a composite audio signal corresponding to the low-frequency bitstream and the high-frequency bitstream.

22. a memory having computer programs or computer-executable instructions stored therein; and a processor which, when executing a computer program or computer-executable instructions stored in the memory, implements the audio encoding method of any one of claims 1 to 9 or the audio decoding method of any one of claims 10 to 19.

23. A computer program causing a computer to execute the audio encoding method according to any one of claims 1 to 9 or the audio decoding method according to any one of claims 10 to 19.

Images