Audio encoding method and apparatus, audio decoding method and apparatus, device, and storage medium
By downsampling and feature extraction of audio signals, and using neural networks to perform low-frequency and high-frequency feature encoding of audio signals, the problem of low audio encoding efficiency in the prior art is solved, and efficient audio encoding is achieved.
Patent Information
- Application Number
- PCT/CN2024/091202
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-24
- Filing Date
- 2024-05-06
- Publication Date
- 2024-11-28
AI Technical Summary
While ensuring audio quality, existing audio encoding technologies are difficult to improve coding efficiency, resulting in waste of network bandwidth resources.
By downsampling the audio signal, low-frequency signals and high-frequency signals are extracted respectively, and feature extraction and encoding is used for neural networks. The low-frequency feature dimension is greater than the high-frequency feature dimension, reducing the feature dimension and encoding it to achieve efficient encoding.
While ensuring audio quality, the audio encoding efficiency is significantly improved and the consumption of network bandwidth resources is reduced.
Smart Images

Figure CN2024091202_28112024_PF_FP_ABST
Abstract
Description
Audio encoding method, audio decoding method, device, equipment and storage medium
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] The embodiments of this application are based on and claim the priority of Chinese patent application with application number 202310597138.3 and application date May 24, 2023. The entire contents of the Chinese patent application are hereby introduced into the embodiments of this application as a reference. Technical Field
[0003] The present application relates to artificial intelligence technology, and in particular to an audio encoding method, an audio decoding method, an apparatus, a device, and a storage medium. Background Art
[0004] Artificial Intelligence (AI) is a comprehensive field in computer science. By studying the design principles and implementation methods of various intelligent machines, AI enables them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in even more areas and play an increasingly important role.
[0005] Audio coding and decoding technology is a key application in the field of artificial intelligence and a core technology for communication services, including remote audio and video calls. Simply put, speech coding technology aims to transmit as much voice information as possible using less network bandwidth. From the perspective of Shannon's information theory, speech coding is a form of source coding. The goal of source coding is to compress the data volume as much as possible at the encoding end, removing redundancy, while enabling lossless (or near-lossless) recovery at the decoding end.
[0006] In related technologies, during the encoding process, in order to ensure audio quality, the efficiency of audio encoding will be greatly reduced.
[0007] Summary of the Invention
[0008] Embodiments of the present application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve audio encoding efficiency while ensuring audio quality.
[0009] The technical solution of the embodiment of the present application is implemented as follows:
[0010] The present application provides an audio encoding method, which is applied to an electronic device and includes:
[0011] Downsampling the audio signal to obtain a low-frequency signal of the audio signal;
[0012] Performing low-frequency feature extraction processing on the low-frequency signal to obtain low-frequency features of the audio signal;
[0013] Performing high-frequency analysis on the audio signal to obtain high-frequency features of the audio signal;
[0014] The characteristic dimension of the high-frequency feature is lower than the characteristic dimension of the low-frequency feature;
[0015] The low-frequency features are encoded to obtain a low-frequency code stream of the audio signal, and the high-frequency features are encoded to obtain a high-frequency code stream of the audio signal.
[0016] The present invention provides an audio decoding method for an electronic device, including:
[0017] Decoding a low-frequency code stream of an audio signal to obtain low-frequency features corresponding to the low-frequency code stream, and decoding a high-frequency code stream of the audio signal to obtain high-frequency features corresponding to the high-frequency code stream;
[0018] The low-frequency code stream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature;
[0019] Performing low-frequency feature reconstruction processing on the low-frequency feature to obtain a low-frequency signal corresponding to the low-frequency feature;
[0020] performing upsampling processing on the low-frequency signal to obtain an upsampled signal of the low-frequency signal;
[0021] Signal reconstruction processing is performed on the high-frequency features and the up-sampled signal to obtain a synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0022] In the above technical solution, the upsampling process is implemented through multiple cascaded decoding layers;
[0023] The upsampling process is performed on the convolution feature to obtain the upsampled feature of the low-frequency feature, including:
[0024] Performing upsampling processing on the convolutional features through a first decoding layer in the multiple cascaded decoding layers;
[0025] Outputting the upsampling result of the first decoding layer to the subsequent cascaded decoding layers, and continuing the upsampling process and outputting the upsampling result through the subsequent cascaded decoding layers until it is output to the last decoding layer;
[0026] An upsampling result output by the last decoding layer is determined as an upsampling feature of the low-frequency feature.
[0027] The present invention provides an audio encoding device, including:
[0028] A downsampling module is configured to perform downsampling processing on the audio signal to obtain a low-frequency signal of the audio signal;
[0029] a low-frequency extraction module configured to perform low-frequency feature extraction processing on the low-frequency signal to obtain the low-frequency features of the audio signal;
[0030] a high-frequency analysis module configured to perform high-frequency analysis processing on the audio signal to obtain high-frequency features of the audio signal;
[0031] The characteristic dimension of the high-frequency feature is lower than the characteristic dimension of the low-frequency feature;
[0032] The encoding module is configured to encode the low-frequency features to obtain a low-frequency code stream of the audio signal, and to encode the high-frequency features to obtain a high-frequency code stream of the audio signal.
[0033] The present invention provides an audio decoding device, including:
[0034] a decoding module configured to decode a low-frequency code stream of the audio signal to obtain low-frequency features corresponding to the low-frequency code stream, and to decode a high-frequency code stream of the audio signal to obtain high-frequency features corresponding to the high-frequency code stream;
[0035] The low-frequency code stream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature;
[0036] a low-frequency reconstruction module configured to perform low-frequency feature reconstruction processing on the low-frequency feature to obtain a low-frequency signal corresponding to the low-frequency feature;
[0037] an upsampling module configured to perform upsampling processing on the low-frequency signal to obtain an upsampled signal of the low-frequency signal;
[0038] The signal reconstruction module is configured to perform signal reconstruction processing on the high-frequency features and the up-sampled signal to obtain a synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0039] An embodiment of the present application provides an electronic device, comprising:
[0040] Memory for storing computer programs or computer-executable instructions;
[0041] The processor is used to implement the audio encoding method or audio decoding method provided in the embodiment of the present application when executing the computer program or computer executable instructions stored in the memory.
[0042] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the audio encoding method or audio decoding method provided in the embodiment of the present application is implemented.
[0043] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the audio encoding method or audio decoding method provided in the embodiment of the present application is implemented.
[0044] The embodiments of the present application have the following beneficial effects:
[0045] The audio signal is downsampled to obtain a low-frequency signal. Since the low-frequency signal has a greater impact on audio coding than the high-frequency signal in the audio signal, differentiated signal processing is used to extract the low-frequency and high-frequency features of the audio signal respectively, so that the feature dimension of the high-frequency feature is lower than that of the low-frequency feature. The low-frequency and high-frequency features with reduced feature dimensions are encoded separately, thereby improving the audio coding efficiency while ensuring the audio quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] FIG1 is a schematic diagram showing a comparison of spectrums at different bit rates provided in an embodiment of the present application;
[0047] FIG2 is a schematic diagram of the architecture of an audio codec system provided in an embodiment of the present application;
[0048] 3A-3B are schematic structural diagrams of electronic devices provided in embodiments of the present application;
[0049] FIG4A is a schematic diagram of a common convolutional network provided in an embodiment of the present application;
[0050] FIG4B is a schematic diagram of a dilated convolutional network provided in an embodiment of the present application;
[0051] FIG5 is a schematic diagram of frequency band extension provided by an embodiment of the present application;
[0052] FIG6A is a schematic diagram of a first flow chart of an audio encoding method provided in an embodiment of the present application;
[0053] FIG6B is a schematic diagram of a second flow chart of the audio encoding method provided in an embodiment of the present application;
[0054] FIG6C is a schematic diagram of a third flow chart of the audio encoding method provided in an embodiment of the present application;
[0055] FIG6D is a schematic diagram of a fourth flow chart of the audio encoding method provided in an embodiment of the present application;
[0056] FIG7A is a schematic diagram of a first flow chart of an audio decoding method provided in an embodiment of the present application;
[0057] FIG7B is a schematic diagram of a second flow chart of the audio decoding method provided in an embodiment of the present application;
[0058] FIG7C is a schematic diagram of a third flow chart of the audio decoding method provided in an embodiment of the present application;
[0059] FIG7D is a schematic diagram of a fourth flow chart of the audio decoding method provided in an embodiment of the present application;
[0060] FIG8 is a schematic diagram of an end-to-end voice communication link provided in an embodiment of the present application;
[0061] FIG9 is a schematic diagram of a flow chart of a low-rate neural network encoding and decoding method provided in an embodiment of the present application;
[0062] FIG10 is a schematic diagram of a first neural network provided in an embodiment of the present application;
[0063] FIG11 is a schematic diagram of a second neural network provided in an embodiment of the present application. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0065] In the following description, the terms "first\second" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0066] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.
[0068] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0069] 1) Neural Network (NN): A mathematical model that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system to process information by adjusting the connections between its numerous nodes.
[0070] 2) Deep Learning (DL): A new research direction in the field of machine learning (ML), deep learning studies the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sounds. Its ultimate goal is to enable machines to have human-like analytical learning capabilities and to recognize data such as text, images, and sounds.
[0071] 3) Quantization: This is the process of approximating a signal's continuous values (or a large number of discrete values) to a finite number (or a small number) of discrete values. Quantization includes vector quantization (VQ) and scalar quantization.
[0072] Vector quantization is an effective lossy compression technique, based on Shannon's rate-distortion theory. The basic principle of vector quantization is to replace the input vector with the index of the codeword in the codebook that best matches it for transmission and storage, while decoding requires only a simple table lookup. For example, several scalar data items form a vector space, which is then divided into several small regions. During quantization, the vectors falling into these small regions are replaced with the corresponding indices of the input vector.
[0073] Scalar quantization is the quantization of scalars, i.e. one-dimensional vector quantization, which divides the dynamic range into several small intervals, each of which has a representative value (i.e., index). When the input signal falls into a certain interval, the input signal is quantized to the representative value.
[0074] 4) Entropy Coding: This is a lossless coding method that uses the entropy principle to prevent any information loss during the encoding process. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exp-Golomb coding, and arithmetic coding.
[0075] Speech coding technology aims to transmit as much voice information as possible using less network bandwidth. Speech codecs can achieve compression ratios of over 10 times. This means that 10MB of speech data can be compressed by the codec to only 1MB, significantly reducing the bandwidth required to transmit the information. For example, for a wideband speech signal with a sampling rate of 16,000Hz, if a 16-bit sampling depth (the level of detail in recording speech intensity during sampling) is used, the bit rate (the amount of data transmitted per unit time) of the uncompressed version is 256kbps. Using speech coding technology, even with lossy encoding, the quality of the reconstructed speech signal within a bit rate range of 10-20kbps can be close to that of the uncompressed version, and may even be perceived as indistinguishable. For services requiring higher sampling rates, such as ultra-wideband speech at 32,000Hz, the bit rate must be at least 30kbps.
[0076] To ensure smooth communication within communication systems, the industry deploys standard voice codec protocols, such as those from international and domestic standards organizations like ITU-T, 3GPP, IETF, AVS, and CCSA, as well as standards like G.711, G.722, the AMR series, EVS, and OPUS. Figure 1 shows a schematic diagram comparing spectra at different bit rates, demonstrating the relationship between compression bit rate and quality. Curve 101 shows the spectrum of the original speech (i.e., the uncompressed signal); Curve 102 shows the spectrum of the OPUS encoder at a bit rate of 20 kbps; and Curve 103 shows the spectrum of the OPUS encoder at a bit rate of 6 kbps. As shown in Figure 1, as the encoding bit rate increases, the compressed signal becomes closer to the original signal.
[0077] In related technologies, the principles of speech coding are roughly as follows: speech coding can directly encode speech waveform samples sample by sample; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.
[0078] The above coding principles all come from voice signal modeling, that is, compression methods based on signal processing, which cannot guarantee the encoding quality of audio. In order to improve coding efficiency while ensuring voice quality, an embodiment of the present application provides an audio encoding method, an audio decoding method, a device, an electronic device, a computer-readable storage medium, and a computer program product based on artificial intelligence. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as a terminal device, or as a server, or implemented in collaboration with a terminal device and a server. The following is an example of an electronic device implemented as a terminal device.
[0079] For example, see Figure 2, which is a schematic diagram of the architecture of the audio encoding and decoding system 10 provided in an embodiment of the present application. The audio decoding system 10 includes: a server 200, a network 300, a terminal device 400 (i.e., an encoding end) and a terminal device 500 (i.e., a decoding end), wherein the network 300 can be a local area network, a wide area network, or a combination of the two.
[0080] In some embodiments, a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as an instant messaging client, a web conferencing client, a live broadcast client, a browser, etc. In response to an audio collection instruction triggered by a sender (such as the initiator of a web conferencing session, a host, or the initiator of a voice call), the client 410 calls the microphone of the terminal device 400 to collect audio signals, and performs audio encoding processing on the collected audio signals to obtain bit streams (high-frequency bit streams and low-frequency bit streams).
[0081] For example, the client 410 calls the audio encoding method provided in the embodiment of the present application to encode the collected audio signal, that is, downsampling the audio signal to obtain a low-frequency signal of the audio signal; performing low-frequency feature extraction on the low-frequency signal to obtain the low-frequency features of the audio signal; performing high-frequency analysis on the audio signal to obtain the high-frequency features of the audio signal; wherein the feature dimension of the high-frequency features is lower than the feature dimension of the low-frequency features; encoding the low-frequency features to obtain a low-frequency code stream of the audio signal, and encoding the high-frequency features to obtain a high-frequency code stream of the audio signal. The encoding end (i.e., the terminal device 400) combines signal processing technology with artificial intelligence technology to downsample the audio signal to obtain a low-frequency signal. Since the low-frequency signal has a greater impact on audio encoding than the high-frequency signal in the audio signal, the low-frequency features and high-frequency features of the audio signal are extracted separately through differentiated signal processing, so that the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature, and the low-frequency features and high-frequency features with reduced feature dimensions are encoded separately, thereby improving audio encoding efficiency while ensuring audio quality.
[0082] The client 410 can send the code stream of the audio signal (i.e., the high-frequency code stream and the low-frequency code stream) to the server 200 through the network 300, so that the server 200 sends the code stream (the high-frequency code stream and the low-frequency code stream) to the terminal device 500 associated with the recipient (such as the participants of the online conference, the audience, the recipient of the voice call, etc.).
[0083] After receiving the code stream (high-frequency code stream and low-frequency code stream) sent by the server 200, the client 510 (such as an instant messaging client, a web conferencing client, a live broadcast client, a browser, etc.) running on the terminal device 500 can perform audio decoding processing on the code stream to obtain an audio signal (i.e., a synthesized audio signal), thereby realizing audio communication.
[0084] For example, the client 510 calls the audio decoding method provided in the embodiment of the present application to decode the received code stream (high-frequency code stream and low-frequency code stream), that is, the low-frequency code stream of the audio signal is decoded to obtain the low-frequency features corresponding to the low-frequency code stream, and the high-frequency code stream of the audio signal is decoded to obtain the high-frequency features corresponding to the high-frequency code stream; the low-frequency features are reconstructed to obtain the low-frequency signal corresponding to the low-frequency features; the low-frequency signal is up-sampled to obtain the up-sampled signal of the low-frequency signal; the high-frequency features and the up-sampled signal are reconstructed to obtain the synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0085] In some embodiments, the embodiments of the present application can be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize data calculation, storage, processing, and sharing.
[0086] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become an important support. The service interaction functions between the above-mentioned servers 200 can be realized through cloud technology.
[0087] For example, the server 200 shown in Figure 2 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal devices 400 and 500 shown in Figure 2 can be various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, in-vehicle devices), smart phones, smart speakers, smart watches, smart TVs, in-vehicle terminals, etc., but are not limited thereto. The terminal devices (e.g., terminal devices 400 and 500) and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0088] In some embodiments, the terminal device or server 200 can also implement the audio encoding method or audio decoding method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP, a web conferencing APP, or an instant messaging APP, etc.; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it into a browser environment. In short, the above-mentioned computer program can be any form of application, module or plug-in.
[0089] In some embodiments, multiple servers may form a blockchain, and server 200 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information can be transmitted between nodes through the above information connections. Among them, data related to the audio encoding method or audio decoding method provided in the embodiments of the present application (such as audio encoding logic, audio decoding logic, high-frequency bit stream, and low-frequency bit stream) can be stored on the blockchain.
[0090] 3A , which is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application. Taking the electronic device 500 as a terminal device as an example, the electronic device 500 shown in FIG3A includes: at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The various components in the electronic device 500 are coupled together via a bus system 550. It will be understood that the bus system 550 is used to implement connection and communication between these components. In addition to including a data bus, the bus system 550 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in FIG3A , various buses are labeled as the bus system 550.
[0091] The processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0092] The user interface 540 includes one or more output devices 541 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 540 also includes one or more input devices 542, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0093] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 520.
[0094] The memory 550 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.
[0095] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.
[0096] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0097] A network communication module 552 for reaching other computing devices via one or more (wired or wireless) network interfaces 530 , exemplary network interfaces 530 including Bluetooth, WiFi, and USB;
[0098] In some embodiments, the audio encoding device or audio decoding device provided in the embodiments of the present application can be implemented in software. Figure 3A shows an audio encoding device 555 stored in the memory 550, which can be software in the form of programs and plug-ins, including the following software modules: downsampling block 5551, low-frequency extraction module 5552, high-frequency analysis module 5553, encoding module 5554, wherein the downsampling block 5551, low-frequency extraction module 5552, high-frequency analysis module 5553, encoding module 5554 are used to implement audio encoding functions. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented.
[0099] 3B , which is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application. The structure of electronic device 600 is similar to that of electronic device 500. The electronic device 600 shown in FIG3B includes at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The various components in electronic device 500 are coupled together via a bus system 550.
[0100] The user interface 540 includes one or more output devices 541 that enable presentation of media content. The user interface 540 also includes one or more input devices 542.
[0101] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.
[0102] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0103] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 530. Exemplary network interfaces 530 include Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB).
[0104] Figure 3B shows an audio decoding device 556 stored in the memory 550, which can be software in the form of a program and plug-in, including the following software modules: a decoding module 5561, a low-frequency reconstruction module 5562, an upsampling module 5563, and a signal reconstruction module 5564, wherein the decoding module 5561, the low-frequency reconstruction module 5562, the upsampling module 5563, and the signal reconstruction module 5564 are used to implement the audio decoding function. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented.
[0105] Before specifically introducing the audio encoding method and audio decoding method provided in the embodiments of the present application, the following introduces the dilated convolutional network and frequency band extension technology.
[0106] Referring to Figures 4A and 4B, Figure 4A is a schematic diagram of a normal convolutional network provided in an embodiment of the present application, and Figure 4B is a schematic diagram of a dilated convolutional network provided in an embodiment of the present application. Compared with ordinary convolutional networks, dilated convolution can increase the receptive field while keeping the size of the feature map unchanged, and can also avoid errors caused by upsampling and downsampling. Although the convolution kernel sizes (Kernel Size) shown in Figures 4A and 4B are both 3×3; however, the receptive field 401 of the ordinary convolution shown in Figure 4A is only 3, while the receptive field 402 of the dilated convolution shown in Figure 4B reaches 5. That is to say, for a convolution kernel of size 3×3, the receptive field of the ordinary convolution shown in Figure 4A is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1; while the receptive field of the dilated convolution shown in Figure 4B is 5, and the dilation rate is 2.
[0107] The convolution kernel can also be moved on a plane similar to Figure 4A or Figure 4B. This involves the concept of stride rate. For example, each time the convolution kernel shifts 1 square, the corresponding stride rate is 1.
[0108] There's also the concept of convolution channels, which refers to the number of convolution kernel parameters used to perform the convolution analysis. In theory, a greater number of channels provides a more comprehensive signal analysis and higher accuracy; however, a higher number of channels also increases complexity. For example, a 1×320 tensor can be convolved using 24 channels, resulting in a 24×320 tensor as the output.
[0109] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the size of the convolution kernel can be set to 1×3), the expansion rate, the shift rate and the number of channels can be defined by yourself according to actual application needs. The embodiments of this application do not make specific restrictions on this.
[0110] Figure 5 shows a schematic diagram of frequency band extension (or band replication). First, the wideband signal is reconstructed, then copied onto an ultra-wideband signal, and finally shaped based on the ultra-wideband envelope. The frequency domain implementation scheme shown in Figure 5 specifically includes: 1) implementing a core layer encoding at a low sampling rate; 2) selectively copying the low-frequency spectrum to the high-frequency portion; and 3) performing gain control on the copied high-frequency spectrum based on pre-recorded boundary information (such as describing the energy correlation between high and low frequencies). With a bit rate of only 1-2 kbps, the sampling rate can be doubled.
[0111] As previously mentioned, the audio encoding method provided in the embodiments of the present application can be implemented by various types of electronic devices, such as a terminal, a server, or a combination thereof. Therefore, the execution entities of each step will not be repeated below. Referring to FIG. 6A , FIG. 6A is a first flow chart of the audio encoding method provided in the embodiments of the present application. The audio encoding function is implemented by the audio encoding method, and the steps shown in FIG. 6A are described below.
[0112] In step 101, down-sampling processing is performed on the audio signal to obtain a low-frequency signal of the audio signal.
[0113] As an example of obtaining an audio signal, the encoding end responds to the audio collection instruction triggered by the sender (such as the initiator of the online conference, the host, the initiator of the voice call, etc.), and calls the microphone of the terminal device of the encoding end to collect the audio signal to obtain the audio signal x(n) (also called the input signal).
[0114] After acquiring the audio signal, the audio signal is downsampled to extract the low-frequency signal from the audio signal. Since the low-frequency signal has a greater impact on audio encoding than the high-frequency signal, the low-frequency signal and the high-frequency signal can be subsequently differentiated for signal processing.
[0115] The audio signal includes a low-frequency part and a high-frequency part. The low-frequency signal is separated from the audio signal of a specific sampling rate by a filter based on the characteristics of the audio signal. The high-frequency signal is separated from the audio signal of a specific sampling rate. For example, if the effective bandwidth of the audio signal x(n) is 0-16kHz, the effective bandwidth of the low-frequency signal is 0-8kHz, and the effective bandwidth of the high-frequency signal x is 0-8kHz. HBThe effective bandwidth of (n) may be 6-16 kHz. In addition, the embodiment of the present application does not limit the frequency band division of the audio signal. For example, the audio signal may be divided evenly or unevenly to obtain uniform low-frequency signals and high-frequency signals.
[0116] It should be noted that the audio signal is a discrete digital signal, and the audio signal includes a plurality of first sample points obtained by sampling, wherein the first sample points are sampling values obtained by sampling a continuous analog signal.
[0117] In some embodiments, the audio signal includes a plurality of first sample points obtained by sampling; step 101 may be implemented by performing downsampling processing on the first sample points included in the audio signal through a downsampling filter to obtain a low-frequency signal of the audio signal.
[0118] In the field of digital signal processing, downsampling filters are used to reduce the sampling rate of audio signals to reduce data volume, reduce system complexity, or adapt to specific application requirements. The downsampling factor of the downsampling filter can be a multiple of 2, such as 2, 4, 8, etc.
[0119] Here, the downsampling filter performs the following processing: performing digital signal-based filtering on the first sampling point of the audio signal to obtain a filtered audio signal; and performing digital signal-based downsampling on the filtered audio signal to obtain a low-frequency signal of the audio signal. The digital signal-based filtering can be low-pass filtering or band-pass filtering.
[0120] For example, in the field of digital signal processing, the audio signal is first filtered through a downsampling filter to remove high-frequency components and aliasing interference in the audio signal to ensure that the downsampled audio signal does not lose necessary information; then, the downsampling filter is used to retain one sampling point at every certain sampling point in the filtered audio signal, thereby reducing the sampling rate of the audio signal.
[0121] Take the audio signal x(n) including 640 sample points as an example to illustrate that the downsampling filter is used to extract the low-frequency signal x from the audio signal x(n). LB (n), the downsampling filter adopts a downsampling filter with a factor of 1 / 2, the effective bandwidth of the low-frequency signal is 0-8kHz, and the number of sample points per frame is 320.
[0122] In step 102, low-frequency feature extraction processing is performed on the low-frequency signal to obtain low-frequency features of the audio signal.
[0123] Here, since low-frequency signals have a greater impact on audio coding than high-frequency signals, the first neural network model can be used to perform low-frequency feature extraction processing on the low-frequency signals to obtain low-frequency features. This can minimize the feature dimension of the low-frequency features while ensuring the completeness of the low-frequency features, thereby improving the audio coding effect. The low-frequency features are features that characterize the low-frequency signals, and the feature dimension of the low-frequency features is smaller than the feature dimension of the low-frequency signals.
[0124] It should be noted that the embodiments of the present application are not limited to the model structure of the first neural network. For example, the first neural network can be a deep neural network, a convolutional neural network, etc.
[0125] Refer to Figure 6B, which is a second flow chart of the audio encoding method provided in an embodiment of the present application. Figure 6B shows that step 102 in Figure 6A can be implemented through steps 1021-1024: in step 1021, the low-frequency signal is convolved to obtain the convolution feature of the low-frequency signal; in step 1022, the convolution feature is pooled to obtain the pooled feature of the low-frequency signal; in step 1023, the pooled feature is down-sampled to obtain the down-sampled feature of the low-frequency signal; in step 1024, the down-sampled feature is convolved to obtain the low-frequency feature of the audio signal.
[0126] In the field of audio codecs, neural network (NN) operations such as convolution, pooling, and downsampling play a crucial role in processing audio signals and extracting features. In audio codecs, convolution can be used to extract local features in audio signals. By applying a convolution kernel (a learnable filter), convolution can be performed on the time dimension of the audio signal to capture patterns and resonances. Convolution can extract both time-domain and frequency-domain features from audio signals, which can be used for tasks such as noise reduction, feature extraction, and signal separation. Pooling reduces the time dimension of audio signals, thereby reducing data complexity and computational complexity. Pooling samples a local region of the input signal and aggregates the information in that region, such as the maximum or average value, to produce a more compact feature representation. In audio signals, pooling can help improve network robustness and generalization, reducing the risk of overfitting. Downsampling can be used to reduce the sampling rate of an audio signal, that is, to lower the frequency of the signal. Downsampling helps reduce data size, save storage and transmission overhead, and reduce complexity. Downsampling can be performed in the frequency or time domain to reduce the sampling rate or the dimensionality of the signal. In the field of audio codecs, operations such as convolution, pooling, and downsampling can be used to implement tasks such as feature extraction, encoding, and decoding of audio signals by constructing appropriate neural network structures. These operations help improve the efficiency and quality of audio signal processing, expanding the application scope of audio codec technology in fields such as audio processing, speech recognition, and music generation.
[0127] As shown in Figure 10, based on the low-frequency signal x LB (n) Call the first neural network model to generate a lower-dimensional feature vector F LB (n), namely low-frequency features. First, in step 1021, the input low-frequency signal x is processed by causal convolution 1001. LB (n) is convolved to obtain a 16×320 convolution feature. Then, the 24×320 convolution feature in step 1022 is pooled with a factor of 2 (i.e., preprocessing 1002) to obtain a 16×160 pooled feature. Next, in step 1023, the 24×160 pooled feature is downsampled by the encoding block to obtain a 256×1 downsampled feature. Finally, in step 1024, the 256×1 downsampled feature is convolved again through the causal convolution 1004 to obtain a 56-dimensional feature vector F LB (n).
[0128] Among them, the downsampling processing in step 1023 is implemented through multiple cascaded coding layers. Therefore, step 1023 can be implemented in the following manner: downsampling processing is performed on the pooling feature through the first coding layer in the multiple cascaded coding layers; the downsampling result of the first coding layer is output to the subsequent cascaded coding layer, and the downsampling processing and downsampling result output are continued through the subsequent cascaded coding layers until it is output to the last coding layer; the downsampling result output by the last coding layer is determined as the downsampling feature of the low-frequency signal.
[0129] As shown in Figure 10, four coding blocks (i.e., coding layers) with different downsampling factors (Down_factor) are cascaded to perform downsampling processing on the pooled features. That is, the 16×160 pooled features are first downsampled by the coding block 1003-1 with a downsampling factor (Down_factor=2) to obtain a downsampling result of 32×80. Then, the 32×80 downsampling result is downsampled by the coding block 1003-2 with a downsampling factor (Down_factor=4) to obtain a downsampling result of 64×20. Then, the 64×20 downsampling result is downsampled by the coding block 1003-3 with a downsampling factor (Down_factor=4) to obtain a downsampling result of 128×5. Finally, the 128×5 downsampling result is downsampled by the coding block 1003-4 with a downsampling factor (Down_factor=5) to obtain a downsampling feature of 256×1. Taking the coding block (Down_factor=4) as an example, one or more dilated convolutions may be performed first, and then pooling may be performed based on the Down_factor to achieve downsampling.
[0130] It should be noted that after processing through one coding layer, the understanding of downsampling features is further deepened. After learning through multiple coding layers, the downsampling features of low-frequency signals can be gradually and accurately learned. By cascading coding layers, the downsampling features of low-frequency signals can be obtained with gradually increasing accuracy.
[0131] In step 103, high-frequency analysis processing is performed on the audio signal to obtain high-frequency features of the audio signal.
[0132] The feature dimension of high-frequency features is lower than that of low-frequency features. For example, the feature dimension of low-frequency features is 56, while the feature dimension of high-frequency features is 8. Because low-frequency signals have a greater impact on audio coding than high-frequency signals, differentiated signal processing is used to extract the low-frequency and high-frequency features of the audio signal, respectively, so that the feature dimension of the high-frequency features is lower than that of the low-frequency features. High-frequency analysis processing is used to reduce the dimensionality of high-frequency signals in audio signals, thereby achieving data compression.
[0133] 6C , which is a third flow chart of the audio encoding method provided in an embodiment of the present application, and FIG6C shows that step 103 in FIG6A can be implemented through step 1031 : in step 1031 , frequency band extension processing is performed on the audio signal to obtain high-frequency features of the audio signal.
[0134] Here, in step 1031 , the high-frequency signal in the audio signal is quickly compressed to extract the high-frequency features.
[0135] 6D , which is a fourth flow chart of the audio encoding method provided in an embodiment of the present application, shows that step 1031 in FIG. 6C can be implemented through steps 10311 to 10314:
[0136] In step 10311, frequency domain transform processing is performed on the multiple second sample points included in the audio signal to obtain transform coefficients corresponding to the multiple second sample points.
[0137] It should be noted that the frequency domain transformation method (that is, the time-frequency transformation method) in step 10311 can be a modified discrete cosine transform (MDCT), a discrete cosine transform (DCT), a fast Fourier transform (FFT), etc. The embodiment of the present application is not limited to the frequency domain transformation method.
[0138] In some embodiments, step 10311 can be implemented by: obtaining multiple third sample points included in a reference audio signal, where the reference audio signal is an audio signal adjacent to the audio signal; based on the multiple third sample points included in the reference audio signal and the multiple second sample points included in the audio signal, performing discrete cosine transform processing on the multiple second sample points included in the audio signal to obtain transform coefficients corresponding to the multiple second sample points respectively.
[0139] As an example of step 10311, for an audio signal x(n) including 640 second sample points, a modified discrete cosine transform (MDCT) method is invoked to generate MDCT coefficients for the 640 second sample points (i.e., transform coefficients corresponding to each of the plurality of second sample points). For example, if the modified discrete cosine transform method uses 50% overlap, the n+1th frame audio signal (i.e., the reference audio signal) and the nth frame audio signal (i.e., the audio signal) can be merged (concatenated) to calculate an MDCT coefficient for 1280 sample points. The valid MDCT coefficients for the first 640 sample points are used as transform coefficients corresponding to each of the plurality of second sample points.
[0140] In step 10312, high-frequency transform coefficients in the transform coefficients corresponding to the plurality of second sample points are divided into a plurality of sub-bands.
[0141] Following the example of step 10311, step 10312 is specifically as follows: the MDCT coefficients of the last 320 second sample points of the 640 second sample points (i.e., the transform coefficients corresponding to the multiple second sample points) represent the high-frequency signal, that is, the high-frequency transform coefficient. The high-frequency transform coefficient is divided into N sub-bands, where the sub-band is a group of multiple adjacent MDCT coefficients. The MDCT coefficients of the 320 second sample points can be divided into 8 sub-bands. For example, the MDCT coefficients of the 320 second sample points can be evenly distributed, that is, the number of sample points included in each sub-band is the same, then each sub-band includes 40 MDCT coefficients of the second sample points. Of course, the embodiment of the present application does not limit the non-uniform division of the 320 second sample points. For example, the low-frequency sub-band includes fewer MDCT coefficients (higher frequency resolution), and the high-frequency sub-band includes more MDCT coefficients (lower frequency resolution).
[0142] According to Nyquist's sampling theorem (to recover the original signal from a sampled signal without distortion, the sampling frequency must be greater than twice the original signal's highest frequency. When the sampling frequency is less than twice the highest frequency, the signal spectrum exhibits aliasing, while when the sampling frequency is greater than twice the highest frequency, the signal spectrum exhibits no aliasing), the MDCT coefficients of the aforementioned 320 second sample points (referred to as points) represent a spectrum between 8 and 16 kHz. However, ultra-wideband voice communications do not necessarily require a spectrum of 16 kHz. For example, if the spectrum is set to 14 kHz, only the MDCT coefficients of the first 240 second sample points need to be considered, and the number of subbands can be controlled to 6 accordingly.
[0143] In step 10313, the transform coefficients included in each sub-band are averaged to obtain the average energy corresponding to each sub-band, and the average energy is used as the sub-band spectrum envelope corresponding to each sub-band.
[0144] It should be noted that the averaging processing in step 10313 can be arithmetic mean or geometric mean, and the embodiment of the present application is not limited to the averaging processing method.
[0145] Taking the geometric mean as an example, step 10313 can be implemented in the following manner: determine the sum of the squares of the transformation coefficients corresponding to the second sample points included in each sub-band; and determine the ratio of the sum of the squares to the number of second sample points included in the sub-band as the average energy corresponding to each sub-band.
[0146] Following the example of step 10312, step 10313 is specifically as follows: for each sub-band, calculate the average energy of all MDCT coefficients in the current sub-band as the sub-band spectrum envelope (the sub-band spectrum envelope is a smooth curve passing through the main peak points of the spectrum). For example, the MDCT coefficients included in the current sub-band are x(n), n = 1, 2, ..., 40, then the average energy corresponding to the current sub-band Y = ((x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40).
[0147] In step 10314, sub-band spectrum envelopes corresponding to the plurality of sub-bands are determined as high-frequency features of the audio signal.
[0148] Following the example of step 10313, step 10314 is specifically as follows: when the 320-point MDCT coefficients are divided into 8 sub-bands, 8 sub-band spectral envelopes can be obtained. These 8 sub-band spectral envelopes are the high-frequency features F of the audio signal. HB (n), which is the high-frequency eigenvector.
[0149] In step 104, the low-frequency features are encoded to obtain a low-frequency code stream of the audio signal, and the high-frequency features are encoded to obtain a high-frequency code stream of the audio signal.
[0150] Here, in the field of digital signal processing, low-frequency features are coded based on digital signals to obtain a low-frequency code stream of the audio signal, and high-frequency features are coded based on digital signals to obtain a high-frequency code stream of the audio signal.
[0151] In some embodiments, step 104 can be implemented by: quantizing the low-frequency features to obtain the index value of the low-frequency features; performing entropy coding on the index value of the low-frequency features to obtain the low-frequency code stream of the audio signal; quantizing the high-frequency features to obtain the index value of the high-frequency features; performing entropy coding on the index value of the high-frequency features to obtain the high-frequency code stream of the audio signal.
[0152] As an example of step 104, for the low-frequency feature F LB (n) and high-frequency features F HB (n), both scalar quantization (each component is quantized separately) and entropy coding methods can be used. In addition, the embodiments of the present application do not limit the technical combination of vector quantization (combining multiple adjacent components into a vector for joint quantization) and entropy coding. The high-frequency code stream and the low-frequency code stream obtained by encoding are combined and transmitted to the decoding end, and the high-frequency code stream and the low-frequency code stream are decoded by the decoding end.
[0153] In summary, the audio encoding method provided in the embodiment of the present application downsamples the audio signal to obtain a low-frequency signal. Since the low-frequency signal has a greater impact on audio encoding than the high-frequency signal in the audio signal, the low-frequency features and high-frequency features of the audio signal are extracted separately through differentiated signal processing, so that the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature, and the low-frequency features and high-frequency features with reduced feature dimensions are encoded separately, thereby improving the audio encoding efficiency while ensuring the audio quality.
[0154] As previously mentioned, the audio decoding method provided in the embodiments of the present application can be implemented by various types of electronic devices. Referring to FIG. 7A , FIG. 7A is a schematic diagram of a first flow chart of the audio decoding method provided in the embodiments of the present application. The audio decoding function is implemented by the audio decoding method, and the steps shown in FIG. 7A are described below.
[0155] In step 201, the low-frequency code stream of the audio signal is decoded to obtain low-frequency features corresponding to the low-frequency code stream, and the high-frequency code stream of the audio signal is decoded to obtain high-frequency features corresponding to the high-frequency code stream.
[0156] The low-frequency code stream is obtained by encoding the low-frequency signal obtained by downsampling the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature.
[0157] For example, after encoding a high-frequency code stream and a low-frequency code stream using the audio encoding method shown in Figure 6A, the encoded high-frequency code stream and low-frequency code stream are merged and transmitted to the decoding end. After the decoding end receives the high-frequency code stream and the low-frequency code stream, it decodes the low-frequency code stream to obtain the low-frequency features corresponding to the low-frequency code stream, and decodes the high-frequency code stream to obtain the high-frequency features corresponding to the high-frequency code stream.
[0158] The decoding in step 201 is the inverse process of the encoding in step 104. In the field of digital signal processing, the low-frequency code stream of the audio signal is decoded based on a digital signal to obtain low-frequency features corresponding to the low-frequency code stream, and the high-frequency code stream of the audio signal is decoded based on a digital signal to obtain high-frequency features corresponding to the high-frequency code stream.
[0159] In some embodiments, step 201 can be implemented by performing entropy decoding on the low-frequency bitstream to obtain an index value corresponding to the low-frequency bitstream; performing inverse quantization on the index value corresponding to the low-frequency bitstream to obtain a low-frequency feature corresponding to the low-frequency bitstream; performing entropy decoding on the high-frequency bitstream to obtain an index value corresponding to the high-frequency bitstream; and performing inverse quantization on the index value corresponding to the high-frequency bitstream to obtain a high-frequency feature corresponding to the high-frequency bitstream. Inverse quantization is implemented by querying a quantization table, which is a mapping table generated by quantization during the encoding process.
[0160] As an example, for the received code stream (high-frequency code stream and low-frequency code stream), entropy decoding is first performed to obtain the index value corresponding to the code stream, and the quantization table is queried by the index value corresponding to the code stream, and the queried feature vector is used as the feature vector corresponding to the code stream, that is, the estimated value F′ of the low-frequency feature vector is obtained. LB (n) (i.e., the low-frequency features corresponding to the low-frequency code stream) and the estimated value F′ of the high-frequency feature vector HB (n) (i.e., the high-frequency features corresponding to the high-frequency bitstream). It should be noted that the decoding process of the received bitstream by the decoding end is the inverse process of the encoding process by the encoding end. Therefore, the values generated during the decoding process are estimates relative to the values during the encoding process. For example, the high-frequency features generated during the decoding process are estimates relative to the high-frequency features during the encoding process.
[0161] In step 202, low-frequency feature reconstruction processing is performed on the low-frequency feature to obtain a low-frequency signal corresponding to the low-frequency feature.
[0162] For example, the low-frequency feature reconstruction in step 202 is the inverse process of the low-frequency feature extraction in step 102. The low-frequency feature is reconstructed by the second neural network model to obtain a low-frequency signal corresponding to the low-frequency feature (an estimated value of the low-frequency signal obtained in the corresponding encoding process).
[0163] It should be noted that the embodiments of the present application are not limited to the model structure of the second neural network. For example, the second neural network can be a deep neural network, a convolutional neural network, etc.
[0164] Refer to Figure 7B, which is a second flow chart of the audio decoding method provided in an embodiment of the present application. Figure 7B shows that step 202 in Figure 7A can be implemented through steps 2021-2024: in step 2021, the low-frequency features are convolved to obtain the convolution features of the low-frequency features; in step 2022, the convolution features are upsampled to obtain the upsampled features of the low-frequency features; in step 2023, the upsampled features are pooled to obtain the pooled features of the low-frequency features; in step 2024, the pooled features are convolved to obtain the low-frequency signals corresponding to the low-frequency features.
[0165] In the field of audio codecs, upsampling is used to increase the resolution of feature maps to more accurately reconstruct audio signals. Upsampling involves interpolation or other forms of upsampling techniques to generate higher-precision feature maps, which helps better restore the original details and characteristics of the audio signal during decoding. By using neural network techniques such as convolution, pooling, and upsampling in audio decoding, useful features can be effectively extracted, computational complexity can be reduced, and the original content of the audio signal can be more accurately reconstructed. These techniques are of great significance for improving the performance and efficiency of audio decoding and help promote the development and application of audio codec technology.
[0166] As shown in Figure 11, based on the low-frequency feature F′ LB (n), call the neural network model shown in Figure 11 to generate the low-frequency signal x′ LB (n). The neural network model shown in Figure 11 is similar to the neural network model shown in Figure 10, such as the causal convolution and post-processing structure are similar to the pre-processing. The structure of the decoding block is symmetrical to the encoding block on the encoding side: the encoding block on the encoding side first performs dilated convolution and then pooling to complete downsampling, while the decoding block on the decoding side first performs pooling to complete upsampling and then performs dilated convolution.
[0167] First, in step 2021, the low-frequency feature F′ of the input is processed by causal convolution 1101. LB(n) is convolved to obtain a 256×1 convolution feature. Then, in step 2022, the 256×1 convolution feature is upsampled to obtain a 16×160 upsampled feature. Next, in step 2023, the 16×160 upsampled feature is pooled (i.e., post-processing 1103) to obtain a 16×320 pooled feature. Finally, in step 2024, the 16×320 pooled feature is convolved again through causal convolution 1104 to obtain a 320-dimensional low-frequency signal x′ LB (n).
[0168] Among them, the upsampling processing in step 2022 is implemented through multiple cascaded decoding layers. Therefore, step 2022 can be implemented in the following manner: upsampling the convolution feature through the first decoding layer in the multiple cascaded decoding layers; outputting the upsampling result of the first decoding layer to the subsequent cascaded decoding layer, and continuing the upsampling processing and upsampling result output through the subsequent cascaded decoding layer until it is output to the last decoding layer; determining the upsampling result output by the last decoding layer as the upsampling feature of the low-frequency feature.
[0169] As an example of step 2022, as shown in Figure 11, four decoding blocks (i.e., decoding layers) with different upsampling factors (Up_factor) are cascaded to upsample the convolution features. That is, the 256×1 convolution features are first upsampled by the decoding block 1102-1 with an upsampling factor (Up_factor=5) to obtain a 128×5 upsampled result. Then, the 128×5 upsampled result is upsampled by the decoding block 1102-2 with an upsampling factor (Up_factor=4) to obtain a 64×20 upsampled result. Then, the 64×20 upsampled result is upsampled by the decoding block 1102-3 with an upsampling factor (Up_factor=4) to obtain a 32×80 upsampled result. Finally, the 32×80 upsampled result is upsampled by the decoding block 1102-4 with an upsampling factor (Up_factor=2) to obtain a 16×160 upsampled feature. Taking the decoding block (Up_factor=4) as an example, pooling can be performed based on the Up_factor first, and then one or more dilated convolutions can be performed to achieve upsampling.
[0170] It should be noted that after processing through a decoding layer, the understanding of upsampled features is further deepened. After learning through multiple decoding layers, the upsampled features of low-frequency features can be gradually and accurately learned. By cascading decoding layers, it is possible to obtain upsampled features of low-frequency features with gradually increasing accuracy.
[0171] In step 203, up-sampling is performed on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal.
[0172] For example, taking the sampling rate of the encoding end as 1 / 2, the low-frequency signal x′ obtained at the decoding end is LB The sampling rate of the audio signal x(n) is only 1 / 2 of the sampling rate of the audio signal x(n) at the encoding end. Therefore, the low-frequency signal x′ needs to be LB (n) Perform upsampling, increase the sampling rate by 1 times, and generate the upsampled signal x up (n), thereby ensuring that the sampling rate of the upsampled signal is the same as the sampling rate of the audio signal.
[0173] In some embodiments, step 203 may be implemented in the following manner: performing upsampling processing on the fourth sampling point included in the low-frequency signal through an upsampling filter to obtain an upsampled signal of the low-frequency signal.
[0174] In the field of digital signal processing, an upsampling filter is used to increase the sampling rate of a signal input to the upsampling filter to better capture detailed information in the signal or expand the spectrum range. The upsampling factor of the upsampling filter can be a multiple of 2, such as 2, 4, 8, etc.
[0175] Here, the upsampling filter performs the following processing: performing upsampling processing based on the digital signal on the fourth sampling point included in the low-frequency signal to obtain an upsampled low-frequency signal; and performing filtering processing based on the digital signal on the upsampled low-frequency signal to obtain an upsampled signal of the low-frequency signal. The filtering processing based on the digital signal can be low-pass filtering or band-pass filtering.
[0176] For example, in the field of digital signal processing, the fourth sampling point included in the low-frequency signal is first up-sampled based on the digital signal through an upsampling filter, where upsampling refers to inserting zero values into the low-frequency signal to increase the sampling rate; the up-sampled low-frequency signal is filtered based on the digital signal to remove aliasing interference introduced by interpolation.
[0177] With low frequency signal x′ LB (n) includes 320 sample points as an example, and the upsampling filter is used to convert the low-frequency signal x′ LB The sampling rate of (n) is doubled, and the upsampling filter uses an upsampling filter with a factor of 2 to convert the low-frequency signal x′ LB The sampling rate of (n) is doubled to generate the upsampled signal x up (n), where each frame x up (n) contains 640 sample points (referred to as points).
[0178] In step 204, signal reconstruction processing is performed on the high-frequency features and the up-sampled signal to obtain a synthesized audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0179] For signal reconstruction, a high-frequency signal is obtained based on the high-frequency features and the frequency band of the up-sampled signal, and a combined audio signal x′(n) is obtained by combining the up-sampled signal and the reconstructed high-frequency signal.
[0180] 7C , which is a third flow chart of the audio decoding method provided in an embodiment of the present application, shows that step 204 in FIG. 7A can be implemented through steps 2041 to 2043:
[0181] In step 2041, frequency domain transform processing is performed on the plurality of fifth sample points included in the up-sampled signal to obtain transform coefficients corresponding to the plurality of fifth sample points.
[0182] It should be noted that the frequency domain transformation method (that is, the time-frequency transformation method) in step 2041 can be a modified discrete cosine transform (MDCT), a discrete cosine transform (DCT), a fast Fourier transform (FFT), etc. The embodiment of the present application is not limited to the frequency domain transformation method.
[0183] In some embodiments, step 2041 can be implemented by: obtaining multiple sample points included in a reference up-sampled signal, wherein the reference up-sampled signal is an up-sampled signal adjacent to the up-sampled signal; based on the multiple sample points included in the reference up-sampled signal and the multiple fifth sample points included in the up-sampled signal, performing discrete cosine transform processing on the multiple fifth sample points included in the up-sampled signal to obtain transform coefficients corresponding to the multiple fifth sample points respectively.
[0184] As an example of step 2041, for an upsampled signal x comprising 640 sample points, up(n), calling the modified discrete cosine transform (MDCT) method to generate MDCT coefficients for 640 sample points (i.e., transform coefficients corresponding to multiple fifth sample points). Specifically, if the modified discrete cosine transform method is 50% overlap, the n+1 frame upsampled signal (i.e., the reference upsampled signal) and the n frame upsampled signal (i.e., the upsampled signal) can be merged (concatenated) to calculate the MDCT of 1280 sample points (including multiple sample points included in the reference upsampled signal and multiple fifth sample points included in the upsampled signal). Since the MDCT of the calculated 1280 sample points is in the form of an upsampled signal, the high-frequency content is missing. Therefore, the low-frequency content of the MDCT of the calculated 1280 sample points can be retained, that is, the valid MDCT coefficients of the first 640 sample points are used as the transform coefficients X corresponding to the multiple fifth sample points. up (n). .
[0185] In step 2042, frequency band extension processing is performed on the transform coefficients and high-frequency features corresponding to the plurality of fifth sample points to obtain high-frequency transform coefficients of the high-frequency signal.
[0186] Here, the high-frequency transform coefficients of the high-frequency signal are reconstructed based on the transform coefficients corresponding to the plurality of fifth sample points and the high-frequency features through the frequency band extension technology.
[0187] 7D , which is a fourth flow chart of the audio decoding method provided in an embodiment of the present application, shows that step 2042 in FIG. 7C can be implemented by following steps 20421 to 20422:
[0188] In step 20421, spectrum replication processing is performed on at least part of the transform coefficients in the first half of the transform coefficients corresponding to the plurality of fifth sample points to obtain reference high-frequency transform coefficients of the reference high-frequency signal.
[0189] In some embodiments, step 20421 can be implemented by performing spectrum replication processing on the first half and the second half transform coefficients of the transform coefficients corresponding to multiple fifth sample points to obtain reference high-frequency transform coefficients of the reference high-frequency signal.
[0190] As an example of step 20421, the transformation coefficients corresponding to the plurality of fifth sample points are X up (n) (including 640-point MDCT coefficients), X upThe low-frequency MDCT coefficients in (n) (i.e., the MDCT coefficients of the first 320 points) are copied to generate reference values of the MDCT coefficients of the high-frequency part, i.e., the reference high-frequency transform coefficients of the reference high-frequency signal. The basic characteristics of the reference speech signal are that the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid the high-frequency MDCT spectrum generated by simple copying containing too many harmonics, X up The last 160 points of the low-frequency MDCT coefficients in (n) (i.e., the second half of the first half of the transform coefficients corresponding to the plurality of fifth sample points) are used as a master, and are copied twice in the high-frequency part of the spectrum to generate reference values of the MDCT coefficients of the 320-point high-frequency signal (i.e., the reference high-frequency transform coefficients of the reference high-frequency signal). Therefore, after the spectrum is copied, X up The last 320 points in (n) are non-zero coefficients.
[0191] Of course, the embodiment of the present application is not limited to up The last 160 points of the low-frequency MDCT coefficients in (n) are used as the master, and X up The low-frequency MDCT coefficients in (n) are used as the master, and the spectrum is copied once to generate the reference values of the MDCT coefficients of 320 points of the high-frequency signal; X up The last 80 points of the low-frequency MDCT coefficients in (n) are used as a master, and the spectrum is replicated four times to generate reference values of the 320-point MDCT coefficients of the high-frequency signal.
[0192] In step 20422, based on the sub-band spectrum envelope corresponding to the high-frequency feature, the reference high-frequency transform coefficient of the reference high-frequency signal is gain-processed to obtain the high-frequency transform coefficient of the high-frequency signal.
[0193] In some embodiments, step 20422 can be implemented in the following manner: based on the sub-band spectral envelope corresponding to the high-frequency feature, the reference high-frequency transform coefficient of the reference high-frequency signal is divided into multiple sub-bands; for any sub-band among the multiple sub-bands, the following processing is performed: the first average energy of the high-frequency sub-band corresponding to the sub-band in the sub-band spectral envelope is determined, and the second average energy of the sub-band is determined; based on the ratio of the first average energy to the second average energy, a gain factor is determined; the gain factor is multiplied by each reference high-frequency transform coefficient included in the sub-band to obtain the high-frequency transform coefficient corresponding to the sub-band; and the high-frequency transform coefficients corresponding to the multiple sub-bands are determined as the high-frequency transform coefficients of the high-frequency signal.
[0194] Following the example of step 20421, step 20422 is specifically as follows: the sub-band spectrum envelope corresponding to the high-frequency feature (for example, 8 sub-band spectrum envelopes) is the estimated value F′ corresponding to the high-frequency feature obtained after decoding HB(n), these 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands, and the reference values of the MDCT coefficients of the 320 points of the generated high-frequency part signal (that is, the reference high-frequency transform coefficients of the reference high-frequency signal) are divided into 8 sub-bands, and based on a high-frequency sub-band and the corresponding sub-band, the reference values of the MDCT coefficients of the 320 points of the high-frequency part signal are gained (that is, multiplication is performed in the frequency domain). For example, the gain factor is calculated based on the average energy of the high-frequency sub-band and the average energy of the corresponding sub-band, and the MDCT coefficient corresponding to each point in the corresponding sub-band is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficient generated by decoding is close to the original coefficient energy at the encoding end.
[0195] For example, assuming the average energy of a subband (i.e., the subband divided by the reference values of the 320-point MDCT coefficients of the generated high-frequency signal) is Y_L, and the average energy of the current high-frequency subband (i.e., the subband corresponding to the subband spectral envelope decoded based on the bitstream) is Y_H, then a gain factor a = sqrt(Y_H / Y_L) is calculated. With the gain factor a, the MDCT coefficients of each point in the subband are directly multiplied by a. The average energy of the MDCT coefficients (i.e., the high-frequency transform coefficients of the high-frequency signal) after gain control is very close to the original average energy at the encoding end, so as to restore the original high-frequency signal as much as possible.
[0196] In step 2043, based on the high frequency transform coefficient and the transform coefficients corresponding to some of the fifth sample points among the plurality of fifth sample points, a synthesized audio signal corresponding to the low frequency code stream and the high frequency code stream is determined.
[0197] In some embodiments, step 2043 can be implemented by: merging the high-frequency transformation coefficients and the transformation coefficients corresponding to the first half of the fifth sample points in multiple fifth sample points to obtain complete transformation coefficients; performing inverse transformation of the spectrum transformation on the complete transformation coefficients to obtain a synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0198] Following the example of step 20422, step 2043 is specifically as follows: up The low-frequency MDCT coefficients in (n) (i.e., the MDCT coefficients of the first 320 points, i.e., the transform coefficients corresponding to some of the fifth sample points in the plurality of fifth sample points) and the MDCT coefficients after gain control (i.e., X upThe reference values of the MDCT coefficients of the last 320 points after the gain in (n), that is, the high-frequency transform coefficients, are merged to obtain the complete transform coefficients (that is, the MDCT coefficients of 640 points, including the low-frequency MDCT coefficients and the high-frequency MDCT coefficients). The complete transform coefficients are subjected to an inverse MDCT transform (that is, the inverse transform of the spectrum transform) to generate 1280-point MDCT estimated values. Through overlapping, the effective MDCT estimated values of the first 640 points are taken as the synthetic audio signal.
[0199] In summary, the audio decoding method provided in the embodiment of the present application decodes the received code stream (i.e., the low-frequency code stream and the high-frequency code stream) to obtain low-frequency features and high-frequency features, and calls the inverse process of the corresponding encoding end for the low-frequency feature vector to complete the reconstruction of the low-frequency part. Then, the reconstructed low-frequency part (i.e., the low-frequency signal) is restored to the same sampling rate as the original input audio signal through upsampling filtering; by combining the low-frequency part and the high-frequency features after upsampling filtering, the high-frequency transform coefficient of the high-frequency signal is reconstructed, and the reconstructed low-frequency part and the high-frequency transform coefficient are merged to ensure that the energy of the reconstructed high-frequency signal is close to the energy of the high-frequency signal at the encoding end. In this way, the frequency domain coefficients from low frequency to high frequency are completely reconstructed.
[0200] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.
[0201] The embodiments of the present application can be applied to various audio scenarios, such as voice calls, instant messaging, etc. The following description will be made using a voice call as an example:
[0202] In related technologies, the principles of speech coding are roughly as follows: speech coding can directly encode speech waveform samples sample by sample; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.
[0203] The above coding principles are all derived from voice signal modeling, i.e., compression methods based on signal processing. In order to improve coding efficiency while ensuring voice quality relative to compression methods based on signal processing. The embodiment of the present application provides a low-rate neural network coding and decoding method (i.e., audio processing method), which, based on the characteristics of the voice signal, separates the most important low-frequency part (i.e., low-frequency signal) by downsampling filtering of a voice signal of a specific sampling rate. For the important low-frequency part (i.e., low-frequency signal), after processing based on the neural network (NN, Neural Network) technology, a feature vector with a lower dimension than the input low-frequency signal is obtained. For the relatively unimportant part (high-frequency part, i.e., high-frequency signal) in the voice signal, fewer bits are used for encoding. For the encoding end, after processing the low-frequency part and the high-frequency part in the voice signal respectively, a feature vector with a lower dimension than the input voice signal (feature vector of low frequency and feature vector of high frequency) is obtained, and the feature vector is compressed and coded. For the decoding end, the received code stream (i.e., low-frequency code stream and high-frequency code stream) is decoded to obtain a feature vector of low frequency and a feature vector of high frequency, and the inverse process of the corresponding encoding end is called for the feature vector of low frequency to complete the reconstruction of the low-frequency part. Then, the reconstructed low-frequency part is restored to the same sampling rate as the original input speech signal through upsampling filtering; the high-frequency part is reconstructed by combining the feature vectors of the low-frequency part and the high-frequency part after upsampling filtering, and the reconstructed low-frequency part and high-frequency part are merged to complete the decoding.
[0204] The embodiment of the present application can be applied to the voice communication link shown in Figure 8. Taking the Voice over Internet Protocol (VoIP) conference system based on the Internet Protocol as an example, the voice codec technology involved in the embodiment of the present application is deployed in the encoding and decoding parts to solve the basic function of voice compression. The encoder is deployed on the uplink client 801, and the decoder is deployed on the downlink client 802. The voice is collected through the uplink client and pre-processed, enhanced, encoded, etc. The encoded code stream is transmitted to the downlink client 802 through the network. The downlink client 802 performs decoding, enhancement, and other processing to play back the decoded voice on the downlink client 802.
[0205] To ensure forward compatibility (i.e., compatibility between the new encoder and the existing encoder), a transcoder must be deployed in the system's backend (i.e., server) to ensure interoperability between the new encoder and the existing encoder. For example, if the transmitter (uplink client) is the new NN encoder and the receiver (downlink client) is the Public Switched Telephone Network (PSTN) (G.722), the backend must execute the NN decoder to generate the voice signal and then call the G.722 encoder to generate a specific bitstream to implement the transcoding function. This allows the receiver to correctly decode the specific bitstream.
[0206] The following describes the low-rate neural network encoding and decoding method (implemented by an audio encoding method and an audio decoding method) provided in an embodiment of the present application in conjunction with FIG9 :
[0207] The following processing is performed on the encoding side:
[0208] For the input speech signal x(n) of the nth frame, a downsampling filter is used to extract the low-frequency signal x of the speech signal x(n). LB (n), referred to as low-frequency signal.
[0209] For the low-frequency signal x LB (n), call the first NN to obtain the low-dimensional feature vector F LB (n) (referred to as low-frequency features), feature vector F LB The dimension of (n) is smaller than that of the low-frequency subband signal to reduce the amount of data. For example, for each frame x LB (n), call the dilated CNN to generate a lower-dimensional feature vector F LB (n). The embodiments of the present application do not limit other NN structures, such as autoencoder (Autoencoder), fully connected (FC) network, long short-term memory (LSTM) network, convolutional neural network (CNN) + LSTM, etc.
[0210] At the same time, the high frequency analysis of the speech signal x(n) is performed to extract the high frequency feature vector F HB(n) (referred to as high-frequency features). Considering that high frequency is not as important to quality as low frequency, the embodiment of the present application is based on the frequency band expansion technology of speech signal analysis, which can realize the generation of high-frequency signals with only 1-2kbps. Among them, the implementation method of high-frequency analysis based on frequency band expansion includes: performing time-frequency transformation on the speech signal x(n) to obtain frequency domain coefficients; merging adjacent coefficients of the frequency domain coefficients into multiple sub-bands; calculating the average energy of each sub-band as the spectral envelope of the sub-band, here, only the frequency domain coefficients related to high frequency can be operated; extracting the spectral envelope related to high frequency as F HB (n).
[0211] The eigenvector corresponding to the subband (i.e. F LB (n) and F HB (n)) performs vector quantization or scalar quantization, and performs entropy coding on the quantized index value, and the encoded code stream is transmitted to the decoding end.
[0212] The following processing is performed on the decoding end:
[0213] Decode the code stream received by the decoding end and obtain the estimated value F′ of the low-frequency feature vector (referred to as low-frequency feature) LB (n) and the estimated value F′ of the high-frequency feature vector (referred to as high-frequency feature) HB (n).
[0214] For the low-frequency part, the estimated value F′ based on the low-frequency eigenvector LB (n) Call the second NN to generate an estimate x′ of the low-frequency signal LB (n).
[0215] The estimated value x′ of the generated low-frequency signal LB (n), perform upsampling filtering to generate x up (n), where x up The sampling rate of (n) is consistent with the sampling rate of the speech signal x(n) input at the encoding end.
[0216] For signal reconstruction, signal reconstruction mainly includes reconstruction of the high-frequency part based on frequency band expansion, and merging the low-frequency part signal and the reconstructed high-frequency signal to obtain the final output signal x′(n). First, for x up(n) is subjected to time-frequency transformation to obtain frequency domain coefficients. As mentioned above, the resolution of this frequency domain coefficient is consistent with the resolution of the input speech signal x(n). Second, since the frequency domain coefficients lack high-frequency components, high-frequency components (i.e., frequency domain coefficients of the high-frequency part) are generated based on the frequency band extension technology, and the high-frequency components are combined with the frequency domain coefficients of the low-frequency part to generate complete frequency domain coefficients. Third, the complete frequency domain coefficients are subjected to time-frequency inverse transformation to obtain the final output signal x′(n). Among them, the frequency band extension technology includes: copying the frequency domain coefficients of the low-frequency part to the high-frequency part; based on the obtained high-frequency feature vector F′ HB (n) The frequency domain coefficients of the high frequency part are energy-adjusted by subband to ensure that the energy of the reconstructed high frequency part is close to the high frequency energy at the encoding end. In this way, the frequency domain coefficients from low frequency to high frequency are completely reconstructed.
[0217] The low-rate neural network encoding method provided in the embodiment of the present application is described in detail below.
[0218] In some embodiments, a speech signal with a sampling rate of Fs = 32000 Hz is used as an example (it should be noted that the method provided in the embodiments of the present application is also applicable to scenarios with other sampling rates, including but not limited to: 8000 Hz, 16000 Hz, and 48000 Hz). At the same time, assuming that the frame length is set to 20 ms, therefore, for Fs = 32000 Hz, each frame contains 640 sample points.
[0219] The following describes the encoding end and the decoding end in detail with reference to the flowchart shown in FIG9 .
[0220] The process on the encoding side is as follows:
[0221] For a speech signal with a sampling rate of Fs=32000 Hz, the input signal of the nth frame includes 640 sample points (abbreviated as points), which are recorded as input signal x(n).
[0222] Step 11: Downsampling filtering.
[0223] The downsampling filter in the embodiment of the present application is to separate the low-frequency signal x from the original input signal x(n). LB (n) The embodiment of the present application adopts a downsampling filter with a factor of 1 / 2, so the effective bandwidth of the low-frequency signal is 0-8kHz, and the number of sample points per frame is 320.
[0224] Step 12: Convert the low frequency signal x LB (n) Input the first NN for data compression.
[0225] Based on the low-frequency signal x LB (n), call the first NN to generate a lower-dimensional feature vector F LB(n). It should be noted that x LB The dimension of (n) is 320, F LB The dimension of (n) is 56. From the perspective of data volume, the first NN plays the role of "dimensionality reduction" and realizes the function of data compression.
[0226] Referring to the network structure diagram of the first NN shown in FIG10 , the process of data compression performed by the first NN is described in detail below:
[0227] First, a 16-channel causal convolution 1001 is called to transform the input tensor (i.e., the low-frequency signal x LB (n)), expanded to a 16×320 tensor.
[0228] Then, the 16×320 tensor is preprocessed 1002. For example, a pooling operation with a factor of 2 is performed on the 16×320 tensor, and the activation function may be ReLU to generate a 16×160 tensor.
[0229] Next, cascade four coding blocks with different downsampling factors (Down_factor) (i.e., coding block 1003-1, coding block 1003-2, coding block 1003-3, and coding block 1003-4). Taking the coding block (Down_factor = 2) as an example, one or more dilated convolutions can be performed first, and the size of each convolution kernel is fixed to 1×3, and the shift rate (Stride Rate) is 1. In addition, the dilation rate (Dilation Rate) of one or more dilated convolutions can be set according to actual needs, such as 3. Of course, the embodiment of the present application does not limit the dilation rate of different dilated convolution settings. Then, the Down_factor of the four coding blocks is set to 2, 4, 4, and 5, respectively, which is equivalent to setting pooling factors of different sizes, which plays a role of downsampling. Finally, the number of channels of the four coding blocks is set to 32, 64, 128, and 256, respectively. Therefore, after 4 coding blocks, the 16×160 tensor is converted into 32×80, 64×20, 128×5 and 256×1 tensors respectively. It should be noted that the embodiments of the present application do not limit other implementations of the coding blocks, such as the number of dilated convolutions (including the expansion rate), the value of the downsampling factor in each coding block, and the number of output channels of each coding block, as long as the above configuration is ensured and the dimensions of the first NN are matched, such as 1×320 input and 56×1 output.
[0230] Finally, the 256×1 tensor is subjected to similar preprocessing causal convolution 1004 to output a 56-dimensional feature vector F LB (n).
[0231] Step 13: Perform high-frequency analysis on the input signal x(n).
[0232] The purpose of high-frequency analysis is to analyze the input signal x(n), extract key information related to the high-frequency signal, and generate a lower-dimensional feature vector F HB (n).
[0233] The embodiment of the present application proposes to perform high-frequency analysis on the input signal based on frequency band expansion (recovering a wideband speech signal from a band-limited narrowband speech signal). The application of frequency band expansion in the embodiment of the present application is described in detail below:
[0234] For an input signal x(n) comprising 640 points, a modified discrete cosine transform (MDCT) is invoked to generate 640-point MDCT coefficients. For example, if the modified discrete cosine transform method uses 50% overlap, the n+1th frame of input signal data can be merged (concatenated) with the nth frame of input signal data to calculate the 1280-point MDCT and obtain 640-point MDCT coefficients.
[0235] For high-frequency analysis, the embodiment of the present application uses the MDCT coefficients of the last 320 points of the 640-point MDCT coefficients to represent the high-frequency signal. The MDCT coefficients of the 320 points (i.e., the high-frequency signal) are divided into N sub-bands. The sub-bands here are formed into a group of multiple adjacent MDCT coefficients. The MDCT coefficients of the 320 points can be divided into 8 sub-bands. For example, the 320 points can be evenly distributed, that is, the number of points included in each sub-band is consistent. Of course, the embodiment of the present application does not limit the 320 points to be unevenly divided. For example, the sub-bands with a lower frequency may include fewer MDCT coefficients (higher frequency resolution), while the sub-bands with a higher frequency may include more MDCT coefficients (lower frequency resolution).
[0236] According to Nyquist's sampling theorem (to recover the original signal from a sampled signal without distortion, the sampling frequency must be greater than twice the original signal's highest frequency. If the sampling frequency is less than twice the highest frequency, the signal spectrum will exhibit aliasing, while if the sampling frequency is greater than twice the highest frequency, the signal spectrum will exhibit no aliasing), the 320-point MDCT coefficients above represent a spectrum between 8 and 16 kHz. However, ultra-wideband voice communication does not necessarily require a spectrum extending to 16 kHz. For example, if the spectrum is set to 14 kHz, only the MDCT coefficients of the first 240 points need to be considered, and the number of subbands can be controlled to 6.
[0237] For each sub-band, calculate the average energy of all MDCT coefficients in the current sub-band as the sub-band spectrum envelope (the sub-band spectrum envelope is a smooth curve passing through the main peak points of the spectrum). For example, the MDCT coefficients included in the current sub-band are x(n), n = 1, 2, ..., 40, then the average energy Y = ((x(1)2 +x(2) 2 +…+x(40) 2 ) / 40). When the 320-point MDCT coefficients are divided into 8 sub-bands, 8 sub-band spectral envelopes can be obtained. These 8 sub-band spectral envelopes are the eigenvectors F of the generated high-frequency signal. HB (n).
[0238] In summary, the high-frequency analysis outputs an 8-dimensional feature vector that represents the key features of the high-frequency portion of the input signal. Therefore, only a small amount of data is needed to represent high-frequency information, significantly improving coding efficiency.
[0239] Step 14: quantization encoding.
[0240] For the eigenvector F of the low-frequency signal LB (n) and the eigenvector F of the high-frequency signal HB (n), both scalar quantization (each component is quantized separately) and entropy coding methods can be used. In addition, the embodiments of the present application do not limit the technical combination of vector quantization (combining multiple adjacent components into a vector for joint quantization) and entropy coding.
[0241] After quantizing and encoding the feature vectors, the corresponding bitstream can be generated. According to experiments, high-quality compression of 32kHz ultra-wideband signals can be achieved at a bitrate of 6-10kbps.
[0242] Therefore, the embodiment of the present application extracts a low-frequency signal by downsampling and filtering the input signal. Since the low-frequency signal has a greater impact on audio coding than the high-frequency signal in the audio signal, the low-frequency features and high-frequency features of the input signal are extracted separately through differentiated signal processing, so that the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature, and the low-frequency features and high-frequency features with reduced feature dimensions are encoded separately, thereby improving the audio coding efficiency while ensuring the audio quality.
[0243] The low-rate neural network encoding method provided in the embodiment of the present application is described in detail below.
[0244] Continuing with the above example, the decoding process is as follows:
[0245] Step 21: quantization decoding.
[0246] Quantization decoding is the inverse process of quantization encoding. For the received bitstream, entropy decoding is first performed to obtain the index value corresponding to the bitstream, and the quantization table is searched by the index value corresponding to the bitstream to obtain the estimated value F′ of the low-frequency feature vector LB (n) and the estimated value F′ of the high-frequency eigenvector HB (n).
[0247] Step 22: The estimated value F′ of the low-frequency eigenvector LB (n) Input the second NN.
[0248] The estimated value F′ based on the low-frequency eigenvector LB (n), call the second NN shown in Figure 11 to generate the estimated value x′ of the low-frequency signal LB (n).
[0249] The second NN is similar to the first, including causal convolution and post-processing similar to the pre-processing in the first. The decoding block is symmetrical to the encoding block: the encoding block performs dilated convolution followed by pooling for downsampling, while the decoding block performs pooling for upsampling followed by dilated convolution.
[0250] Step 23: Estimation of the low-frequency signal x′ LB (n) Perform upsampling filtering.
[0251] It should be noted that the estimated value x′ of the generated low-frequency signal LB The sampling rate of (n) is only 1 / 2 of the sampling rate of the input signal x(n) at the encoder. Therefore, it is necessary to estimate the low-frequency signal x′ LB (n) Perform upsampling filtering to double the sampling rate and generate an upsampled signal x up (n), where each frame x up (n) contains 640 points. The upsampling filtering in the embodiment of the present application is equivalent to an upsampling operation with a factor of 2.
[0252] Step 24: Estimation of the upsampled signal and the high-frequency eigenvector F′ HB (n) Perform signal reconstruction.
[0253] The first step of signal reconstruction is to obtain the upsampled signal x up (n) Performing a modified discrete cosine transform (MDCT) similar to the encoding configuration, that is, calculating the MDCT of 1280 points and obtaining the MDCT coefficients of 640 points, namely X up (n). For example, for an upsampled signal x consisting of 640 points up (n), call the modified discrete cosine transform (MDCT, Modified Discrete Cosine Transform), and compare the upsampled signal of the previous frame with the upsampled signal of this frame x up (n) Merge, calculate MDCT of 1280 points, and obtain MDCT coefficients of 640 points. up(n) is an upsampled signal form, so X up (n) Only the 0-8kHz content is retained, and the 8-16kHz content is missing.
[0254] The second step of signal reconstruction is based on the estimated value F′ of the high-frequency eigenvector obtained HB (n), call the frequency band extension technology to generate the missing high-frequency components. HB (n) is the 8 high-frequency sub-band spectrum envelopes decoded from the bitstream. The specific operations of the second step of signal reconstruction are as follows:
[0255] X up The low-frequency MDCT coefficients in (n) (i.e., the MDCT coefficients of the first 320 points) are copied to generate reference values of the MDCT coefficients of the high-frequency part. Referring to the basic characteristics of the speech signal, the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid the artificial high-frequency MDCT spectrum containing too many harmonics due to simple copying, X up The last 160 points of the low-frequency MDCT coefficients in (n) are used as the master, and the spectrum is copied twice to generate the reference values of the 320-point MDCT coefficients of the high-frequency signal (referred to as the reference values of the 320-point high-frequency MDCT coefficients of the reference high-frequency signal). You can also use X up The low-frequency MDCT coefficients in (n) are used as the master, and the spectrum is copied once to generate the reference values of the MDCT coefficients of 320 points of the high-frequency signal; X up The last 80 points of the low-frequency MDCT coefficients in (n) are used as the master, and the spectrum is replicated 4 times to generate the reference value of the 320-point MDCT coefficients of the high-frequency signal. up The last 320 points in (n) are non-zero coefficients.
[0256] Next, call the 8 sub-band spectrum envelopes obtained previously (that is, the estimated value F′ of the high-frequency feature vector obtained by querying the quantization table) HB (n)), these 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands, and the reference values of the MDCT coefficients of the 320 points of the generated high-frequency part signal are divided into 8 reference high-frequency sub-bands. Based on a high-frequency sub-band and the corresponding reference high-frequency sub-band, the reference values of the MDCT coefficients of the 320 points of the high-frequency part signal are gain controlled (multiplication is performed in the frequency domain). For example, the gain factor is calculated based on the average energy of the high-frequency sub-band and the average energy of the corresponding reference high-frequency sub-band, and the MDCT coefficient corresponding to each point in the corresponding reference high-frequency sub-band is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to the original coefficient energy at the encoding end.
[0257] For example, assuming that the average energy of the reference high-frequency sub-band (i.e., the sub-band divided by the reference values of the 320-point MDCT coefficients of the generated high-frequency signal) is Y_L, and the average energy of the current high-frequency sub-band (i.e., the sub-band corresponding to the sub-band spectrum envelope decoded based on the bitstream) is Y_H, then a gain factor a = sqrt(Y_H / Y_L) is calculated, where sqrt() represents the square root calculation function, which is used to calculate the square root of (Y_H / Y_L). With the gain factor a, the MDCT coefficients of each point in the reference high-frequency sub-band are directly multiplied by a. The average energy of the MDCT coefficients (virtually generated) after gain control is very close to the original ones at the encoding end.
[0258] Finally, for X up The low-frequency MDCT coefficients in (n) (i.e., the MDCT coefficients of the first 320 points) and the MDCT coefficients after gain control (i.e., X up The reference values of the MDCT coefficients of the last 320 points after the gain in (n) are used for inverse MDCT transform to generate estimated values of the MDCT of 1280 points. By overlapping, the estimated values of the MDCT of the first 640 effective points are taken as the estimated values of the original input signal, that is, the output signal x′(n).
[0259] Therefore, the embodiment of the present application decodes the received code stream (i.e., the low-frequency code stream and the high-frequency code stream) to obtain the low-frequency feature vector and the high-frequency feature vector, and calls the inverse process of the corresponding encoding end for the low-frequency feature vector to complete the reconstruction of the low-frequency part. Then, the reconstructed low-frequency part is restored to the same sampling rate as the original input voice signal through upsampling filtering; the high-frequency part is reconstructed by combining the low-frequency part and the high-frequency feature vector after upsampling filtering, and the reconstructed low-frequency part and high-frequency part are merged to ensure that the energy of the reconstructed high-frequency part is close to the high-frequency energy of the encoding end. In this way, the frequency domain coefficients from low frequency to high frequency are completely reconstructed.
[0260] The embodiment of the present application can collect data and jointly train the relevant networks of the encoding and decoding ends to obtain optimal parameters. The user only needs to prepare the data and set the corresponding network structure. After the training is completed in the background, the trained model can be put into use.
[0261] In summary, the low-rate neural network encoding and decoding method provided in the embodiment of the present application, through the organic combination of signal decomposition, signal processing technology and deep neural network, significantly improves the encoding efficiency compared with the signal processing scheme while ensuring audio quality and acceptable complexity.
[0262] So far, the audio encoding method or audio decoding method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the terminal device provided by the embodiment of the present application. The embodiment of the present application also provides an audio encoding device and an audio decoding device. In actual applications, the functional modules in the audio encoding device and the audio decoding device can be implemented by the hardware resources of the electronic device (such as terminal equipment, server or server cluster), such as computing resources such as processors, communication resources (such as for supporting various communication modes such as optical cables and cellular), and memories. Figure 3A shows an audio encoding device 555 stored in the memory 550 and Figure 3B shows an audio decoding device 556 stored in the memory 550, which can be software in the form of programs and plug-ins, for example, software modules designed in programming languages such as C / C++ and Java, application software designed in programming languages such as C / C++ and Java, or dedicated software modules in large software systems, application program interfaces, plug-ins, cloud services, etc. The following examples illustrate different implementation methods.
[0263] The audio encoding device 555 includes a series of modules, including a downsampling block 5551, a low-frequency extraction module 5552, a high-frequency analysis module 5553, and an encoding module 5554. The following further describes how the various modules in the audio encoding device 555 provided in the embodiment of the present application cooperate to implement audio encoding.
[0264] The downsampling module 5551 is configured to perform downsampling processing on the audio signal to obtain a low-frequency signal of the audio signal; the low-frequency extraction module 5552 is configured to perform low-frequency feature extraction processing on the low-frequency signal to obtain the low-frequency features of the audio signal; the high-frequency analysis module 5553 is configured to perform high-frequency analysis processing on the audio signal to obtain the high-frequency features of the audio signal; wherein the feature dimension of the high-frequency features is lower than the feature dimension of the low-frequency features; the encoding module 5554 is configured to encode the low-frequency features to obtain the low-frequency code stream of the audio signal, and encode the high-frequency features to obtain the high-frequency code stream of the audio signal.
[0265] In some embodiments, the audio signal includes a plurality of first sample points obtained by sampling; the downsampling module 5551 is further configured to perform downsampling processing on the first sample points included in the audio signal through a downsampling filter to obtain a low-frequency signal of the audio signal.
[0266] In some embodiments, the downsampling module 5551 is further configured to perform the following processing through the downsampling filter: performing digital signal-based filtering processing on the sampling points included in the audio signal to obtain the filtered audio signal; performing digital signal-based downsampling processing on the filtered audio signal to obtain a low-frequency signal of the audio signal.
[0267] In some embodiments, the low-frequency extraction module 5552 is further configured to perform convolution processing on the low-frequency signal to obtain the convolution features of the low-frequency signal; perform pooling processing on the convolution features to obtain the pooling features of the low-frequency signal; perform down-sampling processing on the pooling features to obtain the down-sampling features of the low-frequency signal; and perform convolution processing on the down-sampling features to obtain the low-frequency features of the audio signal.
[0268] In some embodiments, the downsampling process is implemented through multiple cascaded coding layers; the low-frequency extraction module 5552 is also configured to perform downsampling processing on the pooling feature through the first coding layer among the multiple cascaded coding layers; output the downsampling result of the first coding layer to the subsequent cascaded coding layers, and continue to perform downsampling processing and output downsampling results through the subsequent cascaded coding layers until it is output to the last coding layer; determine the downsampling result output by the last coding layer as the downsampling feature of the low-frequency signal.
[0269] In some embodiments, the high frequency analysis module 5553 is further configured to perform frequency band expansion processing on the audio signal to obtain high frequency characteristics of the audio signal.
[0270] In some embodiments, the high-frequency analysis module 5553 is further configured to perform frequency domain transform processing on the multiple second sample points included in the audio signal to obtain transform coefficients corresponding to the multiple second sample points respectively; divide the high-frequency transform coefficients in the transform coefficients corresponding to the multiple second sample points respectively into multiple sub-bands; perform averaging processing on the transform coefficients included in each of the sub-bands to obtain the average energy corresponding to each of the sub-bands, and determine the average energy as the sub-band spectral envelope corresponding to each of the sub-bands; and determine the sub-band spectral envelopes corresponding to the multiple sub-bands as the high-frequency characteristics of the audio signal.
[0271] In some embodiments, the high-frequency analysis module 5553 is further configured to obtain multiple third sample points included in a reference audio signal, wherein the reference audio signal is an audio signal adjacent to the audio signal; based on the multiple third sample points included in the reference audio signal and the multiple second sample points included in the audio signal, discrete cosine transform processing is performed on the multiple second sample points included in the audio signal to obtain transformation coefficients corresponding to the multiple second sample points respectively.
[0272] In some embodiments, the high-frequency analysis module 5553 is further configured to determine the sum of the squares of the transformation coefficients corresponding to the second sample points included in each of the sub-bands; and determine the ratio of the sum of the squares to the number of second sample points included in the sub-band as the average energy corresponding to each of the sub-bands.
[0273] The audio decoding device 556 includes a series of modules, including a decoding module 5561, a low-frequency reconstruction module 5562, an upsampling module 5563, and a signal reconstruction module 5564. The following further describes the solution of how the various modules in the audio decoding device 555 provided in the embodiment of the present application cooperate to implement audio decoding.
[0274] The decoding module 5561 is configured to decode the low-frequency code stream of the audio signal to obtain the low-frequency features corresponding to the low-frequency code stream, and decode the high-frequency code stream of the audio signal to obtain the high-frequency features corresponding to the high-frequency code stream; wherein the low-frequency code stream is obtained by encoding the low-frequency signal obtained after downsampling the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature; the low-frequency reconstruction module 5562 is configured to perform low-frequency feature reconstruction on the low-frequency feature to obtain the low-frequency signal corresponding to the low-frequency feature; the upsampling module 5563 is configured to perform upsampling on the low-frequency signal to obtain the upsampled signal of the low-frequency signal; the signal reconstruction module 5564 is configured to perform signal reconstruction on the high-frequency features and the upsampled signal to obtain the synthesized audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0275] In some embodiments, the audio signal includes multiple fourth sample points obtained by sampling; the upsampling module 5563 is further configured to upsample the fourth sample points included in the low-frequency signal through an upsampling filter to obtain an upsampled signal of the low-frequency signal.
[0276] In some embodiments, the upsampling module 5563 is further configured to perform the following processing through the upsampling filter: performing upsampling processing based on a digital signal on the fourth sampling point included in the low-frequency signal to obtain the upsampled low-frequency signal; performing filtering processing based on a digital signal on the upsampled low-frequency signal to obtain an upsampled signal of the low-frequency signal.
[0277] In some embodiments, the low-frequency reconstruction module 5562 is further configured to perform convolution processing on the low-frequency features to obtain convolution features of the low-frequency features; perform upsampling processing on the convolution features to obtain upsampling features of the low-frequency features; perform pooling processing on the upsampling features to obtain pooling features of the low-frequency features; and perform convolution processing on the pooling features to obtain a low-frequency signal corresponding to the low-frequency features.
[0278] In some embodiments, the upsampling process is implemented through multiple cascaded decoding layers; the low-frequency reconstruction module 5562 is also configured to perform upsampling on the convolution feature through the first decoding layer among the multiple cascaded decoding layers; output the upsampling result of the first decoding layer to the subsequent cascaded decoding layer, and continue to perform upsampling and output upsampling results through the subsequent cascaded decoding layers until it is output to the last decoding layer; determine the upsampling result output by the last decoding layer as the upsampling feature of the low-frequency feature.
[0279] In some embodiments, the signal reconstruction module 5564 is further configured to perform frequency domain transform processing on the multiple fifth sample points included in the upsampled signal to obtain transform coefficients corresponding to the multiple fifth sample points respectively; perform frequency band expansion processing on the transform coefficients corresponding to the multiple fifth sample points and the high-frequency features respectively to obtain high-frequency transform coefficients of the high-frequency signal; and determine the synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream based on the high-frequency transform coefficients and the transform coefficients corresponding to some of the fifth sample points in the multiple fifth sample points.
[0280] In some embodiments, the signal reconstruction module 5564 is also configured to perform spectrum replication processing on at least part of the transform coefficients in the first half of the transform coefficients corresponding to the multiple fifth sample points to obtain reference high-frequency transform coefficients of the reference high-frequency signal; and perform gain processing on the reference high-frequency transform coefficients of the reference high-frequency signal based on the sub-band spectrum envelope corresponding to the high-frequency feature to obtain the high-frequency transform coefficients of the high-frequency signal.
[0281] In some embodiments, the signal reconstruction module 5564 is also configured to divide the reference high-frequency transform coefficient of the reference high-frequency signal into multiple sub-bands based on the sub-band spectral envelope corresponding to the high-frequency feature; perform the following processing for any of the multiple sub-bands: determine the first average energy of the high-frequency sub-band corresponding to the sub-band in the sub-band spectral envelope, and determine the second average energy of the sub-band; determine a gain factor based on the ratio of the first average energy to the second average energy; multiply the gain factor by each of the reference high-frequency transform coefficients included in the sub-band to obtain the high-frequency transform coefficient corresponding to the sub-band; and determine the high-frequency transform coefficients corresponding to the multiple sub-bands as the high-frequency transform coefficients of the high-frequency signal.
[0282] In some embodiments, the signal reconstruction module 5564 is further configured to perform spectrum replication processing on the first half and the second half of the transform coefficients corresponding to the multiple fifth sample points to obtain reference high-frequency transform coefficients of the reference high-frequency signal.
[0283] In some embodiments, the signal reconstruction module 5564 is further configured to merge the high-frequency transformation coefficients and the transformation coefficients corresponding to the first half of the fifth sample points in the multiple fifth sample points to obtain complete transformation coefficients; and perform inverse transformation of the spectrum transformation on the complete transformation coefficients to obtain a synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0284] The present invention provides a computer program product including a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the audio encoding method or audio decoding method described in the present invention.
[0285] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the audio encoding method or audio decoding method provided by the embodiment of the present application, for example, the audio encoding method shown in Figures 6A-6D, and the audio encoding method shown in Figures 7A-7D.
[0286] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.
[0287] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0288] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).
[0289] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.
[0290] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0291] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.
Claims
1. An audio encoding method, applied to an electronic device, the method comprising: Down-sampling the audio signal to obtain a low-frequency signal of the audio signal; Performing low-frequency feature extraction processing on the low-frequency signal to obtain the low-frequency features of the audio signal; Performing high-frequency analysis processing on the audio signal to obtain high-frequency features of the audio signal; Wherein, the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature; The low-frequency features are encoded to obtain a low-frequency code stream of the audio signal, and the high-frequency features are encoded to obtain a high-frequency code stream of the audio signal.
2. The method according to claim 1, wherein: The audio signal includes a plurality of first sample points obtained by sampling; The down-sampling process is performed on the audio signal to obtain a low-frequency signal of the audio signal, including: Down-sampling is performed on the first sampling point included in the audio signal through a down-sampling filter to obtain a low-frequency signal of the audio signal.
3. The method according to claim 2, wherein: The down-sampling process is performed on the first sampling point included in the audio signal by using a down-sampling filter to obtain a low-frequency signal of the audio signal, including: The following processing is performed by the downsampling filter: Performing a digital signal-based filtering process on the first sampling point included in the audio signal to obtain the filtered audio signal; The filtered audio signal is down-sampled based on a digital signal to obtain a low-frequency signal of the audio signal.
4. The method according to any one of claims 1 to 3, wherein: The step of performing low-frequency feature extraction processing on the low-frequency signal to obtain the low-frequency feature of the audio signal includes: Performing convolution processing on the low-frequency signal to obtain convolution features of the low-frequency signal; Performing pooling processing on the convolution features to obtain pooling features of the low-frequency signal; Downsampling the pooled features to obtain downsampled features of the low-frequency signal; The down-sampled features are convolved to obtain low-frequency features of the audio signal.
5. The method according to claim 4, wherein: The downsampling process is implemented by a plurality of cascaded coding layers; The downsampling process is performed on the pooling feature to obtain the downsampling feature of the low-frequency signal, including: Downsampling the pooled features through a first coding layer of the multiple cascaded coding layers; Outputting the downsampling result of the first coding layer to a subsequent cascaded coding layer, and continuing to perform downsampling processing and output downsampling results through the subsequent cascaded coding layers until outputting to the last coding layer; The down-sampling result output by the last coding layer is determined as the down-sampling feature of the low-frequency signal.
6. The method according to any one of claims 1 to 5, wherein: The performing high frequency analysis processing on the audio signal to obtain the high frequency characteristics of the audio signal includes: The audio signal is subjected to frequency band expansion processing to obtain high frequency characteristics of the audio signal.
7. The method according to claim 6, wherein: The performing frequency band extension processing on the audio signal to obtain the high frequency characteristics of the audio signal includes: Performing frequency domain transformation processing on a plurality of second sample points included in the audio signal to obtain transformation coefficients respectively corresponding to the plurality of second sample points; Dividing high-frequency transform coefficients among the transform coefficients corresponding to the plurality of second sample points respectively into a plurality of sub-bands; Performing averaging processing on the transform coefficients included in each of the sub-bands to obtain average energy corresponding to each of the sub-bands, and determining the average energy as a sub-band spectrum envelope corresponding to each of the sub-bands; Subband spectrum envelopes respectively corresponding to the multiple subbands are determined as high-frequency features of the audio signal.
8. The method according to claim 7, wherein: The performing frequency domain transform processing on the plurality of second sample points included in the audio signal to obtain transform coefficients respectively corresponding to the plurality of second sample points includes: Acquire a plurality of third sample points included in a reference audio signal, wherein the reference audio signal is an audio signal adjacent to the audio signal; Based on the multiple third sample points included in the reference audio signal and the multiple second sample points included in the audio signal, discrete cosine transform processing is performed on the multiple second sample points included in the audio signal to obtain transform coefficients respectively corresponding to the multiple second sample points.
9. The method according to claim 7, wherein: The averaging of the transform coefficients included in each of the sub-bands to obtain average energy corresponding to each of the sub-bands includes: Determine the sum of squares of transform coefficients corresponding to the second sample points included in each of the sub-bands; The ratio of the square sum to the number of second sample points included in the sub-band is determined as the average energy corresponding to each of the sub-bands.
10. An audio decoding method, applied to an electronic device, the method comprising: Decoding a low-frequency code stream of an audio signal to obtain a low-frequency feature corresponding to the low-frequency code stream, and decoding a high-frequency code stream of the audio signal to obtain a high-frequency feature corresponding to the high-frequency code stream; The low-frequency code stream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature; Performing low-frequency feature reconstruction processing on the low-frequency feature to obtain a low-frequency signal corresponding to the low-frequency feature; Performing up-sampling processing on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal; Signal reconstruction processing is performed on the high-frequency features and the up-sampled signal to obtain a synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
11. The method according to claim 10, wherein: The up-sampling process is performed on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal, including: An up-sampling process is performed on the fourth sampling point included in the low-frequency signal through an up-sampling filter to obtain an up-sampled signal of the low-frequency signal.
12. The method according to claim 11, wherein: The step of performing up-sampling processing on the fourth sampling point included in the low-frequency signal by using an up-sampling filter to obtain an up-sampled signal of the low-frequency signal comprises: The following processing is performed by the upsampling filter: Performing up-sampling processing on the fourth sampling point included in the low-frequency signal based on the digital signal to obtain the up-sampled low-frequency signal; The up-sampled low-frequency signal is subjected to filtering processing based on a digital signal to obtain an up-sampled signal of the low-frequency signal.
13. The method according to any one of claims 10 to 12, wherein: The performing low-frequency feature reconstruction processing on the low-frequency feature to obtain a low-frequency signal corresponding to the low-frequency feature includes: Performing convolution processing on the low-frequency features to obtain convolution features of the low-frequency features; Performing upsampling processing on the convolution feature to obtain upsampled features of the low-frequency feature; Performing pooling processing on the upsampled features to obtain pooled features of the low-frequency features; The pooled features are subjected to convolution processing to obtain low-frequency signals corresponding to the low-frequency features.
14. The method according to claim 13, wherein: The upsampling process is implemented by a plurality of cascaded decoding layers; The upsampling process is performed on the convolution feature to obtain the upsampled feature of the low-frequency feature, including: Upsampling the convolutional features through a first decoding layer in the plurality of cascaded decoding layers; Outputting the upsampling result of the first decoding layer to the subsequent cascaded decoding layers, and continuing the upsampling process and outputting the upsampling result through the subsequent cascaded decoding layers until outputting to the last decoding layer; The upsampling result output by the last decoding layer is determined as the upsampling feature of the low-frequency feature.
15. The method according to any one of claims 10 to 12, wherein: The performing signal reconstruction processing on the high frequency feature and the up-sampled signal to obtain a synthetic audio signal corresponding to the low frequency code stream and the high frequency code stream includes: Performing frequency domain transform processing on a plurality of fifth sample points included in the up-sampled signal to obtain transform coefficients respectively corresponding to the plurality of fifth sample points; Performing frequency band extension processing on the transform coefficients respectively corresponding to the plurality of fifth sample points and the high-frequency features to obtain high-frequency transform coefficients of the high-frequency signal; Based on the high-frequency transform coefficient and the transform coefficients respectively corresponding to some of the fifth sample points among the plurality of fifth sample points, a synthesized audio signal corresponding to the low-frequency code stream and the high-frequency code stream is determined.
16. The method according to claim 15, wherein: The performing frequency band extension processing on the transform coefficients respectively corresponding to the plurality of fifth sample points and the high frequency features to obtain high frequency transform coefficients of the high frequency signal includes: Performing spectrum replication processing on at least part of the transform coefficients in the first half of the transform coefficients respectively corresponding to the plurality of fifth sample points to obtain reference high-frequency transform coefficients of the reference high-frequency signal; Based on the sub-band spectrum envelope corresponding to the high-frequency feature, gain processing is performed on the reference high-frequency transform coefficient of the reference high-frequency signal to obtain the high-frequency transform coefficient of the high-frequency signal.
17. The method according to claim 16, wherein: The step of performing gain processing on the reference high-frequency transform coefficient of the reference high-frequency signal based on the sub-band spectrum envelope corresponding to the high-frequency feature to obtain the high-frequency transform coefficient of the high-frequency signal includes: Based on the sub-band spectrum envelope corresponding to the high-frequency feature, dividing the reference high-frequency transform coefficients of the reference high-frequency signal into a plurality of sub-bands; The following processing is performed for each of the plurality of sub-bands: determining a first average energy of a high frequency subband corresponding to the subband in the subband spectral envelope, and determining a second average energy of the subband; determining a gain factor based on a ratio of the first average energy to the second average energy; Multiplying the gain factor by each of the reference high-frequency transform coefficients included in the sub-band to obtain a high-frequency transform coefficient corresponding to the sub-band; The high-frequency transform coefficients respectively corresponding to the plurality of sub-bands are determined as the high-frequency transform coefficients of the high-frequency signal.
18. The method according to claim 16, wherein: The performing spectrum replication processing on at least part of the transform coefficients in the first half of the transform coefficients respectively corresponding to the plurality of fifth sample points to obtain reference high-frequency transform coefficients of the reference high-frequency signal includes: A spectrum replication process is performed on the transform coefficients of the first half and the second half of the transform coefficients respectively corresponding to the plurality of fifth sample points to obtain reference high-frequency transform coefficients of the reference high-frequency signal.
19. The method according to claim 15, wherein: The determining, based on the high frequency transform coefficient and the transform coefficients respectively corresponding to some of the fifth sample points among the plurality of fifth sample points, a synthesized audio signal corresponding to the low frequency code stream and the high frequency code stream comprises: Merging the high-frequency transform coefficient and the transform coefficients corresponding to the first half of the fifth sample points among the plurality of fifth sample points to obtain a complete transform coefficient; The complete transformation coefficients are subjected to an inverse transformation of the spectrum transformation to obtain a synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
20. An audio encoding device, the device comprising: A down-sampling module, configured to perform down-sampling processing on the audio signal to obtain a low-frequency signal of the audio signal; A low-frequency extraction module, configured to perform low-frequency feature extraction processing on the low-frequency signal to obtain the low-frequency features of the audio signal; A high frequency analysis module, configured to perform high frequency analysis processing on the audio signal to obtain high frequency features of the audio signal; Wherein, the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature; The encoding module is configured to encode the low-frequency features to obtain a low-frequency code stream of the audio signal, and to encode the high-frequency features to obtain a high-frequency code stream of the audio signal.
21. An audio decoding device, the device comprising: A decoding module configured to decode a low-frequency code stream of an audio signal to obtain low-frequency features corresponding to the low-frequency code stream, and to decode a high-frequency code stream of the audio signal to obtain high-frequency features corresponding to the high-frequency code stream; The low-frequency code stream is obtained by encoding a low-frequency signal obtained by downsampling the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature; A low-frequency reconstruction module is configured to perform low-frequency feature reconstruction processing on the low-frequency feature to obtain a low-frequency signal corresponding to the low-frequency feature; an up-sampling module, configured to perform up-sampling processing on the low-frequency signal to obtain an up-sampled signal of the low-frequency signal; The signal reconstruction module is configured to perform signal reconstruction processing on the high-frequency features and the up-sampled signal to obtain a synthetic audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
22. An electronic device, comprising: Memory for storing computer programs or computer executable instructions; A processor, used to implement the audio encoding method described in any one of claims 1 to 9, or the audio decoding method described in any one of claims 10 to 19 when executing the computer program or computer executable instructions stored in the memory.
23. A computer-readable storage medium storing a computer program or a computer-executable instruction, wherein the computer program or the computer-executable instruction, when executed by a processor, implements the audio encoding method according to any one of claims 1 to 9, or the audio decoding method according to any one of claims 10 to 19.
24. A computer program product, comprising a computer program or computer executable instructions, which, when executed by a processor, implements the audio encoding method described in any one of claims 1 to 9, or the audio decoding method described in any one of claims 10 to 19.
Citation Information
Patent Citations
Audio processing method and device, equipment, storage medium and computer program product
CN115116456A
Audio encoding and decoding method, device, equipment, medium and program product
CN115116457A
Audio coding method and device, audio decoding method and device, equipment and storage medium
CN117219095A
Encoding device and decoding device
JP2006293400A
