Audio decoding, encoding methods, devices, electronic equipment and storage media

Abstract

This application provides an audio decoding and encoding method, apparatus, electronic device, and storage medium applicable to automotive scenarios. The audio decoding method includes: acquiring a bitstream, wherein the bitstream is obtained by encoding an audio signal; decoding the bitstream to obtain predicted values ​​of feature vectors of the audio signal; performing tag extraction processing on the predicted values ​​of the feature vectors to obtain tag information vectors for signal enhancement, wherein the dimension of the tag information vectors is the same as the dimension of the predicted values ​​of the feature vectors; reconstructing the signal based on the predicted values ​​of the feature vectors and the tag information vectors; and using the predicted values ​​of the reconstructed audio signal as the decoding result of the bitstream. This application effectively suppresses acoustic interference in audio signals, thereby improving the quality of the reconstructed audio signal.

Description

[0001] This application is a divisional application of patent application No. 202210676984.X, filed on June 15, 2022, entitled "Audio Decoding, Encoding Method, Apparatus, Electronic Device and Storage Medium". Technical Field

[0002] This application relates to the field of communication technology, and in particular to an audio decoding and encoding method, apparatus, electronic device and storage medium. Background Technology

[0003] Due to the convenience and immediacy of voice communication, voice calls are becoming increasingly widespread, for example, in transmitting audio signals (such as voice signals) between participants in online conferences. However, during voice calls, the voice signal may be subject to acoustic interference such as noise. This noise can degrade call quality, significantly impacting the user's auditory experience.

[0004] However, there is still no effective solution for how to enhance speech signals to suppress noise. Summary of the Invention

[0005] This application provides an audio decoding and encoding method, apparatus, electronic device, computer-readable storage medium, and computer program product that can effectively suppress acoustic interference in audio signals, thereby improving the quality of the reconstructed audio signals.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides an audio decoding method, including:

[0008] Obtain the bitstream, wherein the bitstream is obtained by encoding an audio signal;

[0009] The bitstream is decoded to obtain the predicted value of the feature vector of the audio signal;

[0010] The predicted values ​​of the feature vector are subjected to label extraction processing to obtain a label information vector for signal enhancement, wherein the dimension of the label information vector is the same as the dimension of the predicted values ​​of the feature vector.

[0011] Signal reconstruction is performed based on the predicted values ​​of the feature vectors and the label information vectors;

[0012] The predicted value of the audio signal reconstructed from the signal is used as the decoding result of the bitstream.

[0013] This application provides an audio decoding device, including:

[0014] An acquisition module is used to acquire a bitstream, wherein the bitstream is obtained by encoding an audio signal;

[0015] A decoding module is used to decode the bitstream to obtain the predicted value of the feature vector of the audio signal;

[0016] The label extraction module is used to perform label extraction processing on the predicted value of the feature vector to obtain a label information vector for signal enhancement, wherein the dimension of the label information vector is the same as the dimension of the predicted value of the feature vector.

[0017] The reconstruction module is used to reconstruct the signal based on the predicted value of the feature vector and the label information vector;

[0018] The determination module is used to take the predicted value of the audio signal reconstructed from the signal as the decoding result of the bitstream.

[0019] This application provides an audio encoding method, including:

[0020] Acquire audio signals;

[0021] The audio signal is encoded to obtain a bitstream, wherein the bitstream is used by an electronic device to execute the audio decoding method provided in the embodiments of this application.

[0022] This application provides an audio encoding device, including:

[0023] The acquisition module is used to acquire audio signals;

[0024] An encoding module is used to encode the audio signal to obtain a bitstream, wherein the bitstream is used by an electronic device to execute the audio decoding method provided in the embodiments of this application.

[0025] This application provides an electronic device, including:

[0026] Memory, used to store executable instructions;

[0027] The processor, when executing executable instructions stored in the memory, implements the audio encoding / decoding method provided in the embodiments of this application.

[0028] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the audio encoding / decoding method provided in this application.

[0029] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the audio encoding / decoding method provided in this application.

[0030] The embodiments of this application have the following beneficial effects:

[0031] By extracting labels from the predicted values ​​of the decoded feature vectors, a label information vector for signal enhancement is obtained. The predicted values ​​of the feature vectors and the label information vector are then combined to reconstruct the signal. Compared to reconstructing the signal solely based on the predicted values ​​of the feature vectors, this method increases the proportion of core components in the audio signal and reduces the proportion of acoustic interference such as noise. This effectively suppresses noise components in the audio signal acquired at the encoding end, thereby improving the quality of the reconstructed audio signal. Attached Figure Description

[0032] Figure 1 This is a schematic diagram comparing the spectrum at different code rates provided in the embodiments of this application;

[0033] Figure 2 This is a schematic diagram of the architecture of the audio codec system 100 provided in the embodiments of this application;

[0034] Figure 3 This is a schematic diagram of the structure of the second terminal device 500 provided in the embodiments of this application;

[0035] Figure 4A This is a flowchart illustrating the audio encoding method provided in an embodiment of this application;

[0036] Figure 4B This is a flowchart illustrating the audio decoding method provided in an embodiment of this application;

[0037] Figure 4C This is a flowchart illustrating the audio encoding / decoding method provided in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram of the structure of the encoding and decoding ends provided in the embodiments of this application.

[0039] Figure 6A and Figure 6B This is a flowchart illustrating the audio decoding method provided in an embodiment of this application;

[0040] Figure 7 This is a schematic diagram of an end-to-end voice communication link provided in an embodiment of this application;

[0041] Figure 8 This is a flowchart illustrating the audio encoding / decoding method provided in an embodiment of this application;

[0042] Figure 9A This is a schematic diagram of a common convolution provided in an embodiment of this application;

[0043] Figure 9B This is a schematic diagram of dilated convolution provided in an embodiment of this application;

[0044] Figure 10 This is a schematic diagram of the spectral response of the low-pass and high-pass portions of the QMF analysis filter bank provided in the embodiments of this application;

[0045] Figure 11A This is a schematic diagram illustrating the principle of obtaining a 4-channel sub-band signal based on a QMF filter bank, as provided in an embodiment of this application.

[0046] Figure 11B This is a schematic diagram illustrating the principle of obtaining a 3-channel sub-band signal based on a QMF filter bank, as provided in an embodiment of this application.

[0047] Figure 12 This is a schematic diagram of the analysis network structure provided in the embodiments of this application;

[0048] Figure 13 This is a schematic diagram of the structure of the augmented network provided in the embodiments of this application;

[0049] Figure 14 This is a schematic diagram of the structure of the synthetic network provided in the embodiments of this application;

[0050] Figure 15 This is a flowchart illustrating the audio encoding / decoding method provided in an embodiment of this application;

[0051] Figure 16 This is a flowchart illustrating the audio encoding / decoding method provided in an embodiment of this application;

[0052] Figure 17 This is a schematic diagram of the structure of the first analysis network provided in the embodiments of this application;

[0053] Figure 18 This is a schematic diagram of the structure of the first enhancement network provided in the embodiments of this application;

[0054] Figure 19 This is a schematic diagram of the structure of the first synthetic network provided in the embodiments of this application;

[0055] Figure 20 This is a schematic diagram comparing the encoding and decoding effects provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0058] It is understood that in the embodiments of this application, data related to user information (such as voice signals emitted by the user) is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0059] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0062] 1) Neural Network (NN): A mathematical model that mimics the behavioral characteristics of animal neural networks to perform distributed parallel information processing. This type of network relies on the complexity of the system to adjust the relationships between a large number of interconnected nodes, thereby achieving the purpose of information processing.

[0063] 2) Deep Learning (DL): This is a new research direction in the field of Machine Learning (ML). Deep learning learns the inherent patterns and representational layers of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to be able to recognize data such as text, images, and sound.

[0064] 3) Vector Quantization (VQ): An efficient lossy compression technique based on Shannon's rate distortion theory. The basic principle of vector quantization is to use the index of the codeword in the codebook that best matches the input vector to replace the input vector for transmission and storage, while decoding only requires a simple table lookup operation.

[0065] 4) Scalar quantization: This involves quantizing a scalar, also known as one-dimensional vector quantization. It divides the dynamic range into several small intervals, each with a representative value. When the input signal falls into a certain interval, it is quantized into that representative value.

[0066] 5) Entropy coding: This is a lossless coding method that does not lose any information during the encoding process according to the entropy principle. It is also a key module in lossy coding and is located at the end of the encoder. Common entropy coding methods include Shannon coding, Huffman coding, Exponential Golomb coding, and arithmetic coding.

[0067] 6) Quadrature Mirror Filters (QMF): This is a filter pair that includes analysis and synthesis. The QMF analysis filter pair is used to decompose the subband signal to reduce the signal bandwidth, so that each subband signal can be processed smoothly by the channel. The QMF synthesis filter pair is used to synthesize the subband signals recovered from the decoder, such as by zero-value interpolation and bandpass filtering, to reconstruct the original audio signal.

[0068] Speech encoding and decoding technology is a core technology in communication services, including remote audio and video calls. Simply put, speech coding technology aims to transmit as much voice information as possible using less network bandwidth. From the perspective of Shannon's information theory, speech coding is a type of source coding. The goal of source coding is to compress the amount of data to be transmitted as much as possible at the encoding end, removing redundancy, while simultaneously recovering the information losslessly (or nearly losslessly) at the decoding end.

[0069] The compression rates of the speech codecs provided by related technologies can reach more than 10 times. This means that 10MB of original speech data can be compressed by the encoder to only 1MB for transmission, significantly reducing the bandwidth resources required to transmit information. For example, for a wideband speech signal with a sampling rate of 16000Hz, if a 16-bit sampling depth is used, the uncompressed version has a bitrate of 256kbps. However, if speech coding technology is used, even with lossy coding, the quality of the reconstructed speech signal can be close to the uncompressed version within a bitrate range of 10-20kbps, and may even be perceived as indistinguishable. If higher sampling rate services are required, such as 32000Hz ultrawideband speech, the bitrate range must be at least 30kbps.

[0070] Traditional speech coding schemes provided by related technologies can generally be divided into three types according to their coding principles: waveform speech coding, parametric speech coding, and hybrid speech coding.

[0071] Waveform coding directly encodes the waveform of the speech signal. The advantage of this coding method is that the encoded speech quality is high, but the compression rate is not high.

[0072] Parametric coding refers to modeling the speech production process, and the encoder's job is to extract the corresponding parameters of the speech signal to be transmitted. The advantage of parametric coding is its extremely high compression rate, but the disadvantage is that the quality of the recovered speech is not high.

[0073] Hybrid coding combines the two coding methods mentioned above. Speech components that can be encoded using parameters are represented by parameters, while components that cannot be effectively represented by parameters are encoded using waveforms. This combination achieves both high coding efficiency and high-quality reconstructed speech.

[0074] Generally, the three coding principles mentioned above are derived from classical speech signal modeling, also known as signal processing-based compression methods. Based on rate-distortion analysis and standardization experience over the past few decades, a bit rate of at least 0.75 bits / sample is recommended to provide ideal speech quality; for a wideband speech signal with a sampling rate of 16000 Hz, this is equivalent to 12 kps. For example, the IETF OPUS standard recommends 16 kps as the recommended bit rate for providing high-quality wideband voice calls.

[0075] For example, see Figure 1 , Figure 1This is a schematic diagram comparing the spectrum at different bitrates provided in an embodiment of this application, illustrating the relationship between compression bitrate and quality. Curve 101 represents the original speech, i.e., the uncompressed audio signal; curve 102 represents the effect of an OPUS encoder at 20kbps; and curve 103 represents the effect of an OPUS encoder at 6kbps. Figure 1 It can be seen that as the encoding bitrate increases, the compressed signal is closer to the original signal.

[0076] However, the applicant found that the solutions provided by the relevant technologies mainly rely on traditional signal processing methods, and it is difficult to significantly reduce the bit rate while maintaining the existing quality.

[0077] In recent years, with the advancement of deep learning, related technologies have also provided solutions for using artificial intelligence to improve coding bitrate.

[0078] However, the applicant also found that while AI-based audio codecs can achieve bitrates below 2kbps, the typical use of generative networks like WaveNet results in extremely high complexity at the decoding end, posing a significant challenge for mobile devices. Furthermore, the absolute quality is significantly inferior to that of traditional signal processing encoders. On the other hand, end-to-end neural network (NN) codecs, with bitrates of 6-10kbps and subjective quality approaching that of traditional signal processing schemes, employ deep learning networks at both the encoding and decoding ends, leading to very high complexity.

[0079] Furthermore, both traditional signal processing solutions and deep neural network-based solutions can only compress speech signals. However, real-world speech communication is affected by acoustic interference such as noise. In other words, there is currently no solution in the relevant technologies that simultaneously achieves speech enhancement and high-quality compression with low bit rates.

[0080] In view of this, embodiments of this application provide an audio decoding and encoding method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively suppress acoustic interference in audio signals while improving encoding efficiency, thereby improving the quality of the reconstructed audio signal. The following describes exemplary applications of the electronic device provided in embodiments of this application. The electronic device provided in embodiments of this application can be implemented as a terminal device, a server, or implemented collaboratively by a terminal device and a server. The following description uses the example of an audio decoding method provided in embodiments of this application being implemented collaboratively by a terminal device and a server.

[0081] For example, see Figure 2 , Figure 2This is a schematic diagram of the architecture of the audio codec system 100 provided in this application embodiment. It aims to support an application that can effectively suppress acoustic interference in the audio signal while improving coding efficiency, thereby improving the quality of the reconstructed audio signal. Figure 2 As shown, the audio encoding and decoding system 100 includes: a server 200, a network 300, a first terminal device 400 (i.e., the encoding end) and a second terminal device 500 (i.e., the decoding end), wherein the network 300 can be a local area network, a wide area network, or a combination of both.

[0082] In some embodiments, a client 410 runs on the first terminal device 400. The client 410 can be various types of clients, such as instant messaging clients, web conferencing clients, live streaming clients, browsers, etc. In response to an audio acquisition command triggered by a sender (e.g., the initiator of a web conference, a broadcaster, or the initiator of a voice call), the client 410 calls the microphone in the terminal device 400 to acquire audio signals and encodes the acquired audio signals to obtain a bitstream. Then, the client 410 can send the bitstream to the server 200 via the network 300, so that the server 200 can send the bitstream to the second terminal device 500 associated with the recipient (e.g., a participant in a web conference, an audience member, or the recipient of a voice call). After receiving the bitstream sent by the server 200, the client 510 (e.g., an instant messaging client, web conferencing client, live streaming client, browser, etc.) can decode the bitstream to obtain the predicted value (also known as the estimated value) of the feature vector of the audio signal. Then, the client 510 can call the enhancement network to perform label extraction processing on the predicted value of the feature vector to obtain the label information vector for signal enhancement. The dimension of the label information vector is the same as the dimension of the predicted value of the feature vector. Subsequently, based on the predicted value of the decoded feature vector and the label information vector obtained after label extraction, the client 510 can call the synthesis network to reconstruct the signal and obtain the predicted value of the audio signal, thereby completing the reconstruction of the audio signal and suppressing the noise components contained in the audio signal collected by the encoding end, thus improving the quality of the reconstructed audio signal.

[0083] The audio encoding and decoding method provided in this application can be widely applied to various types of voice or video call application scenarios, such as in-vehicle voice calls implemented through applications running on in-vehicle terminals, voice or video calls conducted through instant messaging clients, voice calls in game applications, and voice calls in web conferencing clients. For example, voice enhancement can be performed at the receiving end of the voice call or on the server providing voice communication services according to the audio decoding method provided in this application.

[0084] For example, taking a web conferencing scenario as an example, web conferencing is an important part of online work. In a web conferencing session, after the audio acquisition device (e.g., microphone) of the participants acquires the speaker's voice signal, it needs to send the acquired voice signal to other participants. This process involves the transmission and playback of the voice signal among multiple participants. If the noise mixed in the voice signal is not processed, it will greatly affect the auditory experience of the meeting participants. In this scenario, the audio decoding method provided in the embodiments of this application can be applied to enhance the voice signal in the web conferencing, so that the voice signal heard by the meeting participants is the enhanced voice signal (i.e., the noise component in the voice signal acquired at the encoding end is suppressed in the reconstructed voice signal), thereby improving the quality of voice calls in the web conferencing.

[0085] In other embodiments, the embodiments of this application can be implemented using cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize data computing, storage, processing, and sharing.

[0086] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. The service interaction functions between the aforementioned 200 servers can be achieved through cloud technology.

[0087] Example, Figure 2 The server 200 shown can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Figure 2 The terminal devices 400 and 500 shown can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, in-vehicle terminals, etc., but are not limited to these. The terminal devices (e.g., the first terminal device 400 and the second terminal device 500) and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0088] In some embodiments, the terminal device (e.g., the second terminal device 500) or server 200 can also implement the audio decoding method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a live streaming APP, a web conferencing APP, or an instant messaging APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0089] The following continues... Figure 2 The structure of the second terminal device 500 shown in the diagram will be described. See also... Figure 3 , Figure 3 This is a schematic diagram of the structure of the second terminal device 500 provided in the embodiments of this application. Figure 3 The second terminal device 500 shown includes at least one processor 520, a memory 560, at least one network interface 530, and a user interface 540. The various components in the second terminal device 500 are coupled together via a bus system 550. It is understood that the bus system 550 is used to implement communication between these components. In addition to a data bus, the bus system 550 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 3 The general labeled all buses as Bus System 550.

[0090] Processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0091] User interface 540 includes one or more output devices 541 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 540 also includes one or more input devices 542, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0092] The memory 560 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 560 may optionally include one or more storage devices physically located away from the processor 520.

[0093] The memory 560 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 560 described in this application embodiment is intended to include any suitable type of memory.

[0094] In some embodiments, memory 560 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0095] Operating system 561 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks;

[0096] The network communication module 562 is used to reach other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0097] Presentation module 563 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 541 (e.g., a display screen, a speaker, etc.) associated with user interface 540;

[0098] The input processing module 564 is used to detect and translate one or more user inputs or interactions from one or more input devices 542.

[0099] In some embodiments, the audio decoding device provided in this application can be implemented in software. Figure 3An audio decoding device 565 stored in memory 560 is shown. It may be software in the form of programs and plug-ins, including the following software modules: acquisition module 5651, decoding module 5652, tag extraction module 5653, reconstruction module 5654, and determination module 5655. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0100] The audio encoding and decoding method provided in this application will be specifically described below with reference to the exemplary application of the terminal device provided in the embodiments of this application.

[0101] For example, see Figure 4A , Figure 4A This is a flowchart illustrating the audio encoding method provided in an embodiment of this application, as shown below. Figure 4A As shown, the main steps performed at the encoding end include: Step 101, acquiring the audio signal; Step 102, encoding the audio signal to obtain the bitstream.

[0102] For example, see Figure 4B , Figure 4B This is a flowchart illustrating the audio decoding method provided in an embodiment of this application, as shown below. Figure 4B As shown, the main steps performed at the decoding end include: Step 201, acquiring the bitstream; Step 202, decoding the bitstream to obtain the predicted value of the feature vector of the audio signal; Step 203, performing tag extraction processing on the predicted value of the feature vector to obtain the tag information vector for signal enhancement; Step 204, reconstructing the signal based on the predicted value of the feature vector and the tag information vector; Step 205, using the predicted value of the audio signal obtained through signal reconstruction as the decoding result of the bitstream.

[0103] The following will take a conferencing system based on Voice over Internet Protocol (VoIP) as an example to specifically describe the audio encoding and decoding method provided in this application embodiment from the perspective of the interaction between the first terminal device (i.e., the encoding end), the server, and the second terminal device (i.e., the decoding end).

[0104] For example, see Figure 4C , Figure 4C This is a flowchart illustrating the audio encoding / decoding method provided in the embodiments of this application, which will be combined with... Figure 4C The steps shown are explained.

[0105] It should be noted that the steps executed by the terminal device are specifically performed by the client running on the terminal device. For ease of description, this application does not specifically distinguish between the terminal device and the client running on the terminal device. Furthermore, it should be noted that the audio encoding / decoding method provided in this application's embodiments can be executed by various forms of computer programs running on the terminal device, and is not limited to the client running on the terminal device mentioned above. It can also be the operating system 561, software modules, scripts, and applets mentioned above. Therefore, the client example used below should not be considered a limitation of the embodiments of this application.

[0106] In the Figure 4C Before proceeding with the explanation, let's first describe the structure of the encoding and decoding ends.

[0107] For example, see Figure 5 , Figure 5 This is a schematic diagram of the structure of the encoding and decoding ends provided in the embodiments of this application, as shown below. Figure 5 As shown, the encoding end includes an analysis network for feature extraction of the input audio signal, obtaining the feature vector of the audio signal. Then, the feature vector of the audio signal can be quantized and encoded to obtain the bitstream. The decoding end includes an enhancement network and a synthesis network. After decoding the received bitstream to obtain the predicted value of the feature vector of the audio signal, the enhancement network can be called to perform label extraction processing on the predicted value of the feature vector of the audio signal, obtaining a label information vector for signal enhancement. Subsequently, based on the label information vector and the predicted value of the feature vector, the synthesis network can be called to reconstruct the signal and obtain the predicted value of the audio signal.

[0108] The audio encoding and decoding method provided in this application embodiment will be described in detail below, taking into account the above-described structures of the encoding and decoding ends.

[0109] In step 301, the first terminal device acquires an audio signal.

[0110] In some embodiments, in response to an audio acquisition command triggered by a user, the first terminal device calls an audio acquisition device (e.g., a microphone built into the first terminal device or an external microphone) to acquire audio signals, such as the voice signal of a speaker in a web conference scenario or the voice signal of a broadcaster in a live broadcast scenario.

[0111] For example, in a web conferencing scenario, when the web conferencing app running on the first terminal device receives a click from a user (e.g., the initiator of the web conferencing) on ​​the "open microphone" button displayed in the human-computer interaction interface, it calls the microphone (or microphone array) built into the first terminal device to collect the voice signal emitted by the user, thus obtaining the voice signal of the initiator of the web conferencing.

[0112] In step 302, the first terminal device encodes the audio signal to obtain a bitstream.

[0113] In some embodiments, after the first terminal device acquires an audio signal by calling the microphone, it can encode the audio signal to obtain a bitstream in the following manner: first, it calls an analysis network (e.g., a neural network) to perform feature extraction processing on the audio signal to obtain a feature vector of the audio signal; then, it performs quantization processing on the feature vector of the audio signal (e.g., vector quantization or scalar quantization) to obtain the index value of the feature vector; finally, it performs encoding processing on the index value of the feature vector (e.g., entropy encoding) to obtain a bitstream.

[0114] For example, the first terminal device can perform feature extraction processing on the audio signal to obtain the feature vector of the audio signal in the following manner: First, perform convolution processing (e.g., causal convolution) on the audio signal to obtain the convolutional features of the audio signal; then, perform pooling processing on the convolutional features of the audio signal to obtain the pooled features of the audio signal; subsequently, perform downsampling processing on the pooled features of the audio signal to obtain the downsampled features of the audio signal; finally, perform convolution processing on the downsampled features of the audio signal to obtain the feature vector of the audio signal.

[0115] In other embodiments, the first terminal device may further encode the audio signal to obtain a bitstream by: decomposing the acquired audio signal (e.g., by using a 2-channel QMF analysis filter bank) to obtain low-frequency subband signals and high-frequency subband signals; then performing feature extraction processing on the low-frequency subband signals and high-frequency subband signals respectively to obtain feature vectors for the low-frequency subband signals and high-frequency subband signals respectively; subsequently, performing quantization encoding processing on the feature vectors of the low-frequency subband signals to obtain the low-frequency bitstream of the audio signal, and performing quantization encoding processing on the feature vectors of the high-frequency subband signals to obtain the high-frequency bitstream of the audio signal.

[0116] For example, the first terminal device can decompose the audio signal to obtain low-frequency subband signals and high-frequency subband signals in the following manner: First, the audio signal is sampled to obtain a sampled signal, which includes multiple sample points acquired; then, the sampled signal is low-pass filtered to obtain a low-pass filtered signal; subsequently, the low-pass filtered signal is downsampled to obtain the low-frequency subband signal. Similarly, the sampled signal is high-pass filtered to obtain a high-pass filtered signal, and the high-pass filtered signal is downsampled to obtain the high-frequency subband signal.

[0117] In some embodiments, the first terminal device can further encode the audio signal to obtain a bitstream by: firstly, decomposing the acquired audio signal to obtain N sub-band signals (for example, it can be decomposed using a 4-channel QMF analysis filter bank to obtain 4 sub-band signals; for instance, the audio signal can first be subjected to low-pass and high-pass filtering to obtain low-frequency and high-frequency sub-band signals, and then the low-frequency sub-band signals can be subjected to low-pass and high-pass filtering again to obtain sub-band signal 1 and sub-band signal 2 respectively; similarly, the decomposed audio signal can be further processed to obtain the bitstream signal. The high-frequency sub-band signal is then subjected to low-pass and high-pass filtering again, resulting in sub-band signal 3 and sub-band signal 4. Thus, by iterating through two layers of 2-channel QMF analysis and filtering, the audio signal can be decomposed into 4 sub-band signals, where N is an integer greater than 2. Next, feature extraction processing is performed on each sub-band signal to obtain the feature vector of each sub-band signal (for example, for each sub-band signal obtained by decomposition, a neural network model is called to perform feature extraction processing to obtain the feature vector of that sub-band signal). Subsequently, the feature vector of each sub-band signal is quantized and encoded to obtain N sub-bitstreams.

[0118] In step 303, the first terminal device sends a bitstream to the server.

[0119] In some embodiments, after the first terminal device encodes the acquired audio signal to obtain a bitstream, it can send the bitstream to the server via a network.

[0120] In step 304, the server sends the bitstream to the second terminal device.

[0121] In some embodiments, after receiving the bitstream sent by the first terminal device (i.e., the encoding end, such as the terminal device associated with the initiator of the web conference), the server can send the bitstream to the second terminal device (i.e., the decoding end, such as the terminal device associated with the participants of the web conference) via the network.

[0122] In other embodiments, considering forward compatibility, a transcoder can be deployed in the server to address interoperability issues between new encoders (i.e., encoders that encode using artificial intelligence-based methods, such as NN encoders) and traditional encoders (i.e., encoders that encode using time-domain and frequency-domain transformations, such as G.722 encoders). For example, if the first terminal device (i.e., the transmitter) deploys a new NN encoder, while the second terminal device (i.e., the receiver) deploys a traditional decoder (e.g., a G.722 decoder), the second terminal device will be unable to correctly decode the bitstream sent by the first terminal device. To address this, a transcoder can be deployed in the server. For instance, after receiving the bitstream encoded by the NN encoder from the first terminal device, the server can first call the NN decoder to generate an audio signal, and then call the traditional encoder (e.g., a G.722 encoder) to generate a specific bitstream, thus enabling the second terminal device to decode it correctly.

[0123] In step 305, the second terminal device decodes the bitstream to obtain the predicted value of the feature vector of the audio signal.

[0124] In some embodiments, the second terminal device can implement step 305 in the following manner: First, the bitstream is decoded (wherein, the decoding process is the inverse of the encoding process; for example, when the encoding end uses entropy encoding to encode the feature vector of the audio signal to obtain the bitstream, the decoding end can correspondingly use entropy decoding to decode the received bitstream to obtain the index value of the feature vector of the audio signal), and the index value of the feature vector of the audio signal is obtained; then, the quantization table is queried based on the index value (for example, when the encoding end uses the index value of the codeword in the quantization table that best matches the feature vector of the audio signal to replace the feature vector for subsequent encoding, the decoding end can perform a simple table lookup operation based on the index value after decoding the bitstream to obtain the index value, and thus obtain the predicted value of the feature vector of the audio signal), and the predicted value of the feature vector of the audio signal is obtained. In other embodiments, when the bitstream includes a low-frequency bitstream (wherein the low-frequency bitstream is obtained by encoding the low-frequency sub-band signal obtained after decomposing the audio signal) and a high-frequency bitstream (wherein the high-frequency bitstream is obtained by encoding the high-frequency sub-band signal obtained after decomposing the audio signal), the second terminal device can also implement step 305 above in the following way: Decode the low-frequency bitstream (for example, when the encoding end uses entropy encoding to encode the feature vector of the low-frequency sub-band signal, the decoding end can use a corresponding entropy decoding method to decode the low-frequency bitstream) to obtain the predicted value of the feature vector of the low-frequency sub-band signal; decode the high-frequency bitstream to obtain the predicted value of the feature vector of the high-frequency sub-band signal. For example, for the low-frequency bitstream, the second terminal device can first decode the low-frequency bitstream to obtain the index value (assumed to be index value 1) of the feature vector of the low-frequency sub-band signal, and then query the quantization table based on index value 1 to obtain the predicted value of the feature vector of the low-frequency sub-band signal. Similarly, for high-frequency bitstreams, the second terminal device can first decode the high-frequency bitstream to obtain the index value of the feature vector of the high-frequency sub-band signal (assuming it is index value 2), and then look up the quantization table based on index value 2 to obtain the predicted value of the feature vector of the high-frequency sub-band signal. In some embodiments, when the bitstream includes N sub-bitstreams (where the N sub-bitstreams correspond to different frequency bands, and are obtained by encoding the N sub-band signals obtained after decomposing the audio signal, and N is an integer greater than 2), the second terminal device can also implement the above step 105 in the following way: decode the N sub-bitstreams respectively to obtain the predicted value of the feature vector corresponding to the N sub-band signals respectively. It should be noted that the decoding process for the N sub-bitstreams here can be implemented with reference to the decoding process for low-frequency or high-frequency bitstreams described above, and will not be repeated here in this embodiment.

[0125] For example, taking N sub-streams as 4 sub-streams, namely sub-stream 1, sub-stream 2, sub-stream 3, and sub-stream 4, sub-stream 1 is obtained by encoding sub-band signal 1, sub-stream 2 is obtained by encoding sub-band signal 2, sub-stream 3 is obtained by encoding sub-band signal 3, and sub-stream 4 is obtained by encoding sub-band signal 4. After receiving these 4 sub-streams, the second terminal device can decode these 4 sub-streams respectively to obtain the predicted values ​​of the feature vectors corresponding to the 4 sub-band signals, such as the predicted values ​​of the feature vectors of sub-band signal 1, sub-band signal 2, sub-band signal 3, and sub-band signal 4.

[0126] In step 306, the second terminal device performs label extraction processing on the predicted value of the feature vector to obtain a label information vector for signal enhancement.

[0127] Here, the dimension of the label information vector is the same as the dimension of the predicted value of the feature vector. This allows for a perfect concatenation of the predicted value of the feature vector and the label information vector during subsequent signal reconstruction, enhancing all core components of the reconstructed audio signal.

[0128] In some embodiments, the second terminal device can extract labels from the predicted values ​​of the feature vector by calling the enhancement network to obtain a label information vector for signal enhancement. The enhancement network includes convolutional layers, neural network layers, fully connected network layers, and activation layers. The process of extracting the label information vector is described below in conjunction with the above structure of the enhancement network.

[0129] For example, see Figure 6A , Figure 6A This is a flowchart illustrating the audio decoding method provided in an embodiment of this application, as shown below. Figure 6A As shown, Figure 4C Step 306 shown can be achieved through Figure 6A Steps 3061 to 3064 shown are implemented, and will be combined with Figure 6A The steps shown are explained.

[0130] In step 3061, the second terminal device performs convolution processing on the predicted value of the feature vector to obtain a first tensor with the same dimension as the predicted value of the feature vector.

[0131] In some embodiments, the second terminal device can use the predicted value of the feature vector obtained in step 305 as input, and call the convolutional layer (e.g., a one-dimensional causal convolution) included in the enhancement network to generate a first tensor (a tensor is a quantity that includes numerical values ​​in multiple dimensions) with the same dimension as the predicted value of the feature vector. For example, ... Figure 13As shown, the predicted dimension of the feature vector is 56×1, so after causal convolution, a 56×1 tensor is generated.

[0132] In step 3062, the second terminal device performs feature extraction processing on the first tensor to obtain a second tensor with the same dimension as the first tensor.

[0133] In some embodiments, for the first tensor obtained after causal convolution, feature extraction can be performed using neural network layers (e.g., long short-term memory networks, temporal recurrent neural networks, etc.) to generate a second tensor with the same dimensions as the first tensor. For example, as... Figure 13 As shown, the first tensor has a dimension of 56×1. After feature extraction processing through a Long Short-Term Memory (LSTM) network, a 56×1 tensor is generated.

[0134] In step 3063, the second terminal device performs a full connection on the second tensor to obtain a third tensor with the same dimensions as the second tensor.

[0135] In some embodiments, after feature extraction processing by the neural network layers included in the augmented network to obtain a second tensor with the same dimension as the first tensor, the second terminal device can invoke the fully connected network layers included in the augmented network to perform fully connected processing on the second tensor to obtain a third tensor with the same dimension as the second tensor. For example, as... Figure 13 As shown, the second tensor has a dimension of 56×1. After calling a fully connected network layer for fully connected processing, a tensor of 56×1 is generated.

[0136] In step 3064, the second terminal device performs activation processing on the third tensor to obtain a tag information vector for signal enhancement.

[0137] In some embodiments, after the fully connected processing of the fully connected network layers included in the augmented network to obtain a third tensor with the same dimension as the second tensor, the second terminal device can call the activation layer included in the augmented network, i.e., the activation function (e.g., ReLU function, Sigmoid function, Tanh function, etc.), to activate the third tensor. This generates a label information vector with the same dimension as the predicted value of the feature vector. For example, as... Figure 13 As shown, the third tensor has a dimension of 56×1. After calling the ReLU function to activate the third tensor, a label information vector with a dimension of 56×1 is obtained.

[0138] In other embodiments, when the predicted value of the feature vector includes the predicted value of the feature vector of the low-frequency subband signal and the predicted value of the feature vector of the high-frequency subband signal, the second terminal device can also implement step 306 above in the following way: perform label extraction processing on the predicted value of the feature vector of the low-frequency subband signal to obtain a first label information vector for low-frequency subband signal enhancement, wherein the dimension of the first label information vector is the same as the dimension of the predicted value of the feature vector of the low-frequency subband signal; perform label extraction processing on the predicted value of the feature vector of the high-frequency subband signal to obtain a second label information vector for high-frequency subband signal enhancement, wherein the dimension of the second label information vector is the same as the dimension of the predicted value of the feature vector of the high-frequency subband signal.

[0139] For example, the second terminal device can perform the label extraction processing on the predicted value of the feature vector of the low-frequency subband signal to obtain the first label information vector for low-frequency subband signal enhancement in the following way: call the first enhancement network to perform the following processing: perform convolution processing on the predicted value of the feature vector of the low-frequency subband signal to obtain a fourth tensor with the same dimension as the predicted value of the feature vector of the low-frequency subband signal; perform feature extraction processing on the fourth tensor to obtain a fifth tensor with the same dimension as the fourth tensor; perform fully connected processing on the fifth tensor to obtain a sixth tensor with the same dimension as the fifth tensor; perform activation processing on the sixth tensor to obtain the first label information vector for low-frequency subband signal enhancement.

[0140] For example, the second terminal device can perform the label extraction processing on the predicted value of the feature vector of the high-frequency subband signal to obtain the second label information vector for high-frequency subband signal enhancement in the following way: call the second enhancement network to perform the following processing: perform convolution processing on the predicted value of the feature vector of the high-frequency subband signal to obtain a seventh tensor with the same dimension as the predicted value of the feature vector of the high-frequency subband signal; perform feature extraction processing on the seventh tensor to obtain an eighth tensor with the same dimension as the seventh tensor; perform fully connected processing on the eighth tensor to obtain a ninth tensor with the same dimension as the eighth tensor; perform activation processing on the ninth tensor to obtain the second label information vector for high-frequency subband signal enhancement.

[0141] It should be noted that the label extraction process for the predicted feature vectors of low-frequency sub-band signals and the predicted feature vectors of high-frequency sub-band signals is similar to the label extraction process for the predicted feature vectors of audio signals. Please refer to [the relevant documentation / reference]. Figure 6A The implementation details are not repeated here. Furthermore, it should be noted that the structures of the first and second enhancement networks are similar to those of the enhancement networks described above, and will not be repeated here.

[0142] In some embodiments, when the predicted value of the feature vector includes the predicted values ​​of the feature vectors corresponding to the N sub-band signals respectively, the second terminal device can implement the above step 306 in the following way: perform label extraction processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals respectively to obtain N label information vectors for signal enhancement, wherein the dimension of each label information vector is the same as the dimension of the predicted value of the feature vector of the corresponding sub-band signal.

[0143] For example, the second terminal device can perform the above-mentioned label extraction processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals respectively to obtain N label information vectors for signal enhancement in the following way: Based on the predicted value of the feature vector of the i-th sub-band signal, the i-th enhancement network is called to perform label extraction processing to obtain the i-th label information vector for the enhancement of the i-th sub-band signal; where the value range of i satisfies 1≤i≤N, and the dimension of the i-th label information vector is the same as the dimension of the predicted value of the feature vector of the i-th sub-band signal.

[0144] For example, the second terminal device can achieve the above-mentioned prediction value based on the feature vector of the i-th sub-band signal in the following way: call the i-th enhancement network to perform label extraction processing to obtain the i-th label information vector for the enhancement of the i-th sub-band signal: call the i-th enhancement network to perform the following processing: perform convolution processing on the predicted value of the feature vector of the i-th sub-band signal to obtain a tenth tensor with the same dimension as the predicted value of the feature vector of the i-th sub-band signal; perform feature extraction processing on the tenth tensor to obtain an eleventh tensor with the same dimension as the tenth tensor; perform fully connected processing on the eleventh tensor to obtain a twelfth tensor with the same dimension as the eleventh tensor; perform activation processing on the twelfth tensor to obtain the i-th label information vector for the enhancement of the i-th sub-band signal.

[0145] It should be noted that the structure of the i-th enhancement network is similar to the structure of the enhancement network described above, and will not be repeated here in the embodiments of this application.

[0146] In step 307, the second terminal device reconstructs the signal based on the predicted value of the feature vector and the tag information vector to obtain the predicted value of the audio signal.

[0147] In some embodiments, the second terminal device can invoke a synthesis network to reconstruct the signal based on the predicted value of the feature vector and the label information vector, thereby obtaining the predicted value of the audio signal. The synthesis network includes a first convolutional layer, an upsampling layer, a pooling layer, and a second convolutional layer. The signal reconstruction process will be described below in conjunction with the above structure of the synthesis network.

[0148] For example, see Figure 6B , Figure 6B This is a flowchart illustrating the audio decoding method provided in an embodiment of this application, as shown below. Figure 6B As shown, Figure 4C The illustrated step 307 can be achieved through Figure 6B Steps 3071 to 3075 shown are implemented, and will be combined with Figure 6B The steps shown are explained.

[0149] In step 3071, the second terminal device concatenates the predicted value of the feature vector and the label information vector to obtain a concatenated vector.

[0150] In some embodiments, the second terminal device can concatenate the predicted value of the feature vector obtained in step 305 and the label information vector obtained in step 306 to obtain a concatenated vector, and use the concatenated vector as the input of the synthesis network to reconstruct the signal.

[0151] In step 3072, the second terminal device performs a first convolution process on the spliced ​​vector to obtain the convolution features of the audio signal.

[0152] In some embodiments, after concatenating the predicted value of the feature vector and the label information vector to obtain a concatenated vector, the second terminal device can invoke the first convolutional layer (e.g., a one-dimensional causal convolution) of the synthesis network to perform convolution processing on the concatenated vector to obtain the convolutional features of the audio signal. For example, as... Figure 14 As shown, after performing causal convolution on the concatenated vector, a tensor with a dimension of 192×1 is obtained (i.e., the convolutional features of the audio signal).

[0153] In step 3073, the second terminal device performs upsampling processing on the convolutional features to obtain upsampled features of the audio signal.

[0154] In some embodiments, after obtaining the convolutional features of the audio signal, the second terminal device can invoke the upsampling layer included in the synthesis network to perform upsampling processing on the convolutional features of the audio signal. The upsampling processing can be implemented through multiple cascaded decoding layers, with different sampling factors in each decoding layer. The second terminal device can then perform upsampling processing on the convolutional features of the audio signal in the following manner to obtain upsampled features of the audio signal: The convolutional features are upsampled through the first decoding layer in the multiple cascaded decoding layers; the upsampling result of the first decoding layer is output to subsequent cascaded decoding layers, and upsampling processing and upsampling results are continued through these subsequent cascaded decoding layers until the last decoding layer is output; the upsampling result output by the last decoding layer is used as the upsampled features of the audio signal.

[0155] For example, see Figure 14Taking a cascaded decoding layer (also called a decoding block) as an example, three cascaded decoding layers can be cascaded with different upsampling factors (Up_factor). Taking a decoding layer (Up_factor = 8) as an example, one or more dilated convolutions can be performed first, with each convolution kernel size fixed at 1×3 and a stide rate of 1. Furthermore, the dilation rate of the one or more dilated convolutions can be set according to requirements, for example, it can be set to 3. Of course, this embodiment does not limit different dilation rates for different dilated convolutions. Then, the Up_factors of the three decoding layers are set to 8, 5, and 4 respectively, which is equivalent to setting different pooling factors, achieving the effect of upsampling. Finally, the number of channels of the three decoding layers are set to 96, 48, and 24 respectively. Thus, after upsampling through three decoding layers, the convolutional features of the audio signal (e.g., a 192×1 tensor) will be successively converted into tensors of 96×8, 48×40, and 24×160. The 24×160 tensor can then be used as the upsampled feature of the audio signal.

[0156] In step 3074, the second terminal device performs pooling processing on the upsampled features to obtain the pooled features of the audio signal.

[0157] In some embodiments, after upsampling the convolutional features of the audio signal to obtain upsampled features of the audio signal, the second terminal device can call the pooling layer in the synthesis network to perform pooling processing on the upsampled features, for example, performing a pooling operation with a factor of 2 on the upsampled features to obtain the pooled features of the audio signal, for example, see [link to documentation]. Figure 14 If the upsampling feature of the audio signal is a 24×160 tensor, then after pooling (i.e. Figure 14 After the post-processing shown in the figure, a 24×320 tensor (i.e., the pooling feature of the audio signal) is generated.

[0158] In step 3075, the second terminal device performs a second convolution process on the pooled features to obtain the predicted value of the audio signal.

[0159] In some embodiments, after pooling the upsampled features of the audio signal to obtain pooled features of the audio signal, the second terminal device may further call the second convolutional layer (e.g., ...) included in the synthesis network on the pooled features of the audio signal. Figure 14 The causal convolution shown (which performs dilated convolution on pooling features) generates predicted values ​​for the audio signal.

[0160] In other embodiments, when the predicted value of the feature vector includes the predicted value of the feature vector of the low-frequency subband signal and the predicted value of the feature vector of the high-frequency subband signal, the second terminal device can also implement step 307 above in the following way: concatenate the predicted value of the feature vector of the low-frequency subband signal and the first tag information vector (i.e., the tag information vector obtained by tag extraction processing of the predicted value of the feature vector of the low-frequency subband signal) to obtain a first concatenated vector; call the first synthesis network based on the first concatenated vector to reconstruct the signal and obtain the predicted value of the low-frequency subband signal; concatenate the predicted value of the feature vector of the high-frequency subband signal and the second tag information vector (i.e., the tag information vector obtained by tag extraction processing of the predicted value of the feature vector of the high-frequency subband signal) to obtain a second concatenated vector; call the second synthesis network based on the second concatenated vector to reconstruct the signal and obtain the predicted value of the high-frequency subband signal; and synthesize the predicted value of the low-frequency subband signal and the predicted value of the high-frequency subband signal to obtain the predicted value of the audio signal.

[0161] For example, the second terminal device can achieve the above-mentioned signal reconstruction based on the first splicing vector by calling the first synthesis network to obtain the predicted value of the low-frequency subband signal in the following way: calling the first synthesis network to perform the following processing: performing a first convolution processing on the first splicing vector to obtain the convolutional features of the low-frequency subband signal; performing upsampling processing on the convolutional features of the low-frequency subband signal to obtain the upsampled features of the low-frequency subband signal; performing pooling processing on the upsampled features of the low-frequency subband signal to obtain the pooled features of the low-frequency subband signal; performing a second convolution processing on the pooled features of the low-frequency subband signal to obtain the predicted value of the low-frequency subband signal; wherein, the upsampling processing can be implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0162] For example, the second terminal device can achieve the above-mentioned signal reconstruction based on the second splicing vector by calling the second synthesis network to obtain the predicted value of the high-frequency subband signal in the following way: calling the second synthesis network to perform the following processing: performing a first convolution processing on the second splicing vector to obtain the convolutional features of the high-frequency subband signal; performing upsampling processing on the convolutional features of the high-frequency subband signal to obtain the upsampled features of the high-frequency subband signal; performing pooling processing on the upsampled features of the high-frequency subband signal to obtain the pooled features of the high-frequency subband signal; performing a second convolution processing on the pooled features of the high-frequency subband signal to obtain the predicted value of the high-frequency subband signal; wherein, the upsampling processing can be implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0163] It should be noted that the reconstruction process for low-frequency subband signals (i.e., the generation process of predicted values ​​for low-frequency subband signals) and high-frequency subband signals (i.e., the generation process of predicted values ​​for high-frequency subband signals) is similar to the reconstruction process for audio signals (i.e., the generation process of predicted values ​​for audio signals). Please refer to [the relevant documentation / reference]. Figure 6B The description and implementation of the above are not repeated here. Furthermore, it should be noted that the structures of the first and second synthesis networks are similar to those of the synthesis networks described above, and will not be repeated here.

[0164] In other embodiments, when the predicted value of the feature vector includes the predicted values ​​of the feature vectors corresponding to the N sub-band signals respectively, the second terminal device can also implement the above step 307 in the following way: perform one-to-one concatenation processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals and the N tag information vectors to obtain N concatenated vectors; call the j-th synthesis network based on the j-th concatenated vector to reconstruct the signal and obtain the predicted value of the j-th sub-band signal; wherein the value range of j satisfies 1≤j≤N; perform synthesis processing on the predicted values ​​corresponding to the N sub-band signals respectively to obtain the predicted value of the audio signal.

[0165] For example, the second terminal device can achieve the above-mentioned signal reconstruction based on the j-th concatenated vector by calling the j-th synthesis network to obtain the predicted value of the j-th subband signal in the following way: calling the j-th synthesis network to perform the following processing: performing a first convolution processing on the j-th concatenated vector to obtain the convolutional features of the j-th subband signal; performing upsampling processing on the convolutional features of the j-th subband signal to obtain the upsampled features of the j-th subband signal; performing pooling processing on the upsampled features of the j-th subband signal to obtain the pooled features of the j-th subband signal; performing a second convolution processing on the pooled features of the j-th subband signal to obtain the predicted value of the j-th subband signal; wherein, the upsampling processing can be implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0166] It should be noted that the structure of the j-th synthesis network is similar to that of the synthesis network described above, and will not be repeated here in the embodiments.

[0167] In step 308, the second terminal device uses the predicted value of the audio signal obtained through signal reconstruction as the decoding result of the bitstream.

[0168] In some embodiments, after obtaining the predicted value of the audio signal through signal reconstruction, the second terminal device can use the predicted value of the audio signal obtained through signal reconstruction as the decoding result of the bitstream, and send the decoding result to the built-in speaker of the second terminal device for playback.

[0169] The audio decoding method provided in this application extracts labels from the predicted values ​​of the decoded feature vectors to obtain label information vectors for signal enhancement. The method then combines the predicted values ​​of the feature vectors and the label information vectors to reconstruct the signal. Since the label information vectors reflect the core components of the audio signal (i.e., excluding acoustic interference such as noise), compared to reconstructing the signal solely based on the predicted values ​​of the feature vectors, this method combines the predicted values ​​of the feature vectors and the label information vectors to increase the proportion of core components (e.g., human voice) in the audio signal and decrease the proportion of acoustic interference such as noise (e.g., background noise). This effectively suppresses noise components in the audio signal acquired at the encoding end, thereby improving the quality of the reconstructed audio signal.

[0170] The following will use a VoIP conferencing system as an example to illustrate an exemplary application of the embodiments of this application in a real-world application scenario.

[0171] For example, see Figure 7 , Figure 7 This is a schematic diagram of an end-to-end voice communication link provided in an embodiment of this application, such as... Figure 7 As shown, the audio encoding method provided in this application embodiment can be applied at the encoding end (i.e., the sending end of the bitstream), and the audio decoding method provided in this application embodiment can be applied at the decoding end (i.e., the receiving end of the bitstream). This is the core part of the conferencing communication system, solving the basic function of compression. Generally, the encoder is deployed on the uplink client, and the decoder is deployed on the downlink client.

[0172] Furthermore, for backward compatibility, a transcoder also needs to be deployed on the server to address interoperability issues between the new encoder and existing encoders. For example, if the sending end deploys a new NN encoder while the receiving end deploys a traditional Public Switched Telephone Network (PSTN) decoder (e.g., a G.722 decoder), the receiving end will be unable to correctly decode the bitstream directly sent by the sending end. Therefore, after receiving the bitstream from the sending end, the server first needs to execute the NN decoder to generate the voice signal, and then call the G.722 encoder to generate a specific bitstream so that the receiving end can decode it correctly. Similar transcoding scenarios will not be elaborated further.

[0173] The audio encoding and decoding method provided in the embodiments of this application will be described in detail below.

[0174] In some embodiments, see Figure 8 , Figure 8 This is a flowchart illustrating the audio encoding / decoding method provided in an embodiment of this application, as shown below. Figure 8 As shown, the main steps at the encoding end include:

[0175] For the input signal, such as the nth frame of speech signal, denoted as x(n), a feature extraction process is performed using an analysis network to obtain a low-dimensional feature vector, denoted as F(n). Specifically, the dimension of the feature vector F(n) is smaller than the dimension of the input signal x(n), thereby reducing the amount of data. A specific implementation could be to use a dilated convolutional network (DiCNN) to perform feature extraction on the nth frame of speech signal x(n) to generate a lower-dimensional feature vector F(n). It should be noted that this application does not limit other neural network structures, including but not limited to autoencoders (AEs), fully connected networks (FCs), long short-term memory (LSTM) networks, and convolutional neural networks (CNNs) + LSTM. After obtaining the feature vector F(n), it can be vector-quantized or scalar-quantized, and the resulting index values ​​can be entropy-encoded to obtain a bitstream. Finally, the bitstream is transmitted to the decoding end.

[0176] See also Figure 8 The main steps at the decoding end include: decoding the received bitstream to obtain an estimated value of the feature vector, denoted as F′(n). Then, based on the estimated value of the feature vector F′(n), an enhancement network is invoked to generate a label information vector for enhancement, denoted as E(n). Finally, combining the estimated value of the feature vector F′(n) and the label information vector E(n), a synthesis network (corresponding to the inverse process at the encoding end) is invoked to reconstruct the signal and suppress noise components contained in the speech signal acquired at the encoding end, generating a signal estimate corresponding to the input signal x(n), denoted as x′(n).

[0177] To better understand the audio encoding and decoding method provided in this embodiment, before providing a detailed description of the audio encoding and decoding method provided in this embodiment, we will first introduce the dilated convolutional network and the QMF filter bank.

[0178] For example, see Figure 9A and Figure 9B , Figure 9A This is a schematic diagram of a common convolution provided in an embodiment of this application. Figure 9B This is a schematic diagram of dilated convolution provided in an embodiment of this application. Compared to ordinary convolution, dilated convolution was proposed to address the issue of increasing the receptive field while maintaining the same feature map size, thereby avoiding errors caused by upsampling and downsampling. Although Figure 9A and Figure 9BThe kernel sizes shown are all 3×3; however, Figure 9A The receptive field of the ordinary convolution shown is only 3, while Figure 9B The dilated convolution shown has a receptive field of 5. That is, for a convolution kernel of size 3×3, Figure 9A The receptive field shown in the example is 3, and the dilation rate is 1; while Figure 9B The receptive field of the dilated convolution shown is 5, and the dilation rate is 2.

[0179] Convolution kernels can also be used in similar ways. Figure 9A or Figure 9B Moving the kernel on a plane involves the concept of stride rate. For example, if the convolution kernel moves 1 unit at a time, the corresponding stride rate is 1.

[0180] In addition, there is the concept of convolution channel count, which refers to the number of parameters corresponding to the convolution kernels used in convolution analysis. Theoretically, the more channels there are, the more comprehensive the signal analysis and the higher the accuracy; however, the more channels there are, the higher the complexity. For example, for a 1×320 tensor, a 24-channel convolution operation can be used, and the output will be a 24×320 tensor.

[0181] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the size of the convolution kernel is generally 1×3), dilation rate, shift rate, and number of channels can be defined according to actual application needs. This application does not impose specific limitations on these aspects.

[0182] The QMF filter bank will now be explained.

[0183] A QMF filter bank is a filter pair that includes analysis and synthesis. For a QMF analysis filter, an input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, representing the QMF low-pass signal and the QMF high-pass signal, respectively. For example... Figure 10 The figure shows the spectral responses of the low-pass component H_Low(z) and high-pass component H_High(z) of the QMF analysis filter bank. Based on the relevant theoretical knowledge of QMF analysis filter banks, the correlation between the coefficients of the low-pass and high-pass filters can be easily described:

[0184] h High (k)=-1 k h Low (k)

[0185] Among them, h Low (k) represents the coefficients of the low-pass filter, h High (k) represents the coefficient of the high-pass filter.

[0186] Similarly, based on QMF-related theories, QMF synthesized filter banks can also be described based on QMF analysis filter banks H_Low(z) and H_High(z). The detailed mathematical background will not be repeated here.

[0187] G Low (z)=H Low (z)

[0188] G High (z)=(-1)*H High (z)

[0189] Among them, G Low (z) represents the recovered low-pass signal, G High (z) represents the recovered high-pass signal.

[0190] After the low-pass and high-pass signals are recovered at the decoding end, they are synthesized by the QMF synthesis filter bank, which can then recover the reconstructed signal corresponding to the sampling rate Fs of the input signal.

[0191] In addition to the above-mentioned 2-channel QMF scheme, it can also be extended to an N-channel QMF scheme; in particular, a binary method can be used to iteratively perform 2-channel QMF analysis on the current subband signal to obtain a subband signal with lower resolution. Figure 11A This indicates that a two-channel QMF analysis filter with two iterative layers can obtain a four-channel subband signal. Figure 11B Another approach is to consider that the high-frequency signal has little impact on quality and does not require such high-precision analysis; therefore, only a single high-pass filter needs to be applied to the original signal. Similarly, more channels can be implemented, such as 8, 16, or 32 channels, which will not be discussed further here.

[0192] The audio encoding and decoding method provided in the embodiments of this application will be described in detail below.

[0193] In some embodiments, a speech signal with a sampling rate of Fs = 16000Hz is taken as an example (it should be noted that the method provided in this application embodiment is also applicable to other sampling rate scenarios, including but not limited to: 8000Hz, 32000Hz, 48000Hz). Meanwhile, assuming the frame length is set to 20ms, for Fs = 16000Hz, this is equivalent to each frame containing 320 sample points.

[0194] The following will combine Figure 8 The flowchart shown illustrates the audio encoding and decoding method, providing detailed explanations of the processes at both the encoding and decoding ends.

[0195] (I) The process for the encoding end is as follows:

[0196] Step 1: Generation of input signal

[0197] As mentioned earlier, for a speech signal with a sampling rate of Fs = 16000Hz, assuming a frame length of 20ms, the speech signal of the nth frame includes 320 sample points, denoted as the input signal x(n).

[0198] Step 2: Use the analysis network to compress the data.

[0199] The purpose of the analysis network is to generate a lower-dimensional feature vector F(n) based on the input signal x(n) by calling the analysis network (e.g., a neural network). In this embodiment, the dimension of the input signal x(n) is 320, and the dimension of the feature vector F(n) is 56. From the perspective of data volume, after feature extraction by the analysis network, it plays a role in "dimensionality reduction" and achieves the function of data compression.

[0200] For example, see Figure 12 , Figure 12 This is a schematic diagram of the analysis network structure provided in the embodiments of this application, such as... Figure 12 As shown, a 24-channel causal convolution is first invoked to expand the input signal x(n) (where x(n) is a 1×320 tensor) into a 24×320 tensor. Next, the expanded 24×320 tensor is preprocessed. For example, a pooling operation with a factor of 2 can be performed on the expanded 24×320 tensor, and the activation function can be a Linear Rectification Function (ReLU) to generate a 24×160 tensor. Next, three encoding blocks with different downsampling factors can be concatenated. Taking an encoding block (Down_factor = 4) as an example, one or more dilated convolutions can be performed first, with each convolution kernel having a fixed size of 1×3 and a stride rate of 1. Furthermore, the dilation rate of the one or more dilated convolutions can be set according to requirements, such as 3. Of course, this embodiment does not limit different dilation rates to different dilated convolutions. Then, the down_factor of the three encoding blocks is set to 4, 5, and 8 respectively, which is equivalent to setting pooling factors of different sizes, thus achieving downsampling. Finally, the number of channels in the three encoding blocks is set to 48, 96, and 192 respectively. Thus, after downsampling through the three encoding blocks, the 24×160 tensor will be successively transformed into tensors of 48×40, 96×8, and 192×1. Finally, the 192×1 tensor is subjected to a causal convolution similar to preprocessing, which outputs a 56-dimensional feature vector F(n).

[0201] Step 3, quantization encoding

[0202] The feature vector F(n) extracted at the encoding end can be quantized and encoded using scalar quantization (i.e., quantizing each component individually) and entropy encoding. Alternatively, vector quantization (i.e., combining multiple adjacent components into a single vector for joint quantization) and entropy encoding can also be used; this embodiment does not specifically limit the method used.

[0203] After quantizing and encoding the feature vector F(n), a bitstream can be generated. According to experiments, high-quality compression of a 16kHz broadband signal can be achieved with a bitrate of 6-8kbps.

[0204] (II) The decoding process is as follows:

[0205] Step 1, Decoding

[0206] Decoding is the inverse process of encoding. For the received bitstream, decoding is performed, and then the estimated value of the feature vector, denoted as F′(n), is obtained by looking up the quantization table based on the index value obtained from decoding.

[0207] Step 2: Call the augmented network to extract label information vectors.

[0208] The estimated feature vector F′(n) contains a compressed version of the original speech signal acquired at the encoder, reflecting the core components of the speech signal, while also including acoustic interference such as noise introduced during acquisition. Therefore, an augmentation network is used to extract relevant label embedding information from the estimated feature vector F′(n) to generate a relatively clean speech signal during decoding.

[0209] For example, see Figure 13 , Figure 13 This is a schematic diagram of the structure of the augmented network provided in the embodiments of this application, such as... Figure 13 As shown, using the estimated feature vector F′(n) as input, a one-dimensional causal convolution is applied to generate a 56×1 tensor. Then, this 56×1 tensor is passed through an LSTM network to generate another 56×1 tensor. Next, a fully connected (FC) network is applied to generate another 56×1 tensor. Finally, an activation function (such as ReLU, or other activation functions like Sigmoid or Tanh) is applied for activation processing. This generates a label information vector with the same dimensions as the estimated feature vector F′(n), denoted as E(n).

[0210] Step 3: Use the synthesis network to reconstruct the signal.

[0211] The purpose of the synthesis network is to concatenate the estimated value F′(n) of the feature vector obtained from the decoder and the locally generated label information vector E(n) into a 112-dimensional vector, and then call the synthesis network to reconstruct the signal, generating an estimated value of the speech signal, denoted as x′(n). It should be noted that generating the input vector through concatenation is only one method used by the synthesis network; this application does not limit other methods. For example, F′(n) + E(n) can be used as input, resulting in a dimension of 56. For this method, please refer to... Figure 14 The network can be redesigned; the specific implementation details are omitted here.

[0212] For example, see Figure 14 , Figure 14 This is a schematic diagram of the structure of the synthetic network provided in the embodiments of this application, such as... Figure 14 As shown, the structure of the synthesis network is highly similar to that of the analysis network, for example, using causal convolution; however, the input dimension increases to 112 dimensions. The post-processing process is similar to the pre-processing in the analysis network. Furthermore, the structure of the decoding block (also known as the decoding layer) is symmetrical to that of the encoding block (also known as the encoding layer) in the analysis network. For example, the encoding block in the analysis network performs dilated convolution followed by pooling to achieve downsampling, while the decoding block in the synthesis network performs pooling to achieve upsampling before performing dilated convolution. In other words, decoding is the reverse process of encoding. (Refer to...) Figure 12 The description of the embodiments in this application will not be repeated here.

[0213] In this embodiment, optimal parameters can be obtained by jointly training the relevant networks (e.g., analysis and synthesis networks) at the encoding and decoding ends through data collection. Currently, many open-source platforms for neural networks and deep learning are publicly available. Based on these platforms, users only need to prepare data and set the corresponding network structure. After training is completed on the server, the trained network can be put into use. The above embodiments of this application assume that the parameters of the analysis and synthesis networks have already been trained, and only disclose a specific implementation of network input, network structure, and network output. Engineers in related fields can further modify the above configuration according to actual conditions.

[0214] In the above embodiments, for the input signal, the analysis network, enhancement network, and synthesis network are invoked respectively along the encoding and decoding path to complete low bit rate compression and signal reconstruction. However, these networks are highly complex. To reduce complexity, this embodiment can introduce a QMF analysis filter to decompose the input signal into sub-band signals with lower bit rates; then, for each sub-band signal, the input and output dimensions of the neural network will be at least halved. Generally, the computational complexity of neural networks is O(N). 3 Therefore, this "divide and conquer" approach can effectively reduce complexity.

[0215] For example, see Figure 15 , Figure 15 This is a flowchart illustrating the audio encoding / decoding method provided in an embodiment of this application, as shown below. Figure 15 As shown, for the input signal x(n) in the nth frame, the QMF analysis filter decomposes it into two sub-band signals. For example, after the input signal x(n) is decomposed by the QMF analysis filter, the low-frequency sub-band signal can be obtained, denoted as x. LB (n) and the high-frequency sub-band signal, denoted as x HB (n). Next, for the low-frequency subband signal x LB (n) can call the first analysis network to obtain the feature vector of the low-dimensional low-frequency subband signal, denoted as F. LB (n). In particular, the eigenvector F of the low-frequency subband signal. LB The dimension of (n) is smaller than that of the low-frequency subband signal x. LB (n), thus reducing the amount of data. Furthermore, it should be noted that due to the low-frequency subband signal x... LB The resolution of x(n) is half that of the input signal x(n), therefore the parameters of the first analysis network can be halved accordingly, including the feature vector F of the low-frequency subband signal. LB (n).

[0216] After obtaining the eigenvector F of the low-frequency subband signal LB After (n), the eigenvector F of the low-frequency subband signal can be... LB (n) Perform vector quantization or scalar quantization, and entropy encode the index value obtained after quantization to obtain the bit stream, which is then transmitted to the decoding end.

[0217] After receiving the bitstream from the encoder, the decoder can decode the received bitstream to obtain an estimate of the feature vector of the low-frequency subband signal, denoted as F′. LB (n). Next, the estimated value F′ of the eigenvector of the low-frequency subband signal can be used. LB (n), invoke the first enhancement network to generate the tag information vector corresponding to the low-frequency subband signal, denoted as E. LB (n). Finally, combining F′ LB (n) and E LB (n), calling the first synthesis network of the inverse process of the corresponding encoding end to complete the estimation of the low-frequency subband signal, denoted as x′. LB The reconstruction of (n) is performed, and acoustic interference such as noise in the speech signal acquired at the encoding end is suppressed. For ease of description, the functions of the first enhancement network and the first synthesis network are combined into the first synthesis module in the following text, that is, at the decoding end, based on F′ LB (n) and E LB(n), by calling the first synthesis module to reconstruct the signal, the estimated value x′ of the low-frequency subband signal can be obtained. LB (n).

[0218] Similarly, the high-frequency subband signal obtained after decomposing the input signal x(n) through a QMF analysis filter is denoted as x. HB (n), in the encoding and decoding process, the second analysis network and the second synthesis module (including the second enhancement network and the second synthesis network) are called respectively, and the estimated value of the high-frequency subband signal can be obtained at the decoding end, denoted as x′. HB (n). It should be noted that this applies to the high-frequency subband signal x. HB The processing flow of (n) and the low-frequency subband signal x LB The processing flow for (n) is similar; you can refer to the low-frequency subband signal x. LB The processing flow of (n) is implemented, and will not be described again in this embodiment.

[0219] Referring to the processing example of the 2-channel QMF above, and the multi-channel QMF introduced above, the features completed by iterating through the 2-channel QMF can be further extended to, for example... Figure 16 The multi-channel QMF scheme shown (for example, the input signal x(n) can be decomposed into N sub-band signals, and each sub-band signal can be encoded and decoded separately) is similar in principle, so the embodiments of this application will not be described again here.

[0220] The following uses a 2-channel QMF example to illustrate the audio encoding and decoding method provided in the embodiments of this application.

[0221] (I) The process for the encoding end is as follows:

[0222] Step 1: Generation of input signal

[0223] As mentioned earlier, for a speech signal with a sampling rate Fs = 16000Hz, assuming a frame length of 20ms, the speech signal of the nth frame includes 320 sample points, denoted as the input signal xln).

[0224] Step 2, QMF signal decomposition

[0225] As mentioned earlier, for the input signal x(n), a QMF analysis filter (specifically a 2-channel QMF) can be invoked and downsampled to obtain two sub-band signals: the low-frequency sub-band signal x. LB (n) and high-frequency subband signal x HB (n). Among them, the low-frequency subband signal x LB The effective bandwidth of (n) is 0-4kHz, and the high-frequency subband signal x HBThe effective bandwidth of (n) is 4-8kHz, and the number of sample points per frame is 160.

[0226] Step 3: Use the first and second analysis networks to compress the data.

[0227] For example, when decomposing the input signal x(n) into low-frequency subband signals x... LB (n) and high-frequency subband signal x HB (n) followed by the low-frequency subband signal x LB (n) can be called as follows Figure 17 The first analysis network shown performs feature extraction processing to obtain the feature vector F of the low-frequency subband signal. LB (n); similarly, for high-frequency subband signals x HB (n) can be used to call the second analysis network for feature extraction, obtaining the feature vector of the high-frequency subband signal, denoted as F. HB (n).

[0228] It should be noted that since the sampling rate of the subband signal is halved compared to the input signal, in this embodiment, the dimension of the feature vector of the output subband signal can be lower than the dimension of the feature vector of the input signal in the above embodiments. For example, in this embodiment, the dimension of the feature vectors of both the low-frequency subband signal and the high-frequency subband signal can be set to 28. In this way, the dimension of the overall output feature vector is consistent with the dimension of the feature vector of the input signal in the above embodiments, that is, the bit rates of the two are consistent.

[0229] Furthermore, considering that low and high frequencies have different impact factors on speech quality, this application embodiment does not limit the definition of different numbers of dimensions for the feature vectors of different sub-band signals. For example, the dimension of the feature vector of the low-frequency sub-band signal can be set to 32, while the dimension of the feature vector of the high-frequency sub-band signal can be set to 24, thus still ensuring that the total dimension is consistent with the dimension of the feature vector of the input signal. This can be achieved by adjusting the internal parameters of the first and second analysis networks accordingly, which will not be elaborated further in this application embodiment.

[0230] Step 4, quantization encoding

[0231] Similar to the processing of the feature vectors of the input signal, considering that the dimension of the total feature vector remains unchanged, high-quality compression of 16kHz wideband signals can be achieved with a bit rate of 6-8kbps.

[0232] (II) The decoding process is as follows:

[0233] Step 1, Decoding

[0234] Similar to the above embodiments, by decoding the received bitstream, the estimated value F′ of the feature vector of the low-frequency subband signal can be obtained. LB The estimated values ​​of (n) and the eigenvectors of the high-frequency subband signals, F′ HB (n).

[0235] Step 2: Use the first and second augmentation networks to extract the label information vector.

[0236] For example, when decoding the received bitstream, the estimated value F′ of the feature vector of the low-frequency subband signal is obtained. LB The estimated values ​​of (n) and the eigenvectors of the high-frequency subband signals, F′ HB (n) followed by the estimated value F′ of the eigenvector for the low-frequency subband signal. LB (n) can be called as follows Figure 18 The first enhancement network shown collects label embedding information (i.e., the label information vector of the low-frequency part) for low-frequency speech enhancement, denoted as E. LB (n) is used to generate a relatively clean low-frequency subband speech signal during decoding. The calculation process described above can be found in [reference needed]. Figure 13 The implementation details are omitted here. Furthermore, due to the halved sampling rate, the dimension of the output feature vector of the first analysis network in the encoding stage can be adjusted accordingly. Figure 18 The structure (including parameters) of the first augmentation network is shown.

[0237] Similarly, the estimated value F′ of the eigenvector of the decoded high-frequency subband signal... HB (n) can be used to call the second enhancement network to obtain the label information vector of the high-frequency part, denoted as E. HB (n) is used in subsequent processes.

[0238] In summary, after this step, we can obtain the tag information vectors of two sub-band signals, namely the tag information vector E of the low-frequency part. LB (n) and the tag information vector E of the high-frequency part HB (n).

[0239] Step 3: Use the first and second synthesis networks to reconstruct the signal.

[0240] For example, see Figure 19 , Figure 19 This is a schematic diagram of the structure of the first synthetic network provided in the embodiments of this application, as shown below. Figure 19 As shown, the first synthesis network can be invoked to estimate the value F′ based on the feature vector of the low-frequency subband signal. LB (n) and the locally generated low-frequency part of the label information vector E LB(n), the estimated value of the generated low-frequency subband signal, denoted as x′ LB (n). For detailed calculation procedures, please refer to... Figure 14 The description of the above will not be repeated here in the embodiments. Furthermore, due to the halving of the sampling rate, Figure 19 Only a specific configuration of the first synthesis network corresponding to the low-frequency part is provided. The implementation of the high-frequency part is similar and will not be described in detail here.

[0241] This step generates an estimate of the low-frequency subband signal, x′. LB (n) and the estimated value x′ of the high-frequency subband signal HB (n). In particular, acoustic interference such as noise in these two sub-band signals was effectively suppressed.

[0242] Step 4: Perform synthesis processing based on the QMF synthesis filter.

[0243] Based on the first two steps, the estimated value x′ of the low-frequency subband signal is obtained at the decoding end. LB (n) and the estimated value x′ of the high-frequency subband signal HB After (n), simply upsampling and calling the QMF synthesis filter will generate a 320-point reconstructed signal, which is the estimated value x′(n) of the input signal x(n), thus completing the entire decoding process.

[0244] In summary, the embodiments of this application, through the organic combination of signal decomposition and related signal processing techniques with deep neural networks, significantly improve coding efficiency compared to traditional signal processing schemes. With acceptable complexity, speech enhancement is implemented at the decoding end, enabling the reconstruction of clean speech at a low bit rate even under acoustic interference such as noise. For example, see... Figure 20 The voice signal collected at the encoding end contains a large amount of noise interference. Through the voice enhancement and ultra-low bit rate compression scheme provided in this application embodiment, a clean voice signal can be reconstructed at the decoding end, thereby improving the quality of voice calls.

[0245] The following description continues to illustrate the exemplary structure of the audio decoding device 565 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 3 As shown, the software modules stored in the audio decoding device 565 in the memory 560 may include: an acquisition module 5651, a decoding module 5652, a tag extraction module 5653, a reconstruction module 5654, and a determination module 5655.

[0246] The acquisition module 5651 is used to acquire the bitstream, which is obtained by encoding the audio signal; the decoding module 5652 is used to decode the bitstream to obtain the predicted value of the feature vector of the audio signal; the tag extraction module 5653 is used to extract tags from the predicted value of the feature vector to obtain a tag information vector for signal enhancement, wherein the dimension of the tag information vector is the same as the dimension of the predicted value of the feature vector; the reconstruction module 5654 is used to reconstruct the signal based on the predicted value of the feature vector and the tag information vector; and the determination module 5655 is used to use the predicted value of the audio signal obtained through signal reconstruction as the decoding result of the bitstream.

[0247] In some embodiments, the decoding module 5652 is further configured to decode the bitstream to obtain the index value of the feature vector of the audio signal; and query the quantization table based on the index value to obtain the predicted value of the feature vector of the audio signal.

[0248] In some embodiments, the label extraction module 5653 is further configured to perform convolution processing on the predicted value of the feature vector to obtain a first tensor with the same dimension as the predicted value of the feature vector; perform feature extraction processing on the first tensor to obtain a second tensor with the same dimension as the first tensor; perform fully connected processing on the second tensor to obtain a third tensor with the same dimension as the second tensor; and perform activation processing on the third tensor to obtain a label information vector for signal enhancement.

[0249] In some embodiments, the reconstruction module 5654 is further configured to concatenate the predicted value of the feature vector and the label information vector to obtain a concatenated vector; perform a first convolution on the concatenated vector to obtain convolutional features of the audio signal; perform upsampling on the convolutional features to obtain upsampled features of the audio signal; perform pooling on the upsampled features to obtain pooled features of the audio signal; and perform a second convolution on the pooled features to obtain the predicted value of the audio signal.

[0250] In some embodiments, upsampling is implemented through multiple cascaded decoding layers, with different sampling factors in each layer. The reconstruction module 5654 is further configured to perform upsampling on the convolutional features through the first decoding layer in the multiple cascaded decoding layers; output the upsampling result of the first decoding layer to the subsequent cascaded decoding layers, and continue upsampling and outputting upsampling results through the subsequent cascaded decoding layers until the last decoding layer is output; and use the upsampling result output by the last decoding layer as the upsampling feature of the audio signal.

[0251] In some embodiments, the bitstream includes a low-frequency bitstream and a high-frequency bitstream, wherein the low-frequency bitstream is obtained by encoding the low-frequency sub-band signal obtained after decomposing the audio signal, and the high-frequency bitstream is obtained by encoding the high-frequency sub-band signal obtained after decomposing the audio signal; the decoding module 5652 is further configured to decode the low-frequency bitstream to obtain a predicted value of the feature vector of the low-frequency sub-band signal; and to decode the high-frequency bitstream to obtain a predicted value of the feature vector of the high-frequency sub-band signal.

[0252] In some embodiments, the tag extraction module 5653 is further configured to perform tag extraction processing on the predicted values ​​of the feature vectors of the low-frequency subband signal to obtain a first tag information vector for low-frequency subband signal enhancement, wherein the dimension of the first tag information vector is the same as the dimension of the predicted values ​​of the feature vectors of the low-frequency subband signal; and to perform tag extraction processing on the predicted values ​​of the feature vectors of the high-frequency subband signal to obtain a second tag information vector for high-frequency subband signal enhancement, wherein the dimension of the second tag information vector is the same as the dimension of the predicted values ​​of the feature vectors of the high-frequency subband signal.

[0253] In some embodiments, the tag extraction module 5653 is further configured to invoke the first enhancement network to perform the following processing: convolving the predicted value of the feature vector of the low-frequency subband signal to obtain a fourth tensor with the same dimension as the predicted value of the feature vector of the low-frequency subband signal; performing feature extraction processing on the fourth tensor to obtain a fifth tensor with the same dimension as the fourth tensor; performing fully connected processing on the fifth tensor to obtain a sixth tensor with the same dimension as the fifth tensor; and performing activation processing on the sixth tensor to obtain a first tag information vector for low-frequency subband signal enhancement.

[0254] In some embodiments, the tag extraction module 5653 is further configured to invoke the second enhancement network to perform the following processing: convolving the predicted value of the feature vector of the high-frequency subband signal to obtain a seventh tensor with the same dimension as the predicted value of the feature vector of the high-frequency subband signal; performing feature extraction processing on the seventh tensor to obtain an eighth tensor with the same dimension as the seventh tensor; performing fully connected processing on the eighth tensor to obtain a ninth tensor with the same dimension as the eighth tensor; and performing activation processing on the ninth tensor to obtain a second tag information vector for high-frequency subband signal enhancement.

[0255] In some embodiments, the predicted value of the feature vector includes: the predicted value of the feature vector of the low-frequency subband signal and the predicted value of the feature vector of the high-frequency subband signal; the reconstruction module 5654 is further configured to concatenate the predicted value of the feature vector of the low-frequency subband signal and the first tag information vector to obtain a first concatenated vector; call a first synthesis network based on the first concatenated vector to perform signal reconstruction to obtain the predicted value of the low-frequency subband signal; concatenate the predicted value of the feature vector of the high-frequency subband signal and the second tag information vector to obtain a second concatenated vector; call a second synthesis network based on the second concatenated vector to perform signal reconstruction to obtain the predicted value of the high-frequency subband signal; and synthesize the predicted value of the low-frequency subband signal and the predicted value of the high-frequency subband signal to obtain the predicted value of the audio signal.

[0256] In some embodiments, the reconstruction module 5654 is further configured to invoke the first synthesis network to perform the following processing: perform a first convolution processing on the first concatenated vector to obtain convolutional features of the low-frequency subband signal; perform upsampling processing on the convolutional features to obtain upsampled features of the low-frequency subband signal; perform pooling processing on the upsampled features to obtain pooled features of the low-frequency subband signal; and perform a second convolution processing on the pooled features to obtain the predicted value of the low-frequency subband signal; wherein the upsampling processing is implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0257] In some embodiments, the reconstruction module 5654 is further configured to invoke the second synthesis network to perform the following processing: perform a first convolution processing on the second concatenated vector to obtain the convolutional features of the high-frequency subband signal; perform upsampling processing on the convolutional features to obtain the upsampled features of the high-frequency subband signal; perform pooling processing on the upsampled features to obtain the pooled features of the high-frequency subband signal; and perform a second convolution processing on the pooled features to obtain the predicted value of the high-frequency subband signal; wherein the upsampling processing is implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0258] In some embodiments, the bitstream includes N sub-bitstreams, each corresponding to a different frequency band, and is obtained by encoding the N sub-band signals obtained after decomposing the audio signal, where N is an integer greater than 2; the decoding module 5652 is further used to decode the N sub-bitstreams respectively to obtain the predicted values ​​of the feature vectors corresponding to the N sub-band signals respectively.

[0259] In some embodiments, the label extraction module 5653 is further configured to perform label extraction processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals respectively, to obtain N label information vectors for signal enhancement, wherein the dimension of each label information vector is the same as the dimension of the predicted value of the feature vector of the corresponding sub-band signal.

[0260] In some embodiments, the tag extraction module 5653 is further configured to call the i-th enhancement network to perform tag extraction processing based on the predicted value of the feature vector of the i-th sub-band signal, so as to obtain the i-th tag information vector for the enhancement of the i-th sub-band signal; wherein the value range of i satisfies 1≤i≤N, and the dimension of the i-th tag information vector is the same as the dimension of the predicted value of the feature vector of the i-th sub-band signal.

[0261] In some embodiments, the label extraction module 5653 is further configured to call the i-th enhancement network to perform the following processing: convolving the predicted value of the feature vector of the i-th sub-band signal to obtain a tenth tensor with the same dimension as the predicted value of the feature vector of the i-th sub-band signal; performing feature extraction processing on the tenth tensor to obtain an eleventh tensor with the same dimension as the tenth tensor; performing fully connected processing on the eleventh tensor to obtain a twelfth tensor with the same dimension as the eleventh tensor; and performing activation processing on the twelfth tensor to obtain the i-th label information vector for the enhancement of the i-th sub-band signal.

[0262] In some embodiments, the reconstruction module 5654 is further configured to perform one-to-one splicing processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals and the N tag information vectors to obtain N spliced ​​vectors; call the j-th synthesis network based on the j-th spliced ​​vector to reconstruct the signal and obtain the predicted value of the j-th sub-band signal; wherein the value range of j satisfies 1≤j≤N; and perform synthesis processing on the predicted values ​​corresponding to the N sub-band signals to obtain the predicted value of the audio signal.

[0263] In some embodiments, the reconstruction module 5654 is further configured to invoke the j-th synthesis network to perform the following processing: perform a first convolution processing on the j-th concatenated vector to obtain the convolutional features of the j-th subband signal; perform upsampling processing on the convolutional features to obtain the upsampled features of the j-th subband signal; perform pooling processing on the upsampled features to obtain the pooled features of the j-th subband signal; and perform a second convolution processing on the pooled features to obtain the predicted value of the j-th subband signal; wherein the upsampling processing is implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0264] It should be noted that the description of the apparatus in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments; therefore, it will not be repeated. For any technical details not covered in the audio decoding apparatus provided in this application, please refer to... Figure 4C , Figure 6A ,or Figure 6B The description of any of the accompanying drawings is for reference only.

[0265] This embodiment provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the audio encoding / decoding method described above in this embodiment.

[0266] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the audio encoding / decoding method provided in this application. For example, ... Figure 4C The audio encoding and decoding methods are shown.

[0267] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0268] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0269] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0270] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0271] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An audio decoding method, characterized by, The method comprises: acquiring a code stream, wherein the code stream is obtained by encoding an audio signal; decoding the code stream to obtain a predicted value of a feature vector of the audio signal; performing label extraction processing on the predicted value of the feature vector through an enhancement network to obtain a label information vector for signal enhancement; performing splicing processing on the predicted value of the feature vector and the label information vector to obtain a spliced vector; performing first convolution processing on the spliced vector to obtain a convolution feature of the audio signal; performing up-sampling processing on the convolution feature to obtain an up-sampled feature of the audio signal; performing pooling processing on the up-sampled feature to obtain a pooled feature of the audio signal; performing second convolution processing on the pooled feature to obtain a predicted value of the audio signal; taking the predicted value of the audio signal obtained through signal reconstruction as a decoding result of the code stream.

2. The method of claim 1, wherein the enhancement network comprises a convolution layer, a neural network layer, a fully connected network layer, and an activation layer; the label extraction processing on the predicted value of the feature vector through the enhancement network to obtain the label information vector for signal enhancement comprises: performing convolution processing on the predicted value of the feature vector through the convolution layer to obtain an output of the convolution layer; performing feature extraction processing on the output of the convolution layer through the neural network layer to obtain an output of the neural network layer; performing full connection processing on the output of the neural network layer through the fully connected network layer to obtain an output of the fully connected network layer; performing activation processing on the output of the fully connected network layer through the activation layer to obtain the label information vector for signal enhancement, wherein the output of the convolution layer, the output of the neural network layer, the output of the fully connected network layer, and the predicted value of the feature vector have the same dimension.

3. The method of claim 1, wherein the up-sampling processing is implemented through a plurality of cascaded decoding layers, and different decoding layers have different sampling factors; the up-sampling processing on the convolution feature to obtain the up-sampled feature of the audio signal comprises: performing up-sampling processing on the convolution feature through a first decoding layer in the plurality of cascaded decoding layers; outputting the up-sampling result of the first decoding layer to a subsequent cascaded decoding layer, and continuing to perform up-sampling processing and up-sampling result output through the subsequent cascaded decoding layer until outputting to a last decoding layer; outputting the up-sampling result of the last decoding layer as the up-sampled feature of the audio signal.

4. The method of claim 1, wherein, the decoding processing on the code stream to obtain the predicted value of the feature vector of the audio signal comprises: performing decoding processing on the code stream to obtain an index value of the feature vector of the audio signal; querying a quantization table based on the index value to obtain the predicted value of the feature vector of the audio signal.

5. The method of claim 1, wherein The code stream comprises N sub-code streams, the N sub-code streams correspond to different frequency bands, and are obtained by encoding N sub-band signals obtained by decomposing the audio signal, N is an integer greater than 1; The decoding processing of the code stream obtains the predicted value of the feature vector of the audio signal, comprising: The N sub-code streams are decoded respectively to obtain the predicted value of the feature vector corresponding to the N sub-band signals respectively.

6. The method of claim 5, wherein, The number of the enhancement networks is N, and the enhancement networks correspond one-to-one to the predicted values of the feature vectors corresponding to the sub-band signals; The label extraction processing of the predicted value of the feature vector by the enhancement network obtains the label information vector for signal enhancement, comprising: Based on the predicted value of the feature vector of the i-th sub-band signal, an i-th enhancement network is called to perform label extraction processing to obtain an i-th label information vector for i-th sub-band signal enhancement, wherein the value range of i satisfies 1≤i≤N.

7. The method of claim 6, wherein, The i-th enhancement network comprises a convolution layer, a neural network layer, a fully connected network layer and an activation layer; The label extraction processing of the predicted value of the feature vector of the i-th sub-band signal by the i-th enhancement network to obtain the i-th label information vector for i-th sub-band signal enhancement, comprising: Based on the predicted value of the feature vector of the i-th sub-band signal, the i-th enhancement network is called to perform the following processing: The predicted value of the feature vector of the i-th sub-band signal is convoluted by the convolution layer to obtain the output of the convolution layer; The output of the convolution layer is feature extracted by the neural network layer to obtain the output of the neural network layer; The output of the neural network layer is fully connected by the fully connected network layer to obtain the output of the fully connected network layer; The output of the fully connected network layer is activated by the activation layer to obtain the i-th label information vector for i-th sub-band signal enhancement, wherein the dimensions of the output of the convolution layer, the output of the neural network layer, the output of the fully connected network layer and the predicted value of the feature vector are the same.

8. The method according to any one of claims 5 to 7, characterized in that, The method further comprises: The predicted values of the feature vectors corresponding to the N sub-band signals and the N label information vectors are one-to-one spliced to obtain N splicing vectors; Based on the j-th splicing vector, a j-th synthesis network is called to reconstruct the signal to obtain the predicted value of the j-th sub-band signal; wherein the value range of j satisfies 1≤j≤N; The predicted values of the N sub-band signals are synthesized to obtain the predicted value of the audio signal.

9. The method of claim 8, wherein, Based on the j-th splicing vector, the j-th synthesis network is called to reconstruct the signal to obtain the predicted value of the j-th sub-band signal, comprising: Based on the j-th splicing vector, the j-th synthesis network is called to perform the following processing: The j-th splicing vector is first convoluted to obtain the convolution feature of the j-th sub-band signal; The convolution feature is up-sampled to obtain the up-sampled feature of the j-th sub-band signal; pooling processing is performed on the up-sampling feature to obtain a pooling feature of the jth sub-band signal; second convolution processing is performed on the pooling feature to obtain a prediction value of the jth sub-band signal; wherein the up-sampling processing is implemented through a plurality of cascaded decoding layers, and sampling factors of different decoding layers are different.

10. The method of claim 9, wherein the up-sampling processing is implemented through a plurality of cascaded decoding layers, and sampling factors of different decoding layers are different; the up-sampling processing on the convolution feature to obtain the up-sampling feature of the jth sub-band signal comprises: up-sampling processing is performed on the convolution feature through a first decoding layer in the plurality of cascaded decoding layers; the up-sampling result of the first decoding layer is output to a subsequent cascaded decoding layer, and up-sampling processing and up-sampling result output are continued through the subsequent cascaded decoding layer until the last decoding layer is output; the up-sampling result output by the last decoding layer is taken as the up-sampling feature of the jth sub-band signal.

11. An audio encoding method, characterized by, The method comprises: obtaining an audio signal; performing encoding processing on the audio signal to obtain a bitstream, wherein the bitstream is used for an electronic device to perform the audio decoding method of any one of claims 1 to 10.

12. An audio decoding apparatus, characterized by comprising: The apparatus comprises: an obtaining module configured to obtain a bitstream, wherein the bitstream is obtained by encoding an audio signal; a decoding module configured to perform decoding processing on the bitstream to obtain a prediction value of a feature vector of the audio signal; a label extraction module configured to perform label extraction processing on the prediction value of the feature vector through an enhancement network to obtain a label information vector used for signal enhancement; a reconstruction module configured to perform splicing processing on the prediction value of the feature vector and the label information vector to obtain a splicing vector, perform first convolution processing on the splicing vector to obtain a convolution feature of the audio signal, perform up-sampling processing on the convolution feature to obtain an up-sampling feature of the audio signal, perform pooling processing on the up-sampling feature to obtain a pooling feature of the audio signal, and perform second convolution processing on the pooling feature to obtain a prediction value of the audio signal; a determining module configured to take the prediction value of the audio signal obtained through signal reconstruction as a decoding result of the bitstream.

13. The apparatus of claim 12, wherein, The enhancement network comprises a convolution layer, a neural network layer, a fully connected network layer, and an activation layer; the label extraction module is further configured to perform convolution processing on the prediction value of the feature vector through the convolution layer to obtain an output of the convolution layer, perform feature extraction processing on the output of the convolution layer through the neural network layer to obtain an output of the neural network layer, and perform full connection processing on the output of the neural network layer through the fully connected network layer to obtain an output of the fully connected network layer; activation processing is performed on the output of the fully connected network layer through the activation layer to obtain a label information vector used for signal enhancement, wherein the output of the convolution layer, the output of the neural network layer, the output of the fully connected network layer, and the prediction value of the feature vector have the same dimension.

14. The apparatus of claim 12, wherein, The upsampling processing is implemented through a plurality of cascaded decoding layers, and sampling factors of different decoding layers are different; The reconstruction module is further configured to perform upsampling processing on the convolution feature through a first decoding layer in the plurality of cascaded decoding layers; output an upsampling result of the first decoding layer to a subsequent cascaded decoding layer, and continue to perform upsampling processing and upsampling result output through the subsequent cascaded decoding layer until the last decoding layer; and output the upsampling result output by the last decoding layer as the upsampling feature of the audio signal.

15. The apparatus of claim 12, wherein, The decoding module is further configured to perform decoding processing on the code stream to obtain an index value of the feature vector of the audio signal; and query a quantization table based on the index value to obtain a prediction value of the feature vector of the audio signal.

16. The apparatus of claim 15, wherein, The code stream includes N sub-code streams, the N sub-code streams correspond to different frequency bands, and are obtained by respectively encoding N sub-band signals obtained by performing decomposition processing on the audio signal, and N is an integer greater than 1. The decoding module is further configured to perform decoding processing on the N sub-code streams respectively to obtain prediction values of feature vectors corresponding to the N sub-band signals respectively.

17. An audio encoding apparatus characterized by comprising: The apparatus includes: An acquisition module configured to acquire an audio signal; An encoding module configured to perform encoding processing on the audio signal to obtain a code stream, wherein the code stream is used for an electronic device to perform an audio decoding method according to any one of claims 1 to 10.

18. A computer-readable storage medium storing executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform operations comprising: The executable instructions, when executed by the processor, implement the audio decoding method according to any one of claims 1 to 10 or the audio encoding method according to claim 11.

19. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by the processor, implement the audio decoding method according to any one of claims 1 to 10 or the audio encoding method according to claim 11.

Images