Audio decoding method and device, audio coding method and device, electronic equipment and storage medium

Abstract

The invention provides an audio decoding method and device, an audio coding method and device, electronic equipment and a storage medium, which can be applied to a vehicle-mounted scene. The audio decoding method comprises the steps that a code stream is obtained, and the code stream is obtained by encoding an audio signal; decoding the code stream to obtain a predicted value of a feature vector of the audio signal; label extraction processing is conducted on the predicted value of the feature vector, a label information vector used for signal enhancement is obtained, and the dimension of the label information vector is the same as that of the predicted value of the feature vector; performing signal reconstruction based on the predicted value of the feature vector and the label information vector; and taking a predicted value of the audio signal obtained through signal reconstruction as a decoding result of the code stream. According to the invention, the acoustic interference in the audio signal can be effectively suppressed, and the quality of the audio signal obtained through reconstruction is improved.

Description

[0001] This application is a divisional application of the patent application with application number 202210676984.X, application date June 15, 2022, and invention name “audio decoding, encoding method, device, electronic device and storage medium”. Technical Field

[0002] The present application relates to the field of communication technology, and in particular to an audio decoding and encoding method, device, electronic device and storage medium. Background Art

[0003] Due to the convenience and timeliness of voice communication, voice calls are increasingly used, such as transmitting audio signals (such as voice signals) between participants in a network conference. In a voice call, the voice signal may be mixed with acoustic interference such as noise. The noise mixed in the voice signal will cause the call quality to deteriorate, thereby greatly affecting the user's auditory experience.

[0004] However, there is no effective solution in the related art on how to enhance the speech signal to suppress the noise part. Summary of the invention

[0005] The embodiments of the present application provide an audio decoding and encoding method, device, electronic device, computer-readable storage medium and computer program product, which can effectively suppress acoustic interference in audio signals and thereby improve the quality of reconstructed audio signals.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application provides an audio decoding method, including:

[0008] Obtaining a bit stream, wherein the bit stream is obtained by encoding the audio signal;

[0009] Decoding the code stream to obtain a predicted value of the feature vector of the audio signal;

[0010] Performing label extraction processing on the predicted value of the feature vector to obtain a label information vector for signal enhancement, wherein the dimension of the label information vector is the same as the dimension of the predicted value of the feature vector;

[0011] Reconstructing a signal based on the predicted value of the feature vector and the label information vector;

[0012] The predicted value of the audio signal obtained by reconstructing the signal is used as the decoding result of the code stream.

[0013] The present application provides an audio decoding device, including:

[0014] An acquisition module, used to acquire a bit stream, wherein the bit stream is obtained by encoding an audio signal;

[0015] A decoding module, used for decoding the bit stream to obtain a predicted value of the feature vector of the audio signal;

[0016] A label extraction module, used to perform label extraction processing on the predicted value of the feature vector to obtain a label information vector for signal enhancement, wherein the dimension of the label information vector is the same as the dimension of the predicted value of the feature vector;

[0017] A reconstruction module, used for performing signal reconstruction based on the predicted value of the feature vector and the label information vector;

[0018] The determination module is used to use the predicted value of the audio signal obtained by reconstructing the signal as the decoding result of the code stream.

[0019] The present application provides an audio encoding method, including:

[0020] Get audio signal;

[0021] The audio signal is encoded to obtain a code stream, wherein the code stream is used for an electronic device to execute the audio decoding method provided in an embodiment of the present application.

[0022] The present application provides an audio encoding device, including:

[0023] An acquisition module, used for acquiring an audio signal;

[0024] The encoding module is used to encode the audio signal to obtain a code stream, wherein the code stream is used for the electronic device to execute the audio decoding method provided in the embodiment of the present application.

[0025] An embodiment of the present application provides an electronic device, including:

[0026] A memory for storing executable instructions;

[0027] The processor is used to implement the audio encoding and decoding method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0028] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the audio encoding and decoding method provided in the embodiment of the present application when executed by a processor.

[0029] An embodiment of the present application provides a computer program product, including a computer program or instructions, which are used to implement the audio encoding and decoding method provided in the embodiment of the present application when executed by a processor.

[0030] The embodiments of the present application have the following beneficial effects:

[0031] By performing label extraction processing on the predicted value of the decoded feature vector, a label information vector for signal enhancement is obtained, and the signal is reconstructed in combination with the predicted value of the feature vector. In this way, compared with signal reconstruction based only on the predicted value of the feature vector, signal reconstruction based on the predicted value of the feature vector and the label information vector is equivalent to increasing the proportion of core components in the audio signal and reducing the proportion of acoustic interference such as noise, thereby effectively suppressing the noise components included in the audio signal collected by the encoding end, thereby improving the quality of the reconstructed audio signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of spectrum comparison at different bit rates provided in an embodiment of the present application;

[0033] Figure 2 1 is a schematic diagram of the architecture of an audio codec system 100 provided in an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of the structure of a second terminal device 500 provided in an embodiment of the present application;

[0035] Figure 4A It is a flowchart of the audio encoding method provided in an embodiment of the present application;

[0036] Figure 4B It is a flowchart of the audio decoding method provided in an embodiment of the present application;

[0037] Figure 4C It is a flowchart of the audio encoding and decoding method provided in an embodiment of the present application;

[0038] Figure 5 This is a schematic diagram of the structure of the encoding end and the decoding end provided in the embodiment of the present application.

[0039] Fig. 6A and Figure 6B It is a flowchart of the audio decoding method provided in an embodiment of the present application;

[0040] Figure 7 is a schematic diagram of an end-to-end voice communication link provided by an embodiment of the present application;

[0041] Figure 8 It is a flowchart of the audio encoding and decoding method provided in an embodiment of the present application;

[0042] Fig.9A It is a schematic diagram of ordinary convolution provided in an embodiment of the present application;

[0043] Fig. 9B is a schematic diagram of a dilated convolution provided in an embodiment of the present application;

[0044] Fig.10 Schematic diagram of the spectrum response of the low-pass part and the high-pass part of the QMF analysis filter bank provided in an embodiment of the present application;

[0045] Fig.11A It is a schematic diagram of the principle of obtaining a 4-channel sub-band signal based on a QMF filter bank provided in an embodiment of the present application;

[0046] Fig. 11B It is a schematic diagram of the principle of obtaining a 3-channel sub-band signal based on a QMF filter bank provided in an embodiment of the present application;

[0047] Fig.12 is a schematic diagram of the structure of the analysis network provided in the embodiment of the present application;

[0048] Fig.13 is a schematic diagram of the structure of an enhanced network provided in an embodiment of the present application;

[0049] Fig.14 is a schematic diagram of the structure of the synthetic network provided in the embodiment of the present application;

[0050] Fig.15 It is a flowchart of the audio encoding and decoding method provided in an embodiment of the present application;

[0051] Fig.16 It is a flowchart of the audio encoding and decoding method provided in an embodiment of the present application;

[0052] Fig.17 is a schematic diagram of the structure of a first analysis network provided in an embodiment of the present application;

[0053] Fig.18 is a schematic diagram of the structure of a first enhanced network provided in an embodiment of the present application;

[0054] Fig.19 is a schematic diagram of the structure of a first synthetic network provided in an embodiment of the present application;

[0055] Fig. 20 It is a schematic diagram comparing the encoding and decoding effects provided in the embodiments of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0057] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0058] It is understandable that in the embodiments of the present application, when user information and other related data (such as voice signals sent by users) are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0059] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0062] 1) Neural Network (NN): It is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.

[0063] 2) Deep Learning (DL): It is a new research direction in the field of Machine Learning (ML). Deep learning is to learn the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0064] 3) Vector Quantization (VQ): An effective lossy compression technology, whose theoretical basis is Shannon's rate-distortion theory. The basic principle of vector quantization is to replace the input vector with the index of the codeword in the codebook that best matches the input vector for transmission and storage, while decoding only requires a simple table lookup operation.

[0065] 4) Scalar quantization: scalar quantization, i.e. one-dimensional vector quantization, divides the dynamic range into several small intervals, each of which has a representative value. When the input signal falls into a certain interval, it is quantized to the representative value.

[0066] 5) Entropy coding: It is a lossless coding method that does not lose any information according to the entropy principle during the encoding process. It is also a key module in lossy coding and is located at the end of the encoder. Common entropy coding methods include Shannon coding, Huffman coding, Exp-Golomb coding, and arithmetic coding.

[0067] 6) Quadrature Mirror Filters (QMF): It is a filter pair that includes analysis and synthesis. The QMF analysis filter group is used to decompose the sub-band signal to reduce the signal bandwidth so that each sub-band signal can be smoothly processed by the channel; the QMF synthesis filter group is used to synthesize the sub-band signals restored by the decoding end, for example, to reconstruct the original audio signal through zero-value interpolation and bandpass filtering.

[0068] Voice coding and decoding technology is a core technology in communication services including remote audio and video calls. In simple terms, voice coding technology is to use less network bandwidth resources to transmit as much voice information as possible. From the perspective of Shannon's information theory, voice coding is a kind of source coding. The purpose of source coding is to compress the amount of data to be transmitted as much as possible at the encoding end, remove the redundancy in the information, and restore it losslessly (or nearly losslessly) at the decoding end.

[0069] The compression rate of voice codecs provided by related technologies can reach more than 10 times, that is, the original 10MB of voice data only needs 1MB to be transmitted after being compressed by the encoder, which greatly reduces the broadband resources consumed to transmit information. For example, for a broadband voice signal with a sampling rate of 16000Hz, if a 16-bit sampling depth is used, the bit rate of the uncompressed version is 256kps; if voice coding technology is used, even lossy coding, within the bit rate range of 10-20kps, the quality of the reconstructed voice signal can be close to the uncompressed version, and even the auditory perception is no different. If a higher sampling rate service is required, such as 32000Hz ultra-wideband voice, the bit rate range must be at least 30kps.

[0070] Traditional speech coding schemes provided by related technologies can generally be divided into three types according to the coding principle: waveform coding, parametric coding, and hybrid coding.

[0071] Among them, waveform coding is to directly encode the waveform of the speech signal. The advantage of this coding method is that the encoded speech quality is high, but the compression rate is not high.

[0072] Parametric coding refers to modeling the speech process, and what the encoder needs to do is to extract the corresponding parameters of the speech signal to be transmitted. The advantage of parametric coding is that the compression rate is extremely high, but the disadvantage is that the quality of the restored speech is not high.

[0073] Hybrid coding is a combination of the above two coding methods. The speech components that can be coded using parameters are expressed by parameters, and the remaining components that cannot be effectively expressed by parameters are coded using waveform coding. The combination of the two can achieve high coding efficiency and high restored speech quality.

[0074] Generally speaking, the above three coding principles are derived from classic voice signal modeling, also known as compression methods based on signal processing. Based on rate-distortion analysis and combined with decades of standardization experience, a bit rate of at least 0.75 bit / sample is recommended to provide ideal voice quality; for a broadband voice signal with a sampling rate of 16000 Hz, this is equivalent to 12 kps. For example, the IETF OPUS standard recommends 16 kps as the recommended bit rate for providing high-quality broadband voice calls.

[0075] For example, see Figure 1 , Figure 11 is a schematic diagram of spectrum comparison at different bit rates provided by the embodiment of the present application, to demonstrate the relationship between compression bit rate and quality. Curve 101 is the original speech, that is, the uncompressed audio signal; curve 102 is the effect of OPUS encoder 20kbps; curve 103 is the effect of OPUS encoder 6kbps. Figure 1 It can be seen that as the encoding bit rate increases, the compressed signal is closer to the original signal.

[0076] However, the applicant has found that the above solutions provided by the related art are mainly based on traditional signal processing methods. Under the premise of maintaining the existing quality, it is difficult to significantly reduce the bit rate.

[0077] In recent years, with the advancement of deep learning, related technologies have also provided solutions for using artificial intelligence to improve encoding bit rates.

[0078] However, the applicant also found that: for audio codec solutions based on artificial intelligence, although the bit rate can be lower than 2kbps, the general call to generate networks such as Wavenet leads to very high complexity at the decoding end, making it very challenging to use in mobile terminals, and the absolute quality is significantly different from the encoder of traditional signal processing. For end-to-end NN codec solutions, the bit rate is 6-10kbps, and the subjective quality is close to that of traditional signal processing solutions. However, deep learning networks are used at both ends of the codec, resulting in very high complexity.

[0079] In addition, both traditional signal processing solutions and solutions based on deep neural networks can only compress voice signals. However, actual voice communication is affected by acoustic interference such as noise. In other words, there is no solution in the relevant technology that has both voice enhancement and low-bitrate high-quality compression effects.

[0080] In view of this, the embodiments of the present application provide an audio decoding, encoding method, device, electronic device, computer-readable storage medium and computer program product, which can effectively suppress acoustic interference in audio signals while improving encoding efficiency, thereby improving the quality of reconstructed audio signals. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a terminal device, or as a server, or implemented by a terminal device and a server in collaboration. The following is an example of an audio decoding method provided by an embodiment of the present application being implemented in collaboration with a terminal device and a server.

[0081] For example, see Figure 2 , Figure 21 is a schematic diagram of the architecture of the audio codec system 100 provided in an embodiment of the present application, in order to realize an application that can effectively suppress acoustic interference in the audio signal while improving the coding efficiency, thereby improving the quality of the reconstructed audio signal, such as Figure 2 As shown, the audio codec system 100 includes: a server 200, a network 300, a first terminal device 400 (ie, an encoding end) and a second terminal device 500 (ie, a decoding end), wherein the network 300 can be a local area network, a wide area network, or a combination of the two.

[0082] In some embodiments, a client 410 is running on the first terminal device 400, and the client 410 can be various types of clients, such as an instant messaging client, a web conference client, a live broadcast client, a browser, etc. In response to the audio collection instruction triggered by the sender (such as the initiator of the web conference, the anchor, the initiator of the voice call, etc.), the client 410 calls the microphone in the terminal device 400 to collect the audio signal, and encodes the collected audio signal to obtain a code stream. Then, the client 410 can send the code stream to the server 200 through the network 300, so that the server 200 sends the code stream to the second terminal device 500 associated with the recipient (such as the participant of the web conference, the audience, the recipient of the voice call, etc.). After receiving the code stream sent by the server 200, the client 510 (such as an instant messaging client, a web conference client, a live broadcast client, a browser, etc.) can decode the code stream to obtain a predicted value (also called an estimated value) of the feature vector of the audio signal; then the client 510 can also call the enhancement network to perform label extraction processing on the predicted value of the feature vector to obtain a label information vector for signal enhancement, wherein the dimension of the label information vector is the same as the dimension of the predicted value of the feature vector; then the client 510 can call the synthesis network to reconstruct the signal based on the predicted value of the decoded feature vector and the label information vector obtained after the label extraction processing to obtain the predicted value of the audio signal, thereby completing the reconstruction of the audio signal, and suppressing the noise component contained in the audio signal collected by the encoding end, thereby improving the quality of the reconstructed audio signal.

[0083] The audio encoding and decoding method provided in the embodiment of the present application can be widely used in various application scenarios of voice or video calls of different types, such as in-vehicle voice realized by an application running on a vehicle terminal, voice calls or video calls made through an instant messaging client, voice calls in game applications, voice calls in a network conference client, etc. For example, voice enhancement can be performed according to the audio decoding method provided in the embodiment of the present application at the receiving end of a voice call or a server providing voice communication services.

[0084] For example, take the network conference scenario as an example. Network conference is an important part of online office. In the network conference, the sound collection device (such as a microphone) of the participants of the network conference needs to send the collected voice signal to other participants of the network conference after collecting the voice signal of the speaker. This process involves the transmission and playback of voice signals between multiple participants. If the noise mixed in the voice signal is not processed, it will greatly affect the auditory experience of the participants in the conference. In this scenario, the audio decoding method provided in the embodiment of the present application can be used to enhance the voice signal in the network conference, so that the voice signal heard by the conference participants is the enhanced voice signal (that is, the noise component in the voice signal collected by the encoding end is suppressed in the reconstructed voice signal), thereby improving the quality of voice calls in the network conference.

[0085] In other embodiments, the embodiments of the present application can be implemented with the help of cloud technology (Cloud Technology). Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0086] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The service interaction function between the above servers 200 can be realized through cloud technology.

[0087] For example, Figure 2 The server 200 shown in the figure can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. Figure 2 The terminal devices 400 and 500 shown in the figure can be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, vehicle-mounted terminals, etc., but are not limited thereto. The terminal devices (such as the first terminal device 400 and the second terminal device 500) and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0088] In some embodiments, the terminal device (such as the second terminal device 500) or the server 200 can also implement the audio decoding method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP, a web conference APP, or an instant messaging APP, etc.; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0089] Next, continue Figure 2 The structure of the second terminal device 500 shown in FIG. Figure 3 , Figure 3 is a schematic diagram of the structure of a second terminal device 500 provided in an embodiment of the present application, Figure 3 The second terminal device 500 shown includes: at least one processor 520, a memory 560, at least one network interface 530 and a user interface 540. The various components in the second terminal device 500 are coupled together via a bus system 550. It can be understood that the bus system 550 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 550 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 550 is not described in detail. Figure 3 Various buses are labeled as bus system 550 .

[0090] The processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0091] The user interface 540 includes one or more output devices 541 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 540 also includes one or more input devices 542, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0092] The memory 560 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 560 may optionally include one or more storage devices that are physically remote from the processor 520.

[0093] The memory 560 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 560 described in the embodiments of the present application is intended to include any suitable type of memory.

[0094] In some embodiments, memory 560 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0095] Operating system 561, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0096] A network communication module 562, for reaching other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0097] a presentation module 563 for enabling presentation of information via one or more output devices 541 (e.g., display screen, speaker, etc.) associated with the user interface 540 (e.g., a user interface for operating peripherals and displaying content and information);

[0098] The input processing module 564 is used to detect one or more user inputs or interactions from one of the one or more input devices 542 and translate the detected inputs or interactions.

[0099] In some embodiments, the audio decoding device provided in the embodiments of the present application can be implemented in software. Figure 3An audio decoding device 565 stored in a memory 560 is shown, which may be software in the form of a program and a plug-in, etc., and includes the following software modules: an acquisition module 5651, a decoding module 5652, a label extraction module 5653, a reconstruction module 5654 and a determination module 5655. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0100] The audio encoding and decoding method provided in the embodiment of the present application will be specifically described below in combination with the exemplary application of the terminal device provided in the embodiment of the present application.

[0101] For example, see Figure 4A , Figure 4A is a flowchart of the audio encoding method provided in the embodiment of the present application, such as Figure 4A As shown, the main steps performed at the encoding end include: step 101, obtaining an audio signal; step 102, encoding the audio signal to obtain a bit stream.

[0102] For example, see Figure 4B , Figure 4B is a flowchart of an audio decoding method provided in an embodiment of the present application, such as Figure 4B As shown, the main steps performed at the decoding end include: step 201, obtaining a bit stream; step 202, decoding the bit stream to obtain a predicted value of a feature vector of an audio signal; step 203, performing label extraction processing on the predicted value of the feature vector to obtain a label information vector for signal enhancement; step 204, reconstructing the signal based on the predicted value of the feature vector and the label information vector; step 205, using the predicted value of the audio signal obtained by signal reconstruction as a decoding result of the bit stream.

[0103] The following will take a conference system based on voice transmission over Internet Protocol (VoIP, Voice over Internet Protocol) as an example to specifically explain the audio encoding and decoding method provided in an embodiment of the present application from the perspective of interaction between a first terminal device (i.e., an encoding end), a server, and a second terminal device (i.e., a decoding end).

[0104] For example, see Figure 4C , Figure 4C is a flowchart of the audio encoding and decoding method provided in the embodiment of the present application, which will be combined with Figure 4C The steps shown are explained.

[0105] It should be noted that the steps executed by the terminal device are specifically executed by the client running on the terminal device. For the sake of convenience, this application does not make a specific distinction between the terminal device and the client running on the terminal device. In addition, it should also be noted that the audio encoding and decoding method provided in the embodiment of the present application can be executed by various forms of computer programs running on the terminal device, and is not limited to the client running on the above-mentioned terminal device, but can also be the operating system 561, software module, script and applet described above. Therefore, the example of the client in the following text should not be regarded as a limitation on the embodiment of the present application.

[0106] In the right Figure 4C Before proceeding with the description, the structures of the encoding end and the decoding end are first described.

[0107] For example, see Figure 5 , Figure 5 is a schematic diagram of the structure of the encoding end and the decoding end provided in the embodiment of the present application, such as Figure 5 As shown, the encoding end includes an analysis network for performing feature extraction processing on the input audio signal to obtain the feature vector of the audio signal, and then the feature vector of the audio signal can be quantized and encoded to obtain a bit stream. The decoding end includes an enhancement network and a synthesis network. After decoding the received bit stream to obtain the predicted value of the feature vector of the audio signal, the enhancement network can be called to perform label extraction processing on the predicted value of the feature vector of the audio signal to obtain a label information vector for signal enhancement, and then the synthesis network can be called to perform signal reconstruction based on the label information vector and the predicted value of the feature vector to obtain the predicted value of the audio signal.

[0108] The audio encoding and decoding method provided in the embodiment of the present application will be specifically described below in combination with the above-mentioned structures of the encoding end and the decoding end.

[0109] In step 301, a first terminal device obtains an audio signal.

[0110] In some embodiments, the first terminal device responds to an audio collection instruction triggered by the user and calls an audio collection device (such as a built-in microphone in the first terminal device or an external microphone) to collect audio signals to obtain audio signals, such as the voice signal of a speaker in a network conference scenario, the voice signal of a host in a live broadcast scenario, etc.

[0111] For example, taking an online conference scenario as an example, when the online conference APP running on the first terminal device receives a click operation from the user (for example, the initiator of the online conference) on the "microphone" button displayed in the human-computer interaction interface, the microphone (or microphone array) of the first terminal device is called to collect the voice signal emitted by the user to obtain the voice signal of the initiator of the online conference.

[0112] In step 302, the first terminal device encodes the audio signal to obtain a code stream.

[0113] In some embodiments, after calling the microphone to collect the audio signal, the first terminal device can encode the audio signal in the following manner to obtain a code stream: first call the analysis network (such as a neural network) to perform feature extraction on the audio signal to obtain a feature vector of the audio signal, then quantize the feature vector of the audio signal (such as vector quantization or scalar quantization) to obtain the index value of the feature vector, and finally encode the index value of the feature vector (such as entropy coding) to obtain a code stream.

[0114] For example, the first terminal device can perform feature extraction processing on the audio signal in the following manner to obtain a feature vector of the audio signal: first, perform convolution processing (such as causal convolution) on the audio signal to obtain the convolution feature of the audio signal; then, perform pooling processing on the convolution feature of the audio signal to obtain the pooling feature of the audio signal; then, perform downsampling processing on the pooled feature of the audio signal to obtain the downsampled feature of the audio signal; finally, perform convolution processing on the downsampled feature of the audio signal to obtain the feature vector of the audio signal.

[0115] In other embodiments, the first terminal device may also encode the audio signal in the following manner to obtain a code stream: decompose the collected audio signal (for example, it may be decomposed by a 2-channel QMF analysis filter group) to obtain a low-frequency sub-band signal and a high-frequency sub-band signal; then perform feature extraction on the low-frequency sub-band signal and the high-frequency sub-band signal respectively to obtain a feature vector of the low-frequency sub-band signal and a feature vector of the high-frequency sub-band signal respectively; then perform quantization encoding on the feature vector of the low-frequency sub-band signal to obtain a low-frequency code stream of the audio signal, and perform quantization encoding on the feature vector of the high-frequency sub-band signal to obtain a high-frequency code stream of the audio signal.

[0116] For example, the first terminal device can decompose the audio signal in the following manner to obtain a low-frequency sub-band signal and a high-frequency sub-band signal: first, the audio signal is sampled to obtain a sampled signal, wherein the sampled signal includes a plurality of sample points obtained by acquisition; then, the sampled signal is low-pass filtered to obtain a low-pass filtered signal; then, the low-pass filtered signal is down-sampled to obtain a low-frequency sub-band signal. Similarly, the sampled signal is high-pass filtered to obtain a high-pass filtered signal, and the high-pass filtered signal is down-sampled to obtain a high-frequency sub-band signal.

[0117] In some embodiments, the first terminal device may also encode the audio signal in the following manner to obtain a bit stream: first, the collected audio signal is decomposed to obtain N sub-band signals (for example, a 4-channel QMF analysis filter group may be used to decompose the audio signal to obtain 4 sub-band signals. For example, the audio signal may first be low-pass filtered and high-pass filtered to obtain a low-frequency sub-band signal and a high-frequency sub-band signal. Then, the low-frequency sub-band signal may be low-pass filtered and high-pass filtered again to obtain sub-band signal 1 and sub-band signal 2 correspondingly. Similarly, the decomposed audio signal may be The high-frequency subband signal is low-pass filtered and high-pass filtered again to obtain subband signal 3 and subband signal 4 accordingly. In this way, the audio signal can be decomposed into 4 subband signals by iterating two layers of 2-channel QMF analysis filtering), where N is an integer greater than 2; then, feature extraction is performed on each subband signal to obtain a feature vector of each subband signal (for example, for each subband signal obtained by decomposition, a neural network model is called to perform feature extraction to obtain a feature vector of the subband signal); then, the feature vector of each subband signal is quantized and encoded to obtain N sub-code streams.

[0118] In step 303, the first terminal device sends a code stream to the server.

[0119] In some embodiments, after the first terminal device encodes the collected audio signal to obtain a code stream, the code stream can be sent to the server through the network.

[0120] In step 304, the server sends a code stream to the second terminal device.

[0121] In some embodiments, after receiving the code stream sent by the first terminal device (i.e., the encoding end, such as the terminal device associated with the initiator of the online conference), the server can send the code stream to the second terminal device (i.e., the decoding end, such as the terminal device associated with the participants of the online conference) through the network.

[0122] In other embodiments, considering forward compatibility, a transcoder can be deployed in the server to solve the interoperability problem between the new encoder (i.e., an encoder that encodes based on artificial intelligence, such as an NN encoder) and the traditional encoder (i.e., an encoder that encodes based on the transformation of the time domain and the frequency domain, such as a G.722 encoder). For example, if the new NN encoder is deployed in the first terminal device (i.e., the transmitting end), and the traditional decoder (e.g., the G.722 decoder) is deployed in the second terminal device (i.e., the receiving end), the second terminal device will not be able to correctly decode the code stream sent by the first terminal device. In view of the above situation, a transcoder can be deployed in the server. For example, after receiving the code stream encoded by the NN encoder sent by the first terminal device, the server can first call the NN decoder to generate an audio signal, and then call the traditional encoder (e.g., the G.722 encoder) to generate a specific code stream, so that the second terminal device can decode correctly.

[0123] In step 305, the second terminal device decodes the code stream to obtain a predicted value of the feature vector of the audio signal.

[0124] In some embodiments, the second terminal device can implement step 305 in the following manner: first, decode the code stream (wherein the decoding process is an inverse process of the encoding process, for example, when the encoding end adopts entropy encoding to encode the feature vector of the audio signal to obtain the code stream, the decoding end can correspondingly adopt entropy decoding to decode the received code stream to obtain the index value of the feature vector of the audio signal), and obtain the index value of the feature vector of the audio signal; then query the quantization table based on the index value (for example, when the encoding end adopts the index value of the codeword in the quantization table that best matches the feature vector of the audio signal, instead of the feature vector for subsequent encoding, the decoding end decodes the code stream and obtains the index value. After that, it can perform a simple table lookup operation based on the index value to obtain the predicted value of the feature vector of the audio signal) to obtain the predicted value of the feature vector of the audio signal. In other embodiments, when the code stream includes a low-frequency code stream (wherein the low-frequency code stream is obtained by encoding a low-frequency subband signal obtained after decomposition of the audio signal) and a high-frequency code stream (wherein the high-frequency code stream is obtained by encoding a high-frequency subband signal obtained after decomposition of the audio signal), the second terminal device may also implement the above step 305 in the following manner: decoding the low-frequency code stream (for example, when the encoding end encodes the feature vector of the low-frequency subband signal in an entropy encoding manner, the decoding end may decode the low-frequency code stream in a corresponding entropy decoding manner) to obtain a predicted value of the feature vector of the low-frequency subband signal; decoding the high-frequency code stream to obtain a predicted value of the feature vector of the high-frequency subband signal. For example, for the low-frequency code stream, the second terminal device may first decode the low-frequency code stream to obtain an index value (assuming an index value of 1) of the feature vector of the low-frequency subband signal, and then query the quantization table based on the index value 1 to obtain a predicted value of the feature vector of the low-frequency subband signal. Similarly, for the high-frequency code stream, the second terminal device can first decode the high-frequency code stream to obtain the index value of the feature vector of the high-frequency sub-band signal (assuming the index value is 2), and then query the quantization table based on the index value 2 to obtain the predicted value of the feature vector of the high-frequency sub-band signal. In some embodiments, when the code stream includes N sub-code streams (wherein the N sub-code streams correspond to different frequency bands, and are obtained by respectively encoding the N sub-band signals obtained after the audio signal is decomposed, and N is an integer greater than 2), the second terminal device can also implement the above step 105 in the following manner: respectively decode the N sub-code streams to obtain the predicted values ​​of the feature vectors corresponding to the N sub-band signals. It should be noted that the decoding process for the N sub-code streams here can be implemented by referring to the decoding process for the low-frequency code stream or the high-frequency code stream above, and the embodiments of the present application will not be repeated here.

[0125] For example, take N sub-streams as 4 sub-streams, namely sub-stream 1, sub-stream 2, sub-stream 3 and sub-stream 4, where sub-stream 1 is obtained by encoding sub-band signal 1, sub-stream 2 is obtained by encoding sub-band signal 2, sub-stream 3 is obtained by encoding sub-band signal 3, and sub-stream 4 is obtained by encoding sub-band signal 4. After receiving the 4 sub-streams, the second terminal device can decode the 4 sub-streams respectively to obtain the predicted values ​​of the feature vectors corresponding to the 4 sub-band signals, for example, including the predicted value of the feature vector of sub-band signal 1, the predicted value of the feature vector of sub-band signal 2, the predicted value of the feature vector of sub-band signal 3 and the predicted value of the feature vector of sub-band signal 4.

[0126] In step 306, the second terminal device performs label extraction processing on the predicted value of the feature vector to obtain a label information vector for signal enhancement.

[0127] Here, the dimension of the label information vector is the same as the dimension of the predicted value of the feature vector. In this way, when the signal is reconstructed later, the predicted value of the feature vector and the label information vector can be perfectly spliced, so that all the core components of the reconstructed audio signal can be enhanced.

[0128] In some embodiments, the second terminal device can perform label extraction processing on the predicted value of the feature vector by calling the enhanced network to obtain a label information vector for signal enhancement, wherein the enhanced network includes a convolutional layer, a neural network layer, a fully connected network layer and an activation layer. Below, the process of extracting the label information vector is explained in combination with the above structure of the enhanced network.

[0129] For example, see Fig. 6A , Fig. 6A is a flowchart of an audio decoding method provided in an embodiment of the present application, such as Fig. 6A As shown, Figure 4C Step 306 shown may be performed by Fig. 6A Steps 3061 to 3064 shown are implemented by combining Fig. 6A The steps shown are explained.

[0130] In step 3061, the second terminal device performs convolution processing on the predicted value of the feature vector to obtain a first tensor of the same dimension as the predicted value of the feature vector.

[0131] In some embodiments, the second terminal device can use the predicted value of the feature vector obtained in step 305 as input, call the convolution layer (e.g., a one-dimensional causal convolution) included in the enhanced network, and generate a first tensor (a tensor is a quantity including multiple dimensional values) of the same dimension as the predicted value of the feature vector, for example, Fig.13As shown, the dimension of the predicted value of the feature vector is 56×1, and after causal convolution processing, a 56×1 tensor is generated.

[0132] In step 3062, the second terminal device performs feature extraction processing on the first tensor to obtain a second tensor of the same dimension as the first tensor.

[0133] In some embodiments, for the first tensor obtained after the causal convolution process, the neural network layer (such as a long short-term memory network, a time recursive neural network, etc.) included in the enhanced network can be used to perform feature extraction processing to generate a second tensor with the same dimension as the first tensor. Fig.13 As shown in the figure, the dimension of the first tensor is 56×1. After a layer of long short-term memory (LSTM) network is used for feature extraction, a 56×1 tensor is generated.

[0134] In step 3063, the second terminal device performs full connection processing on the second tensor to obtain a third tensor of the same dimension as the second tensor.

[0135] In some embodiments, after the feature extraction process of the neural network layer included in the enhanced network is performed to obtain a second tensor of the same dimension as the first tensor, the second terminal device can call the fully connected network layer included in the enhanced network to perform full connection processing on the second tensor to obtain a third tensor of the same dimension as the second tensor. Fig.13 As shown in the figure, the dimension of the second tensor is 56×1, so a fully connected network layer is called to perform full connection processing to generate a 56×1 tensor.

[0136] In step 3064, the second terminal device performs activation processing on the third tensor to obtain a label information vector for signal enhancement.

[0137] In some embodiments, after the third tensor of the same dimension as the second tensor is obtained through the full connection processing of the fully connected network layer included in the enhanced network, the second terminal device can call the activation layer included in the enhanced network, that is, the activation function (such as ReLU function, Sigmoid function, Tanh function, etc.) to activate the third tensor, so that a label information vector of the same dimension as the predicted value of the feature vector is generated. For example, Fig.13 As shown, the dimension of the third tensor is 56×1. After calling the ReLU function to activate the third tensor, a label information vector with a dimension of 56×1 is obtained.

[0138] In other embodiments, when the predicted value of the feature vector includes the predicted value of the feature vector of the low-frequency subband signal and the predicted value of the feature vector of the high-frequency subband signal, the second terminal device can also implement the above-mentioned step 306 in the following manner: performing label extraction processing on the predicted value of the feature vector of the low-frequency subband signal to obtain a first label information vector for low-frequency subband signal enhancement, wherein the dimension of the first label information vector is the same as the dimension of the predicted value of the feature vector of the low-frequency subband signal; performing label extraction processing on the predicted value of the feature vector of the high-frequency subband signal to obtain a second label information vector for high-frequency subband signal enhancement, wherein the dimension of the second label information vector is the same as the dimension of the predicted value of the feature vector of the high-frequency subband signal.

[0139] For example, the second terminal device can implement the above-mentioned label extraction processing on the predicted value of the feature vector of the low-frequency subband signal in the following manner to obtain the first label information vector for low-frequency subband signal enhancement: call the first enhancement network to perform the following processing: perform convolution processing on the predicted value of the feature vector of the low-frequency subband signal to obtain a fourth tensor of the same dimension as the predicted value of the feature vector of the low-frequency subband signal; perform feature extraction processing on the fourth tensor to obtain a fifth tensor of the same dimension as the fourth tensor; perform full connection processing on the fifth tensor to obtain a sixth tensor of the same dimension as the fifth tensor; perform activation processing on the sixth tensor to obtain the first label information vector for low-frequency subband signal enhancement.

[0140] For example, the second terminal device can implement the above-mentioned label extraction processing on the predicted value of the feature vector of the high-frequency sub-band signal in the following manner to obtain the second label information vector for high-frequency sub-band signal enhancement: call the second enhancement network to perform the following processing: perform convolution processing on the predicted value of the feature vector of the high-frequency sub-band signal to obtain a seventh tensor of the same dimension as the predicted value of the feature vector of the high-frequency sub-band signal; perform feature extraction processing on the seventh tensor to obtain an eighth tensor of the same dimension as the seventh tensor; perform full connection processing on the eighth tensor to obtain a ninth tensor of the same dimension as the eighth tensor; perform activation processing on the ninth tensor to obtain a second label information vector for high-frequency sub-band signal enhancement.

[0141] It should be noted that the label extraction process for the predicted value of the feature vector of the low-frequency subband signal and the label extraction process for the predicted value of the feature vector of the high-frequency subband signal are similar to the label extraction process for the predicted value of the feature vector of the audio signal, which can be referred to Fig. 6A In addition, it should be noted that the structures of the first enhanced network and the second enhanced network are similar to the structure of the enhanced network described above, and the embodiments of the present application are not repeated here.

[0142] In some embodiments, when the predicted value of the feature vector includes the predicted values ​​of the feature vectors corresponding to N sub-band signals, the second terminal device can implement the above-mentioned step 306 in the following manner: perform label extraction processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals, respectively, to obtain N label information vectors for signal enhancement, wherein the dimension of each label information vector is the same as the dimension of the predicted value of the feature vector of the corresponding sub-band signal.

[0143] By way of example, the second terminal device can implement the above-mentioned label extraction processing for the predicted values ​​of the characteristic vectors corresponding to the N subband signals respectively in the following manner to obtain N label information vectors for signal enhancement: based on the predicted value of the characteristic vector of the i-th subband signal, call the i-th enhancement network to perform label extraction processing to obtain the i-th label information vector for enhancing the i-th subband signal; wherein the value range of i satisfies 1≤i≤N, and the dimension of the i-th label information vector is the same as the dimension of the predicted value of the characteristic vector of the i-th subband signal.

[0144] For example, the second terminal device can implement the above-mentioned predicted value based on the feature vector of the i-th subband signal in the following manner, calling the i-th enhancement network to perform label extraction processing, and obtaining the i-th label information vector for enhancing the i-th subband signal: calling the i-th enhancement network to perform the following processing: performing convolution processing on the predicted value of the feature vector of the i-th subband signal to obtain a tenth tensor of the same dimension as the predicted value of the feature vector of the i-th subband signal; performing feature extraction processing on the tenth tensor to obtain an eleventh tensor of the same dimension as the tenth tensor; performing full connection processing on the eleventh tensor to obtain a twelfth tensor of the same dimension as the eleventh tensor; performing activation processing on the twelfth tensor to obtain the i-th label information vector for enhancing the i-th subband signal.

[0145] It should be noted that the structure of the i-th enhanced network is similar to the structure of the enhanced network mentioned above, and will not be described in detail in the embodiments of the present application.

[0146] In step 307, the second terminal device performs signal reconstruction based on the predicted value of the feature vector and the label information vector to obtain the predicted value of the audio signal.

[0147] In some embodiments, the second terminal device can call the synthesis network to perform signal reconstruction based on the predicted value of the feature vector and the label information vector to obtain the predicted value of the audio signal, wherein the synthesis network includes a first convolutional layer, an upsampling layer, a pooling layer, and a second convolutional layer. Below, the signal reconstruction process is explained in combination with the above structure of the synthesis network.

[0148] For example, see Figure 6B , Figure 6B is a flowchart of an audio decoding method provided in an embodiment of the present application, such as Figure 6B As shown, Figure 4C Step 307 shown may be performed by Figure 6B Steps 3071 to 3075 shown are implemented by combining Figure 6B The steps shown are explained.

[0149] In step 3071, the second terminal device concatenates the predicted value of the feature vector and the label information vector to obtain a concatenated vector.

[0150] In some embodiments, the second terminal device may concatenate the predicted value of the feature vector obtained based on step 305 and the label information vector obtained based on step 306 to obtain a concatenated vector, and use the concatenated vector as an input of a synthesis network to perform signal reconstruction.

[0151] In step 3072, the second terminal device performs a first convolution process on the concatenated vector to obtain a convolution feature of the audio signal.

[0152] In some embodiments, after concatenating the predicted value of the feature vector and the label information vector to obtain the concatenated vector, the second terminal device may call the first convolution layer (e.g., a one-dimensional causal convolution) included in the synthesis network to convolve the concatenated vector to obtain the convolution feature of the audio signal. Fig.14 As shown, after the concatenated vector is causally convolved, a tensor with a dimension of 192×1 is obtained (i.e., the convolution feature of the audio signal).

[0153] In step 3073, the second terminal device upsamples the convolution feature to obtain the upsampled feature of the audio signal.

[0154] In some embodiments, after obtaining the convolution features of the audio signal, the second terminal device can call the upsampling layer included in the synthesis network to upsample the convolution features of the audio signal, wherein the upsampling process can be implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different. The second terminal device can upsample the convolution features of the audio signal in the following manner to obtain the upsampled features of the audio signal: upsample the convolution features through the first decoding layer of multiple cascaded decoding layers; output the upsampling result of the first decoding layer to the subsequent cascaded decoding layers, and continue to perform upsampling processing and output the upsampling results through the subsequent cascaded decoding layers until it is output to the last decoding layer; use the upsampling result output by the last decoding layer as the upsampled feature of the audio signal.

[0155] For example, see Fig.14, taking multiple cascaded decoding layers (also known as decoding blocks) as 3 cascaded decoding layers as an example, 3 decoding layers with different upsampling factors (Up_factor) can be cascaded. Taking the decoding layer (Up_factor=8) as an example, one or more dilated convolutions can be performed first, and the size of each convolution kernel is fixed to 1×3 and the shift rate (Stride Rate) is 1. In addition, the dilation rate (Dilation Rate) of one or more dilated convolutions can be set according to demand, for example, it can be set to 3. Of course, the embodiment of the present application does not limit the setting of different dilation rates for different dilated convolutions. Then, the Up_factor of the three decoding layers is set to 8, 5, and 4, respectively, which is equivalent to setting pooling factors of different sizes, which plays the role of upsampling. Finally, the number of channels of the three decoding layers is set to 96, 48, and 24, respectively. In this way, after upsampling through three decoding layers, the convolutional features of the audio signal (e.g., a 192×1 tensor) will be converted into 96×8, 48×40, and 24×160 tensors respectively, and the 24×160 tensor can be used as the upsampled features of the audio signal.

[0156] In step 3074, the second terminal device performs pooling processing on the upsampled features to obtain pooled features of the audio signal.

[0157] In some embodiments, after upsampling the convolutional features of the audio signal to obtain the upsampled features of the audio signal, the second terminal device may call the pooling layer in the synthesis network to perform pooling on the upsampled features, for example, performing a pooling operation of a factor of 2 on the upsampled features to obtain the pooled features of the audio signal, for example, see Fig.14 , the upsampled features of the audio signal are 24×160 tensors, which are then pooled (i.e. Fig.14 After the post-processing shown in ), a 24×320 tensor (i.e., the pooled features of the audio signal) is generated.

[0158] In step 3075, the second terminal device performs a second convolution process on the pooled features to obtain a predicted value of the audio signal.

[0159] In some embodiments, after performing pooling processing on the upsampled features of the audio signal to obtain the pooled features of the audio signal, the second terminal device may also call the second convolutional layer (e.g. Fig.14 The causal convolution shown in Figure 1 performs a dilated convolution on the pooled features to generate a predicted value for the audio signal.

[0160] In other embodiments, when the predicted value of the feature vector includes the predicted value of the feature vector of the low-frequency subband signal and the predicted value of the feature vector of the high-frequency subband signal, the second terminal device can also implement the above-mentioned step 307 in the following manner: splicing the predicted value of the feature vector of the low-frequency subband signal and the first label information vector (i.e., the label information vector obtained by performing label extraction processing on the predicted value of the feature vector of the low-frequency subband signal) to obtain a first splicing vector; calling the first synthesis network to perform signal reconstruction based on the first splicing vector to obtain the predicted value of the low-frequency subband signal; splicing the predicted value of the feature vector of the high-frequency subband signal and the second label information vector (i.e., the label information vector obtained by performing label extraction processing on the predicted value of the feature vector of the high-frequency subband signal) to obtain a second splicing vector; calling the second synthesis network to perform signal reconstruction based on the second splicing vector to obtain the predicted value of the high-frequency subband signal; and synthesizing the predicted value of the low-frequency subband signal and the predicted value of the high-frequency subband signal to obtain the predicted value of the audio signal.

[0161] For example, the second terminal device can implement the above-mentioned calling the first synthesis network based on the first splicing vector to perform signal reconstruction and obtain the predicted value of the low-frequency sub-band signal in the following manner: calling the first synthesis network to perform the following processing: performing a first convolution processing on the first splicing vector to obtain the convolution feature of the low-frequency sub-band signal; performing an upsampling processing on the convolution feature of the low-frequency sub-band signal to obtain the upsampling feature of the low-frequency sub-band signal; performing a pooling processing on the upsampling feature of the low-frequency sub-band signal to obtain the pooling feature of the low-frequency sub-band signal; performing a second convolution processing on the pooling feature of the low-frequency sub-band signal to obtain the predicted value of the low-frequency sub-band signal; wherein the upsampling processing can be implemented by multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0162] For example, the second terminal device can implement the above-mentioned calling the second synthesis network based on the second splicing vector to perform signal reconstruction and obtain the predicted value of the high-frequency sub-band signal in the following manner: calling the second synthesis network to perform the following processing: performing a first convolution processing on the second splicing vector to obtain the convolution feature of the high-frequency sub-band signal; performing upsampling processing on the convolution feature of the high-frequency sub-band signal to obtain the upsampling feature of the high-frequency sub-band signal; performing pooling processing on the upsampling feature of the high-frequency sub-band signal to obtain the pooling feature of the high-frequency sub-band signal; performing a second convolution processing on the pooling feature of the high-frequency sub-band signal to obtain the predicted value of the high-frequency sub-band signal; wherein the upsampling processing can be implemented by multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0163] It should be noted that the reconstruction process of the low-frequency sub-band signal (i.e., the process of generating the predicted value of the low-frequency sub-band signal) and the reconstruction process of the high-frequency sub-band signal (i.e., the process of generating the predicted value of the high-frequency sub-band signal) are similar to the reconstruction process of the audio signal (i.e., the process of generating the predicted value of the audio signal). Figure 6B In addition, it should be noted that the structures of the first synthesis network and the second synthesis network are similar to the structure of the synthesis network described above, and the embodiments of the present application are not repeated here.

[0164] In other embodiments, when the predicted value of the feature vector includes the predicted values ​​of the feature vectors corresponding to N sub-band signals, the second terminal device can also implement the above-mentioned step 307 in the following manner: one-to-one splicing processing is performed on the predicted values ​​of the feature vectors corresponding to the N sub-band signals and the N label information vectors to obtain N splicing vectors; based on the j-th splicing vector, the j-th synthesis network is called to perform signal reconstruction to obtain the predicted value of the j-th sub-band signal; wherein the value range of j satisfies 1≤j≤N; the predicted values ​​corresponding to the N sub-band signals are synthesized to obtain the predicted value of the audio signal.

[0165] For example, the second terminal device can implement the above-mentioned calling the jth synthesis network based on the jth splicing vector to perform signal reconstruction and obtain the predicted value of the jth subband signal in the following manner: calling the jth synthesis network to perform the following processing: performing a first convolution processing on the jth splicing vector to obtain the convolution feature of the jth subband signal; performing an upsampling processing on the convolution feature of the jth subband signal to obtain the upsampling feature of the jth subband signal; performing a pooling processing on the upsampling feature of the jth subband signal to obtain the pooling feature of the jth subband signal; performing a second convolution processing on the pooling feature of the jth subband signal to obtain the predicted value of the jth subband signal; wherein the upsampling processing can be implemented by multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0166] It should be noted that the structure of the j-th synthesis network is similar to the structure of the synthesis network mentioned above, and will not be described in detail in this embodiment.

[0167] In step 308, the second terminal device uses the predicted value of the audio signal obtained by signal reconstruction as the decoding result of the code stream.

[0168] In some embodiments, after obtaining the predicted value of the audio signal through signal reconstruction, the second terminal device can use the predicted value of the audio signal obtained through signal reconstruction as the decoding result of the code stream, and send the decoding result to the built-in speaker of the second terminal device for playback.

[0169] The audio decoding method provided in the embodiment of the present application performs label extraction processing on the predicted value of the decoded feature vector to obtain a label information vector for signal enhancement, and combines the predicted value of the feature vector and the label information vector to reconstruct the signal. Since the label information vector reflects the core components of the audio signal (i.e., does not include acoustic interference such as noise), compared to signal reconstruction based solely on the predicted value of the feature vector, the embodiment of the present application combines the predicted value of the feature vector and the label information vector to reconstruct the signal, which is equivalent to increasing the proportion of core components (such as human voices) in the audio signal and reducing the proportion of acoustic interference such as noise (such as background sound) in the audio signal, thereby effectively suppressing the noise components included in the audio signal collected by the encoding end, thereby improving the quality of the reconstructed audio signal.

[0170] Below, a VoIP conference system is taken as an example to illustrate an exemplary application of the embodiment of the present application in an actual application scenario.

[0171] For example, see Figure 7 , Figure 7 is a schematic diagram of an end-to-end voice communication link provided by an embodiment of the present application, such as Figure 7 As shown, the audio encoding method provided by the embodiment of the present application can be applied at the encoding end (i.e., the transmitting end of the code stream), and the audio decoding method provided by the embodiment of the present application can be applied at the decoding end (i.e., the receiving end of the code stream). This is the core part of the communication system such as the conference, which solves the basic function of compression. Generally, the encoder is deployed on the uplink client and the decoder is deployed on the downlink client.

[0172] In addition, considering forward compatibility, a transcoder needs to be deployed in the server as well to solve the interoperability problem between the new encoder and the existing encoder. For example, if the sender deploys a new NN encoder and the receiver deploys a traditional public switched telephone network (PSTN) decoder (such as a G.722 decoder), the receiver will not be able to correctly decode the code stream sent directly by the sender. Therefore, after receiving the code stream sent by the sender, the server first needs to execute the NN decoder to generate a voice signal, and then call the G.722 encoder to generate a specific code stream so that the receiver can decode it correctly. Similar transcoding scenarios are no longer expanded.

[0173] The audio encoding and decoding method provided in the embodiment of the present application is described in detail below.

[0174] In some embodiments, see Figure 8 , Figure 8 is a flowchart of the audio encoding and decoding method provided in the embodiment of the present application, such as Figure 8 As shown, the main steps of the encoding end include:

[0175] For an input signal, such as the nth frame of speech signal, denoted as x(n), the analysis network is called to perform feature extraction processing to obtain a low-dimensional feature vector, denoted as F(n); in particular, the dimension of the feature vector F(n) is smaller than the dimension of the input signal x(n), thereby reducing the amount of data. A specific implementation may be to call a dilated convolutional network (Dilated CNN) to perform feature extraction processing on the nth frame of speech signal x(n) to generate a lower-dimensional feature vector F(n). It should be noted that the embodiments of the present application do not limit other NN structures, including but not limited to autoencoders (AE, Autoencoder), fully connected (FC, Full-Connection) networks, long short-term memory (LSTM, Long Short-Term Memory) networks, convolutional neural networks (CNN, Convolutional Neural Network) + LSTM, etc. After obtaining the feature vector F(n), the feature vector F(n) can be vector quantized or scalar quantized, and the index value obtained after quantization is entropy encoded to obtain a bitstream, and finally the bitstream is transmitted to the decoding end.

[0176] Continue to see Figure 8 The main steps of the decoding end include: decoding the received bitstream to obtain the estimated value of the feature vector, denoted as F′(n). Then, based on the estimated value of the feature vector F′(n), the enhancement network is called to generate the label information vector for enhancement, denoted as E(n). Finally, the estimated value of the feature vector F′(n) and the label information vector E(n) are combined to call the synthesis network (corresponding to the inverse process of the encoding end) to reconstruct the signal, suppress the noise components contained in the speech signal collected by the encoding end, and generate a signal estimate corresponding to the input signal x(n), denoted as x′(n).

[0177] In order to better understand the audio codec method provided in the present embodiment, before specifically describing the audio codec method provided in the present embodiment, the dilated convolutional network and the QMF filter group are first introduced.

[0178] For example, see Fig.9A and Fig. 9B , Fig.9A is a schematic diagram of a common convolution provided in an embodiment of the present application, Fig. 9B Schematic diagram of the dilated convolution provided in the embodiment of the present application. Compared with ordinary convolution, the dilated convolution is proposed to solve the problem of increasing the receptive field while keeping the size of the feature map unchanged, thereby avoiding errors caused by upsampling and downsampling. Fig.9A and Fig. 9BThe convolution kernel size shown in is 3×3; however, Fig.9A The receptive field of the ordinary convolution shown is only 3, while Fig. 9B The receptive field of the dilated convolution shown in the figure reaches 5. That is, for a convolution kernel of size 3×3, Fig.9A The receptive field of the ordinary convolution shown is 3, and the dilation rate is 1; Fig. 9B The dilated convolution shown has a receptive field of 5 and a dilation rate of 2.

[0179] The convolution kernel can also be used in similar Fig.9A or Fig. 9B The concept of stride rate is involved here. For example, assuming that the convolution kernel shifts 1 grid each time, the corresponding shift rate is 1.

[0180] In addition, there is the concept of the number of convolution channels, which is the number of convolution kernel parameters used to perform convolution analysis. Theoretically, the more channels there are, the more comprehensive the signal analysis is and the higher the accuracy is; however, the more channels there are, the higher the complexity is. For example, for a 1×320 tensor, a 24-channel convolution operation can be used, and the output is a 24×320 tensor.

[0181] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the size of the convolution kernel is generally 1×3), expansion rate, shift rate and number of channels can be defined according to actual application needs. The embodiments of the present application do not make specific limitations on this.

[0182] The QMF filter bank is described below.

[0183] The QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, which represent the QMF low-pass signal and the QMF high-pass signal respectively. Fig.10 As shown, it is the spectrum response of the low-pass part H_Low(z) and the high-pass part H_High(z) of the QMF analysis filter bank. Based on the relevant theoretical knowledge of the QMF analysis filter bank, the correlation between the coefficients of the above low-pass filter and high-pass filter can be easily described:

[0184] h High (k) = -1 k h Low (k)

[0185] Among them, h Low (k) represents the coefficient of low-pass filtering, h High (k) represents the coefficient of high-pass filtering.

[0186] Similarly, according to QMF related theories, the QMF synthesis filter group can also be described based on the QMF analysis filter group H_Low(z) and H_High(z). The detailed mathematical background will not be repeated here.

[0187] G Low (z) = H Low (z)

[0188] G High (z)=(-1)*H High (z)

[0189] Among them, G Low (z) represents the restored low-pass signal, G High (z) represents the recovered high-pass signal.

[0190] After the decoding end recovers the low-pass signal and the high-pass signal, they are synthesized by the QMF synthesis filter bank to recover the reconstructed signal with the sampling rate Fs corresponding to the input signal.

[0191] In addition, in addition to the above 2-channel QMF scheme, it can also be expanded to an N-channel QMF scheme; in particular, a binary number method can be used to iteratively perform 2-channel QMF analysis on the current sub-band signal to obtain a sub-band signal with lower resolution. Fig.11A It means that by iterating two layers of 2-channel QMF analysis filter, a 4-channel sub-band signal can be obtained. Fig. 11B This is another implementation method. Considering that the high-frequency signal has little impact on the quality, it does not require such high-precision analysis; therefore, only one high-pass filter is required for the original signal. Similarly, more channels can be implemented, such as 8, 16, and 32 channels, which will not be further expanded here.

[0192] The audio encoding and decoding method provided in the embodiment of the present application is described in detail below.

[0193] In some embodiments, a speech signal with a sampling rate of Fs=16000 Hz is taken as an example (it should be noted that the method provided in the embodiment of the present application is also applicable to scenarios with other sampling rates, including but not limited to: 8000 Hz, 32000 Hz, 48000 Hz). At the same time, assuming that the frame length is set to 20 ms, therefore, for Fs=16000 Hz, it is equivalent to each frame containing 320 sample points.

[0194] The following will be combined Figure 8 The flowchart of the audio encoding and decoding method shown in FIG. 1 is a flowchart of the audio encoding and decoding method, and the processes of the encoding end and the decoding end are described in detail respectively.

[0195] (I) The process on the encoding side is as follows:

[0196] Step 1: Generation of input signal

[0197] As mentioned above, for a speech signal with a sampling rate of Fs=16000 Hz, assuming that the frame length is 20 ms, the speech signal of the nth frame includes 320 sample points, which are recorded as input signal x(n).

[0198] Step 2: Call the analysis network for data compression

[0199] The purpose of the analysis network is to generate a lower-dimensional feature vector F(n) based on the input signal x(n) by calling the analysis network (e.g., a neural network). In this embodiment, the dimension of the input signal x(n) is 320, and the dimension of the feature vector F(n) is 56. From the perspective of data volume, after feature extraction by the analysis network, it plays a role of "dimensionality reduction" and realizes the function of data compression.

[0200] For example, see Fig.12 , Fig.12 is a schematic diagram of the structure of the analysis network provided in the embodiment of the present application, such as Fig.12 As shown, first call a 24-channel causal convolution to expand the input signal x(n) (where the input signal x(n) is a 1×320 tensor) to a 24×320 tensor. Then preprocess the expanded 24×320 tensor. For example, the expanded 24×320 tensor can be pooled with a factor of 2, and the activation function can be a linear rectifier function (ReLU) to generate a 24×160 tensor. Next, three encoding blocks with different downsampling factors (Down_factor) can be cascaded. Taking the encoding block (Down_factor=4) as an example, one or more dilated convolutions can be performed first, and the size of each convolution kernel is fixed to 1×3 and the stride rate (Stride Rate) is 1. In addition, the dilation rate (Dilation Rate) of one or more dilated convolutions can be set according to demand, for example, it can be set to 3. Of course, the embodiment of the present application does not limit different dilated convolutions to different dilation rates. Then, the Down_factor of the three coding blocks is set to 4, 5, and 8, which is equivalent to setting pooling factors of different sizes, which plays a role of downsampling. Finally, the number of channels of the three coding blocks is set to 48, 96, and 192, respectively. In this way, after the three coding blocks are downsampled, the 24×160 tensor will be converted into 48×40, 96×8, and 192×1 tensors respectively. Finally, the 192×1 tensor can be output as a 56-dimensional feature vector F(n) after similar preprocessing causal convolution.

[0201] Step 3: Quantization encoding

[0202] For the feature vector F(n) extracted by the encoding end, scalar quantization (i.e., each component is quantized separately) and entropy coding can be used for quantization coding. Of course, vector quantization (i.e., multiple adjacent components are combined into a vector for joint quantization) and entropy coding can also be used for quantization coding, which is not specifically limited in the embodiments of the present application.

[0203] After quantizing and encoding the feature vector F(n), a bit stream can be generated. According to experiments, high-quality compression of a 16kHz broadband signal can be achieved with a bit rate of 6-8kbps.

[0204] (II) The process of decoding is as follows:

[0205] Step 1. Decoding

[0206] Decoding is the reverse process of encoding. The received bitstream is decoded, and then the quantization table is queried based on the index value obtained by decoding to obtain the estimated value of the feature vector, which is recorded as F′(n).

[0207] Step 2: Call the enhanced network to extract the label information vector

[0208] The estimated value F′(n) of the feature vector contains a compressed version of the original speech signal collected by the encoder, reflecting the core components of the speech signal, and also contains acoustic interference such as noise mixed during collection. Therefore, the enhancement network is used to collect relevant label embedding information from the estimated value F′(n) of the feature vector to generate a relatively clean speech signal during decoding.

[0209] For example, see Fig.13 , Fig.13 is a schematic diagram of the structure of the enhanced network provided in the embodiment of the present application, such as Fig.13 As shown in the figure, with the estimated value F′(n) of the feature vector as the input, a one-dimensional causal convolution is called to generate a 56×1 tensor. Then, the 56×1 tensor is passed through a layer of LSTM network to generate a 56×1 tensor. Then, a fully connected (FC, Full-Connection) network is called to generate a 56×1 tensor. Finally, the activation function (for example, ReLU, of course, it can also be other activation functions, such as Sigmoid function, Tanh function, etc.) is called for activation processing. In this way, a label information vector of the same dimension as the estimated value F′(n) of the feature vector is generated, denoted as E(n).

[0210] Step 3: Call the synthesis network to reconstruct the signal

[0211] The purpose of the synthesis network is to concatenate the estimated value F′(n) of the feature vector obtained by the decoding end and the locally generated label information vector E(n) into a 112-dimensional vector, and then call the synthesis network to reconstruct the signal to generate an estimated value of the speech signal, denoted as x′(n). It should be noted that the synthesis network generates input vectors by concatenation, which is only one way. The embodiments of the present application do not limit other methods. For example, F′(n)+E(n) can be used as input, and the dimension is 56. For this method, please refer to Fig.14 The network can be redesigned, and the embodiments will not be described in detail here.

[0212] For example, see Fig.14 , Fig.14 is a schematic diagram of the structure of the synthetic network provided in the embodiment of the present application, such as Fig.14 As shown in the figure, the structure of the synthetic network is highly similar to that of the analysis network, such as causal convolution; but the dimension of the input volume has increased to 112 dimensions. The post-processing process is similar to the pre-processing in the analysis network. In addition, the structure of the decoding block (also called the decoding layer) is symmetrical to the encoding block (also called the encoding layer) in the analysis network. For example, the encoding block in the analysis network first performs dilated convolution and then pooling to complete downsampling, while the decoding block in the synthetic network first performs pooling to complete upsampling and then performs dilated convolution. In other words, decoding is the inverse process of encoding, which can be referred to Fig.12 The description of the embodiments of the present application will not be repeated here.

[0213] In the embodiment of the present application, the relevant networks (such as analysis network and synthesis network) of the encoding end and the decoding end can be jointly trained by collecting data to obtain the optimal parameters. At present, there are many open source platforms for neural networks and deep learning. Based on the above open source platforms, users only need to prepare data and set the corresponding network structure. After the server completes the training, the trained network can be put into use. The above embodiment of the present application assumes that the parameters of the analysis network and the synthesis network have been trained, and only discloses a specific network input, network structure and network output. The engineers in the relevant fields can further modify the above configuration according to the actual situation.

[0214] In the above embodiment, for the input signal, the analysis network, enhancement network and synthesis network are called respectively on the encoding and decoding path to complete low bit rate compression and signal reconstruction. However, the complexity of these networks is relatively high. In order to reduce the complexity, the embodiment of the present application can introduce a QMF analysis filter to decompose the input signal into sub-band signals with lower bit rates; then for each sub-band signal, the input and output dimensions of the neural network will be at least halved. In general, the computational complexity of the neural network is O(N 3 ), therefore, this “divide and conquer” idea can effectively reduce complexity.

[0215] For example, see Fig.15 , Fig.15 is a flowchart of the audio encoding and decoding method provided in the embodiment of the present application, such as Fig.15 As shown, for the input signal x(n) of the nth frame, the QMF analysis filter is used to decompose it into two sub-band signals. For example, after the input signal x(n) is decomposed by the QMF analysis filter, a low-frequency sub-band signal can be obtained, which is recorded as x LB (n) and the high frequency subband signal, denoted as x HB Next, for the low-frequency subband signal x LB (n), the first analysis network can be called to obtain the low-dimensional feature vector of the low-frequency subband signal, denoted as F LB (n). In particular, the eigenvector F of the low-frequency subband signal LB The dimension of (n) is smaller than the low-frequency subband signal x LB (n), thereby reducing the amount of data. In addition, it should be noted that since the low-frequency sub-band signal x LB The resolution of (n) is half that of the input signal x(n), so the parameters of the first analysis network can be halved accordingly, including the eigenvector F of the low-frequency subband signal LB (n).

[0216] After obtaining the eigenvector F of the low-frequency subband signal LB (n) After that, the eigenvector F of the low-frequency subband signal can be LB (n) Perform vector quantization or scalar quantization, and entropy encode the index value obtained after quantization to obtain a bit stream, which is then transmitted to the decoding end.

[0217] After receiving the code stream sent by the encoder, the decoder can decode the received code stream to obtain the estimated value of the eigenvector of the low-frequency subband signal, which is recorded as F′. LB (n). Then, we can use the estimated value F′ of the eigenvector of the low-frequency subband signal LB (n), call the first enhancement network to generate a label information vector corresponding to the low-frequency subband signal, denoted as E LB (n). Finally, combined with F′ LB (n) and E LB (n), call the first synthesis network of the inverse process corresponding to the encoding end to complete the estimated value of the low-frequency subband signal, recorded as x′ LB (n), and suppress the acoustic interference such as noise contained in the speech signal collected by the encoder. For the convenience of description, the functions of the first enhancement network and the first synthesis network are combined into the first synthesis module, that is, at the decoding end, based on F′ LB (n) and E LB(n), call the first synthesis module to reconstruct the signal, and obtain the estimated value x′ of the low-frequency subband signal LB (n).

[0218] Similarly, for the input signal x(n) decomposed by the QMF analysis filter, the high-frequency subband signal is denoted as x HB (n), in the encoding and decoding process, the second analysis network and the second synthesis module (including the second enhancement network and the second synthesis network) are called respectively, and the estimated value of the high frequency subband signal can be obtained at the decoding end, which is recorded as x′ HB (n). It should be noted that for the high frequency sub-band signal x HB (n) processing flow and low frequency sub-band signal x LB The processing flow of (n) is similar, and can refer to the low-frequency sub-band signal x LB The processing flow of (n) is implemented, and the present embodiment will not be repeated here.

[0219] Referring to the above 2-channel QMF processing example, as well as the multi-channel QMF introduced above, the features completed by iterating the 2-channel QMF can be further extended to Fig.16 The multi-channel QMF solution shown (for example, the input signal x(n) can be decomposed into N sub-band signals, and encoding and decoding processing can be performed on each sub-band signal separately) is similar in principle, so the embodiments of the present application will not be repeated here.

[0220] The following uses 2-channel QMF as an example to illustrate the audio encoding and decoding method provided in the embodiment of the present application.

[0221] (I) The process on the encoding side is as follows:

[0222] Step 1: Generation of input signal

[0223] As mentioned above, for a speech signal with a sampling rate of Fs=16000 Hz, assuming that the frame length is 20 ms, the speech signal of the nth frame includes 320 sample points, which are recorded as input signal xln).

[0224] Step 2: QMF signal decomposition

[0225] As mentioned above, for the input signal x(n), the QMF analysis filter (here specifically 2-channel QMF) can be called and downsampled to obtain two sub-band signals, namely the low-frequency sub-band signal x and LB (n) and the high frequency subband signal x HB (n). Among them, the low-frequency subband signal x LB The effective bandwidth of (n) is 0-4kHz, and the high-frequency sub-band signal x HBThe effective bandwidth of (n) is 4-8kHz, and the number of sample points per frame is 160.

[0226] Step 3: Call the first analysis network and the second analysis network for data compression

[0227] For example, when decomposing the input signal x(n) into low-frequency subband signals x LB (n) and the high frequency subband signal x HB (n) Then, for the low-frequency subband signal x LB (n), you can call Fig.17 The first analysis network shown in FIG. 1 performs feature extraction processing to obtain the feature vector F of the low-frequency subband signal. LB (n); Similarly, for the high frequency subband signal x HB (n), the second analysis network can be called to perform feature extraction processing to obtain the feature vector of the high-frequency sub-band signal, denoted as F HB (n).

[0228] It should be noted that, since the sampling rate of the subband signal is halved relative to the input signal, in this embodiment, the dimension of the feature vector of the output subband signal can be lower than the dimension of the feature vector of the input signal in the above embodiment. For example, in this embodiment, the dimension of the feature vector of the low-frequency subband signal and the dimension of the feature vector of the high-frequency subband signal can both be set to 28. In this way, the dimension of the feature vector of the overall output is consistent with the dimension of the feature vector of the input signal in the above embodiment, that is, the bit rates of the two are consistent.

[0229] In addition, considering that low frequency and high frequency have different influencing factors on speech quality, the embodiment of the present application does not limit the definition of different numbers of dimensions for the feature vectors of different sub-band signals. For example, the dimension of the feature vector of the low-frequency sub-band signal can be set to 32, while the dimension of the feature vector of the high-frequency sub-band signal is set to 24, which still ensures that the total dimension is consistent with the dimension of the feature vector of the input signal. In view of the above situation, it can be achieved by adjusting the internal parameter quantities of the first analysis network and the second analysis network accordingly, and the embodiment of the present application will not be repeated here.

[0230] Step 4: Quantization encoding

[0231] Similar to the processing of the feature vector of the input signal, considering that the dimension of the total feature vector remains unchanged, high-quality compression of the 16kHz broadband signal can be achieved through a bit rate of 6-8kbps.

[0232] (II) The process of decoding is as follows:

[0233] Step 1. Decoding

[0234] Similar to the above embodiment, by decoding the received code stream, the estimated value F′ of the feature vector of the low-frequency sub-band signal can be obtained. LB (n) and the estimated value F′ of the eigenvector of the high-frequency subband signal HB (n).

[0235] Step 2: Call the first enhanced network and the second enhanced network to extract the label information vector

[0236] For example, after decoding the received code stream, the estimated value F′ of the eigenvector of the low-frequency subband signal is obtained. LB (n) and the estimated value F′ of the eigenvector of the high-frequency subband signal HB (n) After that, the estimated value F′ of the eigenvector of the low-frequency subband signal LB (n), you can call Fig.18 The first enhancement network shown collects label embedding information (i.e., label information vector of the low-frequency part) for low-frequency speech enhancement, denoted by E LB (n) is used to generate a relatively clean low-frequency subband speech signal during decoding. The above calculation process can be referred to Fig.13 In addition, due to the halving of the sampling rate, the dimension of the output feature vector of the first analysis network in the encoding end can be referred to, and the corresponding adjustment Fig.18 The structure of the first enhanced network (including the number of parameters) is shown.

[0237] Similarly, the estimated value F′ of the eigenvector of the decoded high-frequency subband signal HB (n), the second enhanced network can be called to obtain the label information vector of the high-frequency part, which is recorded as E HB (n), for subsequent processes.

[0238] In short, after this step is executed, the label information vectors of the two sub-band signals can be obtained, which are the label information vector E of the low-frequency part and LB (n) and the label information vector E of the high-frequency part HB (n).

[0239] Step 3: Call the first synthesis network and the second synthesis network to reconstruct the signal

[0240] For example, see Fig.19 , Fig.19 is a schematic diagram of the structure of the first synthetic network provided in the embodiment of the present application, such as Fig.19 As shown, the first synthesis network can be called based on the estimated value F′ of the eigenvector of the low-frequency subband signal LB (n) and the label information vector E of the low-frequency part generated locally LB(n), generate the estimated value of the low-frequency subband signal, denoted as x′ LB (n). For the specific calculation process, please refer to Fig.14 In addition, due to the halving of the sampling rate, Fig.19 Only a specific configuration of the first synthesis network corresponding to the low-frequency part is provided, and the implementation form of the high-frequency part is similar and will not be described in detail here.

[0241] After this step, the estimated value x′ of the low-frequency subband signal is generated LB (n) and the estimated value x′ of the high frequency subband signal HB (n). In particular, acoustic interference such as noise in the two sub-band signals is effectively suppressed.

[0242] Step 4: Synthesis processing based on QMF synthesis filter

[0243] Based on the first two steps, the estimated value x′ of the low-frequency subband signal is obtained at the decoding end LB (n) and the estimated value x′ of the high frequency subband signal HB After that, it is only necessary to upsample and call the QMF synthesis filter to generate a 320-point reconstructed signal, that is, the estimated value x′(n) of the input signal x(n), thus completing the entire decoding process.

[0244] In summary, the embodiments of the present application significantly improve the coding efficiency compared with the traditional signal processing scheme by organically combining signal decomposition and related signal processing technology with deep neural network. When the complexity is acceptable, the speech enhancement is implemented at the decoding end, so that the effect of reconstructing clean speech can be achieved with low bit rate under acoustic interference such as noise. For example, see Fig. 20 The voice signal collected by the encoding end is mixed with a large amount of noise interference. Through the voice enhancement and ultra-low bit rate compression scheme provided in the embodiment of the present application, a clean voice signal can be reconstructed at the decoding end, thereby improving the quality of the voice call.

[0245] The following further describes an exemplary structure of the audio decoding device 565 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 3 As shown, the software modules stored in the audio decoding device 565 of the memory 560 may include: an acquisition module 5651, a decoding module 5652, a label extraction module 5653, a reconstruction module 5654 and a determination module 5655.

[0246] An acquisition module 5651 is used to acquire a bitstream, wherein the bitstream is obtained by encoding an audio signal; a decoding module 5652 is used to decode the bitstream to obtain a predicted value of a feature vector of the audio signal; a label extraction module 5653 is used to perform label extraction on the predicted value of the feature vector to obtain a label information vector for signal enhancement, wherein the dimension of the label information vector is the same as the dimension of the predicted value of the feature vector; a reconstruction module 5654 is used to reconstruct a signal based on the predicted value of the feature vector and the label information vector; a determination module 5655 is used to use the predicted value of the audio signal obtained by signal reconstruction as a decoding result of the bitstream.

[0247] In some embodiments, the decoding module 5652 is further used to decode the bit stream to obtain the index value of the feature vector of the audio signal; and query the quantization table based on the index value to obtain the predicted value of the feature vector of the audio signal.

[0248] In some embodiments, the label extraction module 5653 is also used to perform convolution processing on the predicted value of the feature vector to obtain a first tensor of the same dimension as the predicted value of the feature vector; perform feature extraction processing on the first tensor to obtain a second tensor of the same dimension as the first tensor; perform full connection processing on the second tensor to obtain a third tensor of the same dimension as the second tensor; perform activation processing on the third tensor to obtain a label information vector for signal enhancement.

[0249] In some embodiments, the reconstruction module 5654 is also used to concatenate the predicted value of the feature vector and the label information vector to obtain a concatenated vector; perform a first convolution on the concatenated vector to obtain a convolution feature of the audio signal; perform upsampling on the convolution feature to obtain an upsampled feature of the audio signal; perform pooling on the upsampled feature to obtain a pooled feature of the audio signal; and perform a second convolution on the pooled feature to obtain a predicted value of the audio signal.

[0250] In some embodiments, the upsampling process is implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different; the reconstruction module 5654 is also used to upsample the convolution features through the first decoding layer in the multiple cascaded decoding layers; output the upsampling result of the first decoding layer to the subsequent cascaded decoding layers, and continue to perform upsampling processing and output upsampling results through the subsequent cascaded decoding layers until it is output to the last decoding layer; the upsampling result output by the last decoding layer is used as the upsampling feature of the audio signal.

[0251] In some embodiments, the bit stream includes a low-frequency bit stream and a high-frequency bit stream, wherein the low-frequency bit stream is obtained by encoding the low-frequency sub-band signal obtained after decomposition of the audio signal, and the high-frequency bit stream is obtained by encoding the high-frequency sub-band signal obtained after decomposition of the audio signal; the decoding module 5652 is also used to decode the low-frequency bit stream to obtain the predicted value of the eigenvector of the low-frequency sub-band signal; and to decode the high-frequency bit stream to obtain the predicted value of the eigenvector of the high-frequency sub-band signal.

[0252] In some embodiments, the label extraction module 5653 is further used to perform label extraction processing on the predicted value of the feature vector of the low-frequency subband signal to obtain a first label information vector for low-frequency subband signal enhancement, wherein the dimension of the first label information vector is the same as the dimension of the predicted value of the feature vector of the low-frequency subband signal; and to perform label extraction processing on the predicted value of the feature vector of the high-frequency subband signal to obtain a second label information vector for high-frequency subband signal enhancement, wherein the dimension of the second label information vector is the same as the dimension of the predicted value of the feature vector of the high-frequency subband signal.

[0253] In some embodiments, the label extraction module 5653 is also used to call the first enhancement network to perform the following processing: convolution processing is performed on the predicted value of the feature vector of the low-frequency subband signal to obtain a fourth tensor of the same dimension as the predicted value of the feature vector of the low-frequency subband signal; feature extraction processing is performed on the fourth tensor to obtain a fifth tensor of the same dimension as the fourth tensor; full connection processing is performed on the fifth tensor to obtain a sixth tensor of the same dimension as the fifth tensor; activation processing is performed on the sixth tensor to obtain a first label information vector for low-frequency subband signal enhancement.

[0254] In some embodiments, the label extraction module 5653 is also used to call the second enhancement network to perform the following processing: perform convolution processing on the predicted value of the feature vector of the high-frequency sub-band signal to obtain a seventh tensor of the same dimension as the predicted value of the feature vector of the high-frequency sub-band signal; perform feature extraction processing on the seventh tensor to obtain an eighth tensor of the same dimension as the seventh tensor; perform full connection processing on the eighth tensor to obtain a ninth tensor of the same dimension as the eighth tensor; perform activation processing on the ninth tensor to obtain a second label information vector for high-frequency sub-band signal enhancement.

[0255] In some embodiments, the predicted value of the feature vector includes: the predicted value of the feature vector of the low-frequency subband signal, and the predicted value of the feature vector of the high-frequency subband signal; the reconstruction module 5654 is also used to splice the predicted value of the feature vector of the low-frequency subband signal and the first label information vector to obtain a first splicing vector; based on the first splicing vector, calling the first synthesis network to reconstruct the signal to obtain the predicted value of the low-frequency subband signal; splicing the predicted value of the feature vector of the high-frequency subband signal and the second label information vector to obtain a second splicing vector; calling the second synthesis network to reconstruct the signal based on the second splicing vector to obtain the predicted value of the high-frequency subband signal; synthesizing the predicted value of the low-frequency subband signal and the predicted value of the high-frequency subband signal to obtain the predicted value of the audio signal.

[0256] In some embodiments, the reconstruction module 5654 is also used to call the first synthesis network to perform the following processing: perform a first convolution process on the first splicing vector to obtain a convolution feature of the low-frequency sub-band signal; perform an upsampling process on the convolution feature to obtain an upsampling feature of the low-frequency sub-band signal; perform a pooling process on the upsampling feature to obtain a pooling feature of the low-frequency sub-band signal; perform a second convolution process on the pooling feature to obtain a predicted value of the low-frequency sub-band signal; wherein the upsampling process is implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0257] In some embodiments, the reconstruction module 5654 is also used to call the second synthesis network to perform the following processing: perform a first convolution process on the second splicing vector to obtain the convolution feature of the high-frequency sub-band signal; perform an upsampling process on the convolution feature to obtain the upsampling feature of the high-frequency sub-band signal; perform a pooling process on the upsampling feature to obtain the pooling feature of the high-frequency sub-band signal; perform a second convolution process on the pooling feature to obtain the predicted value of the high-frequency sub-band signal; wherein the upsampling process is implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0258] In some embodiments, the code stream includes N sub-code streams, the N sub-code streams correspond to different frequency bands, and are obtained by respectively encoding N sub-band signals obtained after decomposition of the audio signal, where N is an integer greater than 2; the decoding module 5652 is also used to decode the N sub-code streams respectively to obtain predicted values ​​of the feature vectors corresponding to the N sub-band signals respectively.

[0259] In some embodiments, the label extraction module 5653 is also used to perform label extraction processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals respectively, to obtain N label information vectors for signal enhancement, wherein the dimension of each label information vector is the same as the dimension of the predicted value of the feature vector of the corresponding sub-band signal.

[0260] In some embodiments, the label extraction module 5653 is also used to call the i-th enhancement network to perform label extraction processing based on the predicted value of the feature vector of the i-th subband signal, so as to obtain the i-th label information vector for enhancing the i-th subband signal; wherein the value range of i satisfies 1≤i≤N, and the dimension of the i-th label information vector is the same as the dimension of the predicted value of the feature vector of the i-th subband signal.

[0261] In some embodiments, the label extraction module 5653 is also used to call the i-th enhancement network to perform the following processing: perform convolution processing on the predicted value of the feature vector of the i-th subband signal to obtain a tenth tensor of the same dimension as the predicted value of the feature vector of the i-th subband signal; perform feature extraction processing on the tenth tensor to obtain an eleventh tensor of the same dimension as the tenth tensor; perform full connection processing on the eleventh tensor to obtain a twelfth tensor of the same dimension as the eleventh tensor; perform activation processing on the twelfth tensor to obtain an i-th label information vector for enhancing the i-th subband signal.

[0262] In some embodiments, the reconstruction module 5654 is also used to perform one-to-one splicing processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals and the N label information vectors to obtain N splicing vectors; based on the j-th splicing vector, call the j-th synthesis network to perform signal reconstruction to obtain the predicted value of the j-th sub-band signal; wherein the value range of j satisfies 1≤j≤N; and perform synthesis processing on the predicted values ​​corresponding to the N sub-band signals to obtain the predicted value of the audio signal.

[0263] In some embodiments, the reconstruction module 5654 is also used to call the jth synthesis network to perform the following processing: perform a first convolution process on the jth splicing vector to obtain the convolution feature of the jth subband signal; perform an upsampling process on the convolution feature to obtain the upsampling feature of the jth subband signal; perform a pooling process on the upsampling feature to obtain the pooling feature of the jth subband signal; perform a second convolution process on the pooling feature to obtain the prediction value of the jth subband signal; wherein the upsampling process is implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

[0264] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 4C , Fig. 6A ,or Figure 6B The present invention can be understood by referring to the description of any of the accompanying drawings.

[0265] The present embodiment provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the audio encoding and decoding method described in the present embodiment.

[0266] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the audio encoding and decoding method provided by the present application embodiment, for example, Figure 4C The audio coding and decoding method is shown.

[0267] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0268] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0269] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0270] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0271] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An audio decoding method, characterized in that: The method comprises: Obtaining a bit stream, wherein the bit stream is obtained by encoding the audio signal; Decoding the code stream to obtain a predicted value of the feature vector of the audio signal; Performing label extraction processing on the predicted value of the feature vector through an enhancement network to obtain a label information vector for signal enhancement; Reconstructing a signal based on the predicted value of the feature vector and the label information vector; The predicted value of the audio signal obtained by reconstructing the signal is used as the decoding result of the code stream.

2. The method according to claim 1, characterized in that: The enhanced network includes a convolutional layer, a neural network layer, a fully connected network layer and an activation layer; The step of performing label extraction processing on the predicted value of the feature vector through the enhancement network to obtain a label information vector for signal enhancement includes: Performing convolution processing on the predicted value of the feature vector through the convolution layer to obtain the output of the convolution layer; Performing feature extraction processing on the output of the convolution layer through the neural network layer to obtain the output of the neural network layer; Performing full connection processing on the output of the neural network layer through the fully connected network layer to obtain the output of the fully connected network layer; The activation layer performs activation processing on the output of the fully connected network layer to obtain a label information vector for signal enhancement.

3. The method according to claim 1, characterized in that The signal reconstruction based on the predicted value of the feature vector and the label information vector includes: Fusing the predicted value of the feature vector and the label information vector to obtain a fused vector; Performing a first convolution process on the fused vector to obtain a convolution feature of the audio signal; Performing up-sampling processing on the convolution feature to obtain up-sampled features of the audio signal; Performing pooling processing on the upsampled features to obtain pooled features of the audio signal; Perform a second convolution process on the pooled features to obtain a predicted value of the audio signal.

4. The method according to claim 3, characterized in that The step of fusing the predicted value of the feature vector and the label information vector to obtain a fused vector includes: concatenating the predicted value of the feature vector and the label information vector to obtain the fusion vector; or, The predicted value of the feature vector and the label information vector are added together to obtain the fusion vector.

5. The method according to claim 3, characterized in that: The upsampling process is implemented by multiple cascaded decoding layers, and the sampling factors of different decoding layers are different; The up-sampling the convolution feature to obtain the up-sampled feature of the audio signal includes: Upsampling the convolutional features through a first decoding layer in the plurality of cascaded decoding layers; Outputting the upsampling result of the first decoding layer to the subsequent cascaded decoding layers, and continuing the upsampling process and outputting the upsampling result through the subsequent cascaded decoding layers until outputting to the last decoding layer; The upsampling result output by the last decoding layer is used as the upsampling feature of the audio signal.

6. The method according to claim 1, characterized in that The decoding process of the code stream to obtain the predicted value of the feature vector of the audio signal includes: Decoding the code stream to obtain an index value of a feature vector of the audio signal; A quantization table is queried based on the index value to obtain a predicted value of the feature vector of the audio signal.

7. The method according to claim 1, characterized in that The code stream includes N sub-code streams, the N sub-code streams correspond to different frequency bands, and are obtained by respectively encoding N sub-band signals obtained after decomposing the audio signal, where N is an integer greater than 1; The decoding process of the code stream to obtain the predicted value of the feature vector of the audio signal includes: The N sub-code streams are respectively decoded to obtain predicted values ​​of the feature vectors corresponding to the N sub-band signals.

8. The method according to claim 7, characterized in that The number of the enhancement networks is N, and the enhancement networks correspond one-to-one to the predicted values ​​of the feature vectors corresponding to the sub-band signals; The step of performing label extraction processing on the predicted value of the feature vector through the enhancement network to obtain a label information vector for signal enhancement includes: Based on the predicted value of the feature vector of the i-th subband signal, the i-th enhancement network is called to perform label extraction processing to obtain the i-th label information vector for enhancing the i-th subband signal, where the value range of i satisfies 1≤i≤N.

9. The method according to claim 8, characterized in that The i-th enhanced network includes a convolutional layer, a neural network layer, a fully connected network layer and an activation layer; The method of calling the i-th enhancement network to perform label extraction processing based on the predicted value of the feature vector of the i-th subband signal to obtain the i-th label information vector for enhancing the i-th subband signal includes: Based on the predicted value of the feature vector of the i-th subband signal, the i-th enhancement network is called to perform the following processing: Performing convolution processing on the predicted value of the feature vector of the i-th subband signal through the convolution layer to obtain the output of the convolution layer; Performing feature extraction processing on the output of the convolution layer through the neural network layer to obtain the output of the neural network layer; Performing full connection processing on the output of the neural network layer through the fully connected network layer to obtain the output of the fully connected network layer; The activation layer performs activation processing on the output of the fully connected network layer to obtain an i-th label information vector for i-th subband signal enhancement.

10. The method according to any one of claims 7 to 10, characterized in that: The signal reconstruction based on the predicted value of the feature vector and the label information vector includes: Performing one-to-one fusion processing on the predicted values ​​of the feature vectors corresponding to the N sub-band signals and the N label information vectors to obtain N fusion vectors; Based on the jth fusion vector, the jth synthesis network is called to reconstruct the signal to obtain the predicted value of the jth subband signal; wherein the value range of j satisfies 1≤j≤N; The prediction values ​​corresponding to the N sub-band signals are synthesized to obtain the prediction value of the audio signal.

11. The method according to claim 10, characterized in that The step of performing one-to-one fusion processing on the predicted values ​​of the feature vectors respectively corresponding to the N sub-band signals and the N label information vectors to obtain N fusion vectors includes: Performing one-to-one concatenation processing on the predicted values ​​of the feature vectors respectively corresponding to the N sub-band signals and the N label information vectors to obtain the fusion vector; or, The predicted values ​​of the feature vectors corresponding to the N sub-band signals and the N label information vectors are added one by one to obtain the fusion vector.

12. The method according to claim 10, characterized in that The calling the jth synthesis network to perform signal reconstruction based on the jth fusion vector to obtain a predicted value of the jth subband signal includes: Call the j-th synthesis network based on the j-th fusion vector to perform the following processing: Performing a first convolution process on the j-th fusion vector to obtain a convolution feature of the j-th subband signal; Performing up-sampling processing on the convolution feature to obtain an up-sampled feature of the j-th sub-band signal; Performing pooling processing on the up-sampled features to obtain pooled features of the j-th subband signal; Performing a second convolution process on the pooled features to obtain a predicted value of the j-th subband signal; The upsampling process is implemented through multiple cascaded decoding layers, and the sampling factors of different decoding layers are different.

13. The method according to claim 12, characterized in that The upsampling process is implemented by multiple cascaded decoding layers, and the sampling factors of different decoding layers are different; The upsampling process is performed on the convolution feature to obtain the upsampling feature of the j-th subband signal, including: Upsampling the convolutional features through a first decoding layer in the plurality of cascaded decoding layers; Outputting the upsampling result of the first decoding layer to the subsequent cascaded decoding layers, and continuing the upsampling process and outputting the upsampling result through the subsequent cascaded decoding layers until outputting to the last decoding layer; The upsampling result output by the last decoding layer is used as the upsampling feature of the j-th subband signal.

14. An audio encoding method, characterized in that: The method comprises: Get audio signal; The audio signal is encoded to obtain a code stream, wherein the code stream is used for an electronic device to execute the audio decoding method according to any one of claims 1 to 13.

15. A method for processing a code stream, characterized in that: The code stream is decoded based on the audio decoding method according to any one of claims 1 to 13, or is generated by the audio encoding method according to claim 14.

16. A method for processing a code stream, characterized in that: The code stream is stored on a computer-readable storage medium, where the code stream is decoded based on the audio decoding method according to any one of claims 1 to 13, or is generated by the audio encoding method according to claim 14.

17. An audio decoding device, characterized in that: The device comprises: An acquisition module, used to acquire a bit stream, wherein the bit stream is obtained by encoding an audio signal; A decoding module, used for decoding the bit stream to obtain a predicted value of the feature vector of the audio signal; A label extraction module, used to perform label extraction processing on the predicted value of the feature vector through an enhancement network to obtain a label information vector for signal enhancement; A reconstruction module, used for performing signal reconstruction based on the predicted value of the feature vector and the label information vector; The determination module is used to use the predicted value of the audio signal obtained by reconstructing the signal as the decoding result of the code stream.

18. An audio encoding device, characterized in that: The device comprises: An acquisition module, used for acquiring an audio signal; The encoding module is used to encode the audio signal to obtain a code stream, wherein the code stream is used for an electronic device to execute the audio decoding method according to any one of claims 1 to 13.

19. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by the processor, the audio decoding method according to any one of claims 1 to 13 or the audio encoding method according to claim 14 is implemented.

20. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the audio decoding method according to any one of claims 1 to 13 or the audio encoding method according to claim 14 is implemented.

Images